Senior Data Engineer

Position: Senior Data Engineer

Exp: 10+ years

Location: Pune

Senior Data Engineer to support a Teradata Utilization Analysis engagement, which focuses on analyzing Teradata platform utilization to identify cost-reduction opportunities and deliver a repeatable, log-driven analysis solution.

About the Role

The ideal candidate combines hands-on Databricks engineering capability and applied AI/ML skills with exposure to Teradata. The analysis will be performed primarily in Databricks, leveraging DBQL logs, system metadata, and external scheduling/orchestration data sources.

Responsibilities

• Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.
• Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.
• Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.
• Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.
• Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.
• Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.
• Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.
• Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.

Required Skills

Databricks Engineering Skills

• Minimum 4+ years in Databricks Platform (Workspaces, Clusters, Jobs)
• Minimum 4+ years in Apache Spark (PySpark / Spark SQL)
• Minimum 3+ years in Delta Lake
• Minimum 3+ years in Databricks Notebooks & Workflows
• Minimum 5+ years in Python (pandas, numpy, matplotlib)
• Minimum 7+ years in SQL (Spark SQL / ANSI SQL)
• Minimum 2+ years in Unity Catalog / Data Governance in Databricks
• Minimum 3+ years in Dashboarding (Databricks SQL / Power BI)

AI / ML Skills

• Minimum 3+ years in Machine Learning (scikit-learn, MLflow)
• Minimum 2+ years in LLM / Generative AI Integration
• Minimum 3+ years in Feature Engineering on Time-Series Log Data
• Minimum 3+ years in Clustering & Classification Algorithms
• Minimum 2+ years in MLflow / Experiment Tracking
• Minimum 2+ years in Natural Language Processing (NLP)
• Minimum 3+ years in Data Visualization (Plotly, Matplotlib, Seaborn)

Teradata DBA Skills (Good to have)

• Minimum 7+ years in Teradata DBA / Administration
• Minimum 5+ years in Teradata DBQL (Database Query Logging)
• Minimum 5+ years in Teradata System Views & Space Metadata
• Minimum 7+ years in Teradata SQL & Performance Tuning
• Minimum 3+ years in Teradata ResUsage / Workload Statistics
• Minimum 3+ years in Data Classification (hot/warm/cold)
• Minimum 4+ years in ETL Pipeline Assessment (DataStage, Autosys)
• Minimum 3+ years in Teradata BTEQ / FastExport / TPT

Preferred Skills

• Familiarity with DataStage orchestration log parsing
• Prior work on data platform cost optimization or cloud migration readiness assessments
• Experience working in healthcare payer data environments (HIPAA awareness)
• Strong written communication skills for advisory deliverable authoring (Observation Reports, recommendation backlogs)