Senior Data Engineer - Teradata/Databricks

Actively Reviewing

ZingMind Technologies

Noida Full-Time 4–8 yrs exp Posted 6 hours ago · Apply by Sep 14, 2026

Job Description

Work Locations : Any NTT location

Experience (Relevant) : 10+ Years

Shift Timings : 2pm to 11pm IST

Hybrid : 3 days/ week

Start date : ASAP

Position Overview

NTT DATA Americas is seeking a highly skilled Senior Data Engineer to support a Teradata Utilization Analysis engagement, which focuses on analyzing Teradata platform utilization to identify cost-reduction opportunities and deliver a repeatable, log-driven analysis solution.

The ideal candidate combines deep Teradata DBA expertise with hands-on Databricks engineering capability and applied AI/ML skills. The analysis will be performed primarily in Databricks, leveraging DBQL logs, system metadata, and external scheduling/orchestration data sources.

Key Responsibilities

Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.
Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.
Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.
Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.
Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.
Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.
Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.
Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.

Required Skills & Experience

Teradata DBA Skills :

Teradata DBA / Administration : Primary platform under analysis; deep knowledge of system internals required.
Teradata DBQL (Database Query Logging) : Core data source - SQL text, object usage, timestamps, user/app IDs, row counts, steps.
Teradata System Views & Space Metadata : 5+ years - Object inventory, space analysis, last read/write tracking.
Teradata SQL & Performance Tuning : Writing complex analytical queries across DBQL and system catalog tables.
Teradata ResUsage / Workload Statistics : Quantifying CPU/IO cost impact for dataset removal/archival recommendations.
Data Classification (hot/warm/cold) : 3+ years - Categorizing datasets for tiered storage management recommendations.
ETL Pipeline Assessment (DataStage, Autosys) : 4+ years - Identifying and recommending decommission of stale ingestion pipelines.
Teradata BTEQ / FastExport / TPT : 3+ years - Data extraction, log export, and metadata collection utilities.

Databricks Engineering Skills

Databricks Platform (Workspaces, Clusters, Jobs) : Core analysis and solution delivery platform for the engagement.
Apache Spark (PySpark / Spark SQL) : Large-scale log ingestion, transformation, and analytical processing.
Delta Lake : Building reliable, ACID-compliant data pipelines for log-driven analysis.
Databricks Notebooks & Workflows : Developing repeatable, shareable analytical notebooks for customer handoff.
Python (pandas, numpy, matplotlib) : Data wrangling, statistical analysis, and visualization of usage heatmaps.
SQL (Spark SQL / ANSI SQL) : Analytical querying across large DBQL datasets in Databricks.
Unity Catalog / Data Governance in Databricks : Organizing deliverable assets and maintaining data lineage within the solution.
Dashboarding (Databricks SQL / Power BI) : Usage heatmaps, partition access patterns, and recommendation visualizations.

AI / ML Skills

Machine Learning (scikit-learn, MLflow) : Anomaly detection on access patterns, predictive cold-data classification.
LLM / Generative AI Integration : LLM-assisted log interpretation, automated recommendation narrative generation.
Feature Engineering on Time-Series Log Data : 3+ years - Extracting access frequency, recency, and seasonality signals from DBQL logs.
Clustering & Classification Algorithms : 3+ years - Unsupervised grouping of datasets by usage behavior for cold/warm/hot tiers.
MLflow / Experiment Tracking : 2+ years - Tracking model runs for reproducible analysis and customer handoff artifacts.
Natural Language Processing (NLP) : Parsing and classifying SQL text from DBQL for workload pattern analysis.
Data Visualization (Plotly, Matplotlib, Seaborn) : Generating usage heatmaps and partition access charts for stakeholder readouts.

Preferred Qualifications

Familiarity with DataStage orchestration log parsing.
Prior work on data platform cost optimization or cloud migration readiness assessments.
Experience working in healthcare payer data environments (HIPAA awareness).
Strong written communication skills for advisory deliverable authoring (Observation Reports, recommendation backlogs).

(ref:hirist.tech)