EverestDX - Azure Data Engineer

Job Description

Key Responsibilities Data Pipeline Development : Design, develop, and maintain end-to-end Azure Data Factory (ADF) pipelines for ingesting data from multiple structured and unstructured sources. Build scalable ETL/ELT processes to extract, transform, and load data into Azure Data Lake, Azure SQL Data Warehouse / Synapse, and other cloud storage systems. Implement reusable pipeline components and parameterization for optimized workflows. Data Lake & Data Warehouse Engineering : Manage and optimize Azure Data Lake Storage (ADLS) for high-volume data ingestion and processing. Design and develop data models, schemas, and storage layers for enterprise Data Warehouse and analytical workloads. Implement best practices for partitioning, indexing, and performance optimization. ETL Optimization & Performance : Fine-tune ETL processes to maximize efficiency, minimize latency, and enhance reliability. Ensure scalable architecture that supports both batch and real-time data processing. Use monitoring tools to analyze pipeline performance and troubleshoot bottlenecks. Data Quality, Testing & Governance: Implement data validation, testing, and data quality checks throughout the pipeline lifecycle. Ensure accuracy, completeness, and consistency of data across systems. Support data governance initiatives, including metadata management using Azure Data Catalog. Advanced Processing & Architecture : Work with Apache Spark, Databricks, or similar frameworks for large-scale data processing. Contribute to data architecture decisions, design reviews, and cloud migration strategies. Collaborate with architects to implement secure, governed data environments. Documentation & Collaboration : Create and maintain comprehensive documentation including pipeline designs, data dictionaries, mapping documents, and process flows. Collaborate with cross-functional teams including BI, analytics, product owners, and business stakeholders. Provide support for data-related inquiries, enhancements, and issue resolution. Monitoring & Troubleshooting : Monitor scheduled jobs, diagnose data pipeline failures, and perform timely resolutions. Implement alerting, logging, and automated recovery mechanisms to ensure system reliability. Perform root cause analysis and continuous improvement on recurring issues. Skills & Experience Required Technical Skills : Strong hands-on experience with Azure Data Factory (ADF) Expertise in Azure Data Lake Storage (ADLS) Strong knowledge of Data Warehouse concepts and architecture Hands-on experience with Apache Spark, Databricks (preferred), and distributed data processing Proficiency in ETL/ELT development and orchestration Knowledge of Azure Data Catalog, metadata management, and data governance Strong SQL skills for data manipulation, analysis, and optimization Experience with version control (Git), CI/CD for data pipelines (preferred) Soft Skills Strong analytical and problem-solving mindset Excellent communication and documentation skills Ability to collaborate with stakeholders, business units, and vendors Detail-oriented with a commitment to quality and consistency Ability to work independently and in agile teams (ref:hirist.tech)