Senior Data Engineer-AWS, Python

Job Description

Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries. Job Summary For this role, we are seeking a Senior Data Engineer for our Client's ETL Support Team to act as a technical pillar during our Client's transition to an AWS-native data environment. You will be a key escalation point, responsible for handling the most complex data pipeline issues. This role may move beyond simple support; you might be requested to harden and improve our Client's systems by driving performance tuning, cost optimization, and automation initiatives. Work Location: Noida, Sec 62 Working days per week: 5 Rotational Shift : Candidates should be flexible working on a Roster basis including Weekends but maximum 5 days a week Responsibilities: Automate data workflows such as data ingestion, aggregation, and ETL processing. Logging and instrumentation of pipelines and services. Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures. Advanced Support: Lead the monitoring (~2,300 jobs) and support of all scheduled and on-demand ETL jobs, acting as the primary escalation point for the team. Expert Troubleshooting: Own the most complex troubleshooting and Root Cause Analysis (RCA) efforts, ensuring detailed documentation and implementing long-term, structural fixes. Optimization: Proactively perform performance tuning to improve system efficiency and reliability. Actively identify and implement cost optimization strategies for AWS Glue usage. Automation & Development: Design and develop new automation solutions (using Glue, Lambda, and Step Functions) to reduce the need for manual support. Lead development for business enhancement requirements. Data Management: Oversee and execute complex data integrity tasks, including reloading CDC tables and managing data synchronization between external databases (like Teradata) and AWS Glue. Integration: Implement AWS Lambda functions to trigger external events, such as SAS & RMJ, as needed. Decision & Execution: Make critical, autonomous decisions under pressure to resolve incidents and execute complex tasks, ensuring all SLAs are met. Escalate to leadership only when necessary. Mentorship: Mentor junior engineers, lead knowledge transfer sessions with internal resources, and oversee the creation of high-quality technical documentation and runbooks. Qualifications: Technical Expertise: In-depth, expert-level knowledge and extensive hands-on experience with AWS S3, Glue, Lambda, Step Functions and other related services. Demonstrated expertise with a minimum of 5-8 years of relevant experience. Deep understanding of Data engineering, ETL principles and data warehouse design. Optimization Experience: Demonstrable experience in system performance tuning and cloud cost optimization, specifically with AWS Glue. Automation: Proven ability to design and develop automation solutions from the ground up. Problem-Solving: Expert-level troubleshooting and RCA skills. Leadership: Ability to work independently, make critical decisions, and provide technical guidance. Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis. Database Skills: Strong experience with Change Data Capture (CDC), managing database synchronization, and working with Teradata.