Data Engineer-AWS, Python

Job Description

Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries. Job Summary For this role, we are seeking a Senior Data Engineer for our Client's ETL Support Team to act as a technical pillar during our Client's transition to an AWS-native data environment. You will be a key escalation point, responsible for handling the most complex data pipeline issues. This role may move beyond simple support; you might be requested to harden and improve our Client's systems by driving performance tuning, cost optimization, and automation initiatives. Work Location: Noida, Sec 62 Working days per week: 5 Rotational Shift : Candidates should be flexible working on a Roster basis including Weekends but maximum 5 days a week Responsibilities: Job Monitoring: Monitor approximately 2,300 scheduled jobs (daily, weekly, monthly) to ensure timely and successful execution. Execute on-demand jobs as required by the business. Troubleshooting & RCA: Troubleshoot job failures, perform detailed root cause analysis (RCA), and provide clear documentation for all findings. Issue Resolution: Address and resolve bugs and data-related issues reported by the business team. Data Integrity: Verify source file placement in designated directories to maintain data integrity. Reload Change Data Capture (CDC) tables when structural changes occur in source systems. Data Synchronization: Help manage synchronization between external databases (including Teradata write-backs) and AWS Glue tables. Development: Assist in developing new solutions, enhancements, and bug fixes using AWS Glue, Lambda, and Step Functions. Stakeholder Support: Answer questions from the business and support User Acceptance Testing (UAT) inquiries. Decision Making: Make timely decisions to resolve issues, execute tasks efficiently, and escalate complex problems to senior or lead engineers as needed, all while maintaining agreed-upon SLAs. Documentation: Participate in knowledge transfer to internal resources and assist in creating necessary documentation and runbooks. Qualifications: Experience: 3-5 years of relevant experience. Technical Skills: In-depth knowledge of AWS Glue, AWS Lambda, and AWS Step Functions. Core Concepts: A deep understanding of ETL processes and data warehouse design. Problem-Solving: Proven ability to troubleshoot data pipelines and perform root cause analysis (RCA). Should have worked on Python scripting Experience: Hands-on experience with Glue, Lambda, and Step Function development. Availability: Must be able to work a day shift that includes coverage for weekends and holidays on a rotational basis. Preferred Skills: Familiarity with Change Data Capture (CDC) processes, Teradata, and integration with SAS & RMJ events.