Data Engineer - II

Job Description

About Us: Airtel Payments Bank, India's first payments bank is a completely digital and paperless bank. The bank aims to take basic banking services to the doorstep of every Indian by leveraging Airtel's vast retail network in a quick and efficient manner. At Airtel Payments Bank, we?re transforming the way banking operates in the country. Our core business is banking and we?ve set out to serve each unbanked and underserved Indian. Our products and technology aim to take basic banking services to the doorstep of every Indian. We are a fun-loving, energetic and fast growing company that breathes innovation. We encourage our people to push boundaries and evolve from skilled professionals of today to risk-taking entrepreneurs of tomorrow. We hire people from every realm and offer them opportunities that encourage individual and professional growth. We are always looking for people who are thinkers & doers; people with passion, curiosity & conviction; people who are eager to break away from conventional roles and do 'jobs never done before?. About the Team: We are a team of engineers and problem-solvers building scalable, modern data infrastructure. Our mission is to power intelligent decision-making through clean, reliable, and real-time data pipelines using technologies like Kafka, PySpark, Hadoop, Airflow, and AWS. If you love working with data at scale, building cloud-native solutions, and improving pipeline reliability, you'll thrive here. Key Responsibilities: ?Design, build, and maintain scalable ETL/ELT pipelines using Python, PySpark, and SQL. ?Develop real-time data ingestion and streaming solutions using Apache Kafka and AWS Kinesis. ?Leverage the Hadoop ecosystem and AWS EMR for distributed data processing. ?Automate and orchestrate workflows using Apache Airflow (deployed via MWAA on AWS). ?Build and expose data services and APIs using Flask, deployed via Nginx. ?Implement centralized logging and monitoring with Filebeat and the ELK/Opensearch stack. ?Work extensively with AWS services including S3, API Gateway, OpenSearch, Kinesis, and EMR. ?Collaborate with data scientists, analysts, and platform engineers to ensure high-quality, accessible data. Must-Have Skills: ?4?7 years of experience in data engineering. ?Proficiency in Python and SQL (Experience with Oracle or similar RDBMS). ?Good experience with Apache Kafka (producer/consumer architecture, stream processing concepts). ?Hands-on with PySpark, Hadoop & Hive for big data processing. ?Workflow orchestration using Apache Airflow (and optionally MWAA). ?Building APIs with Flask and serving via Nginx. ?Logging and observability using ELK Stack and Filebeat. ?Good familiarity with AWS ecosystem: EMR, Kinesis, S3, OpenSearch, API Gateway, MWAA. Good-to-Have (But not mandatory) ?Experience with AWS Glue, Athena, and Redshift for serverless data processing and warehousing. ?Familiarity with AWS Flink or other stream processing frameworks. ?Exposure to AWS DMS (Data Migration Service) for database migrations and replication tasks. ?Knowledge of AWS QuickSight for dashboarding and BI reporting. ?Understanding of data lake architectures and event-driven processing on AWS Why Join Us? Airtel Payments Bank is transforming from a digital-first bank to one of the largest Fintech company. There could not be a better time to join us and be a part of this incredible journey than now. We at Airtel payments bank don?t believe in all work and no play philosophy. For us, innovation is a way of life and we are a happy bunch of people who have built together an ecosystem that drives financial inclusion in the country by serving 300 million financially unbanked, underbanked, and underserved population of India. Some defining characteristics of life at Airtel Payments Bank are Responsibility, Agility, Collaboration and Entrepreneurial development : these also reflect in our core values that we fondly call RACE..