Python+SQL+Airflow developer

Job Description

Role Purpose

The purpose of this role is to design, test and maintain software programs for operating systems or applications which needs to be deployed at a client end and ensure its meet 100% quality assurance parameters

͏

Do

• Job Description: Airflow / Big Data Engineer

  Role Summary

  We are looking for a highly skilled Airflow / Big Data Engineer with strong hands-on experience in Apache Airflow, Python, and SQL . The ideal candidate will be responsible for building, orchestrating, and optimizing scalable data pipelines in a cloud-based big data environment.

  Key Responsibilities

  • Design, develop, and maintain robust Airflow DAGs to orchestrate complex ETL/ELT workflows with dependencies, retries, and monitoring.
  • Develop Python-based data pipelines for data ingestion, transformation, validation, and automation.
  • Write complex and optimized SQL queries for data extraction, transformation, and analysis.
  • Integrate Airflow with cloud and big data services such as AWS Glue, EMR, Databricks, Snowflake, or similar platforms.
  • Monitor, troubleshoot, and optimize data pipelines to ensure high performance, reliability, and scalability .
  • Implement data quality checks, logging, alerting, and error handling mechanisms.
  • Collaborate closely with data architects, analytics teams, and stakeholders to understand data requirements.
  • Follow best practices for version control (Git), CI/CD, and documentation .

  Mandatory Skills

  • Strong expertise in Apache Airflow (DAG design, scheduling, monitoring, troubleshooting).
  • Strong programming skills in Python for data engineering and automation.
  • Strong SQL skills (complex joins, performance tuning, large datasets).
  • Hands-on experience with Big Data / Cloud data platforms (AWS / Azure / GCP).
  • Experience working with ETL/ELT pipelines and data warehousing concepts.
  • Familiarity with Linux/Unix environments .

͏

2. Perform coding and ensure optimal software/ module development

• Determine operational feasibility by evaluating analysis, problem definition, requirements, software development and proposed software
• Develop and automate processes for software validation by setting up and designing test cases/scenarios/usage cases, and executing these cases
• Modifying software to fix errors, adapt it to new hardware, improve its performance, or upgrade interfaces.
• Analyzing information to recommend and plan the installation of new systems or modifications of an existing system
• Ensuring that code is error free or has no bugs and test failure
• Preparing reports on programming project specifications, activities and status
• Ensure all the codes are raised as per the norm defined for project / program / account with clear description and replication patterns
• Compile timely, comprehensive and accurate documentation and reports as requested
• Coordinating with the team on daily project status and progress and documenting it
• Providing feedback on usability and serviceability, trace the result to quality risk and report it to concerned stakeholders

͏

3. Status Reporting and Customer Focus on an ongoing basis with respect to project and its execution

• Capturing all the requirements and clarifications from the client for better quality work
• Taking feedback on the regular basis to ensure smooth and on time delivery
• Participating in continuing education and training to remain current on best practices, learn new programming languages, and better assist other team members.
• Consulting with engineering staff to evaluate software-hardware interfaces and develop specifications and performance requirements
• Document and demonstrate solutions by developing documentation, flowcharts, layouts, diagrams, charts, code comments and clear code
• Documenting very necessary details and reports in a formal way for proper understanding of software from client proposal to implementation
• Ensure good quality of interaction with customer w.r.t. e-mail content, fault report tracking, voice calls, business etiquette etc
• Timely Response to customer requests and no instances of complaints either internally or externally

͏

Deliver