Airflow Data Engineer

Job Description

Develops and maintains scalable data pipelines to support continuing increases in data volume and complexity. Collaborates with analytics and business teams to improve data models that feed business intelligence tools, increase data accessibility, and fostering data-driven decision making across the organization. Implement processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it Writes unit/integration tests, contributes to engineering wiki, and documents work. Designs and develops data integrations and data quality framework Bachelor of Engineering or equivalent degree in Computer Science or a related technical field At least 7+ years of experience in the Data engineering field. 6+ years of experience in Big Data, Hadoop Ecosystem, Spark, Python, SQL, Hive, Proficiency in relational SQL and NoSQL databases. In-depth skills in developing and maintaining ETL/ELT data pipelines Experience in data modelling techniques such as Kimball star schema. Hands-on experience with Azure Cloud services such as Azure Data Factory (ADF), Databricks, and Cloud Storage. Proficient In Airflow configuration and creating dependencies . Experience in workflow management tools such as Airflow, Data streaming App such as Kafka, and CI/CD tools Excellent communication and collaboration skills, with the ability to work effectively in a team environment