Senior Software Engineer

Job Description

Job Description: Data Engineer with Databricks Expertise Job Summary We are seeking a highly skilled Data Engineer with Databricks certifications and extensive practical experience working with the Databricks Lakehouse platform and its supporting capabilities. The ideal candidate will possess a robust understanding of AWS services relevant to data engineering and demonstrate proficiency in data pipeline development, programming, data modeling, and governance. Key Responsibilities Design and implement data pipelines using AWS and Databricks. Manage and orchestrate data workflows, ensuring efficient data ingestion processes. Utilize Delta Lake for data storage and management, ensuring secure data sharing across platforms with Delta Sharing. Implement data governance and access control using Unity Catalog. Apply the Medallion architecture (Bronze, Silver, Gold layers) to projects. Develop and manage ETL pipelines using AWS Glue. Leverage Amazon S3 for data storage and management. Utilize AWS Lambda and Step Functions for serverless computing in data processing. Apply data warehousing solutions and best practices in data transformation. Monitor and implement data quality checks and governance practices. Collaborate with data scientists, analysts, and other stakeholders in an Agile environment. Deliver large-scale data processing and analytics projects within cloud environments. Required Skills And Qualifications Databricks certifications with practical experience in the Databricks Lakehouse platform. Proficient in AWS services: Glue, S3, Lambda, Step Functions, and data warehousing solutions. Strong programming skills in Python, Scala, or Java. Proficient in SQL for querying and manipulating data. Experience in data modeling for Lakehouse architecture. Knowledge of ETL best practices and data transformation techniques. Understanding of data governance frameworks and practices. Proven track record of delivering data engineering projects, particularly in cloud environments. Experience with large-scale data processing and analytics projects. Ability to work in Agile teams and collaborate effectively with various stakeholders. Preferred Qualifications Advanced AWS certifications. Experience with additional cloud platforms (e.g., Azure). Familiarity with other data engineering tools and frameworks.