Data Engineer
Bengaluru, Karnataka, India
1 month ago
Applicants: 0
N/A
Job Description
? Design, develop, and optimize Spark-based data pipelines on Databricks for large-scale data processing. ? Design, develop, and optimize AWS pipeline as applicable. ? Implement and manage GitHub asset bundles for version control, CI/CD workflows, and collaborative development. ? Leverage Databricks capabilities including Delta Lake, MLflow, and notebooks to build robust data solutions. ? Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality, production-ready datasets. ? Ensure data quality, integrity, and security across all stages of the pipeline. ? Monitor and troubleshoot performance issues in Spark jobs and Databricks clusters. ? Document technical designs, workflows, and best practices. Required Qualifications ? Minimum 5+ years of hands-on experience with Spark in a production environment. ? Strong proficiency in Databricks platform and ecosystem. ? Solid understanding of GitHub asset bundle management, including branching strategies, pull requests, and release workflows. ? Proficient in Python, Scala, or SQL for data engineering tasks. ? Experience with Delta Lake, Structured Streaming, and Databricks Workflows. ? Familiarity with cloud platform (AWS) and AWS data services. ? Excellent problem-solving and communication skills.
Additional Information
- Company Name
- ACL Digital
- Industry
- N/A
- Department
- N/A
- Role Category
- Data Analyst
- Job Role
- Mid-Senior level
- Education
- No Restriction
- Job Types
- Remote
- Gender
- No Restriction
- Notice Period
- Less Than 30 Days
- Year of Experience
- 1 - Any Yrs
- Job Posted On
- 1 month ago
- Application Ends
- N/A
