Job Description

Job Description We are seeking a skilled PySpark Development Engineer to design, develop, and optimize large-scale data processing pipelines. This role is ideal for someone passionate about building robust data solutions using distributed computing frameworks and ensuring high-quality, accurate data delivery. The candidate should have strong technical expertise in PySpark, ETL processes, and cloud-based data engineering, combined with an analytical mindset and problem-solving skills. We value humility, adaptability, technical curiosity, and a commitment to continuous learning. PySpark Development Engineer collaborates closely with Data Architects, QA Engineers, Product Managers, and Project Managers to create scalable data workflows that meet business requirements and adhere to quality standards. The ability to handle complex datasets and transform them into actionable insights is essential. Required Skills Technical Skills : Scripting: Proficiency in Python and Shell scripting for automation and custom transformations. Minimum 4 Years of Data Engineering: Proven experience in designing and implementing data pipelines and transformations using PySpark. PySpark Expertise: Strong hands-on experience with PySpark for distributed data processing and optimization. ETL/Data Engineering: Deep understanding of ETL concepts and experience building automated workflows. SQL Proficiency: Ability to write and optimize complex SQL queries for large datasets. Cloud & Big Data: Experience with AWS services (Glue, S3, EMR) and big data ecosystems. CI/CD & Version Control: Familiarity with Git, Jenkins, and automated deployment pipelines. Scripting: Proficiency in Python and Shell scripting for automation and custom transformations. Data Quality & Validation: Implementing data quality checks and ensuring integrity throughout the pipeline. Soft Skills Exceptional problem-solving and analytical skills. Excellent time management and organizational abilities in high-pressure environments. Strong written and verbal communication skills. Detail-oriented with a proactive approach to identifying and resolving issues. Skills: etl,pyspark,scripting