Big Data Engineer - ETL/PySpark

Job Description

Description Job Summary : We are looking for an experienced Big Data Engineer to design, build, and optimize data pipelines and infrastructure supporting large-scale data processing and analytics. You will be instrumental in migrating our legacy systems to a modern, scalable data platform. Key Responsibilities ETL/ELT Pipeline Development: Design, construct, and maintain robust, scalable, and fault-tolerant ETL/ELT pipelines for ingesting and transforming high-volume, high-velocity data using Apache Spark or Databricks. Data Warehousing: Implement and manage modern data warehousing solutions on the cloud, utilizing platforms like Snowflake, AWS Redshift, or Google BigQuery. Data Architecture: Evaluate and implement optimal database solutions (relational, NoSQL, data lake technologies like Delta Lake or Hudi) for different use cases. Performance Optimization: Tune and optimize Spark jobs and database queries for maximum performance and cost efficiency. Automation & Monitoring: Integrate data pipelines with workflow management tools like Apache Airflow and establish monitoring and alerting for data quality and pipeline health. Required Technical Skills Big Data Frameworks: Expert proficiency with Apache Spark (PySpark or Scala). Programming: Strong skills in Python or Scala. Databases/Warehousing: Expertise in SQL and hands-on experience with modern data warehouses (Snowflake, Redshift, BigQuery). Workflow Orchestration: Hands-on experience with Apache Airflow or similar tools. Cloud & Storage: Experience with cloud storage services (AWS S3, Azure Data Lake Storage, Google Cloud Storage) and cloud platform services. Streaming (Bonus): Familiarity with real-time streaming technologies like Kafka or Kinesis (ref:hirist.tech)