Data Engineer - AWS

Position: Cloud Data Engineer- AWS+Big Data

Purpose of the Position: You will be a critical member of the InfoCepts Cloud Data Architect Team. This position requires a sound knowledge on Big Data using Cloud technologies, EMR, Athena, Spark-Scala, PySpark, S3, AWS Lambda, etc. Strong foundation on database concepts and SQL is also required.

Location: Nagpur/Pune/Chennai/Bangalore

Type of Employment: Full-time

Key Result Areas and Activities:

• Study existing technology landscape and understand current data integration framework and

do impact assessment for the requirements

• Develop/Enhance spark jobs using Scala for new/existing project requirements

• Create new data pipelines for developed/enhanced spark jobs using AWS Lambda or Apache

Airflow

• Analyze data processing, integration, modelling and reporting requirements & Define data

loading strategies considering volume, data types, frequency and analytics specifications

• Manage build phase and quality assure code to ensure fulfilling requirements and adhering to

Cloud architecture. Resolve difficult design and develop issues

Essential Skills:

• In depth knowledge of the following AWS Services required: S3, EC2, EMR, Athena, AWS Glue,

Lambda

• Big Data - Hadoop, Hive, Spark, YARN

• Programming Language – Scala, Python, Shell Scripts

• Operating System - Any flavor of Linux, Windows

• Strong SQL skills

• Expertise in developing ETL workflows comprising complex transformations like SCD,

deduplications, aggregations etc.

• Should have conceptual understanding of AWS VPC, Subnets, Security Groups, & Route Tables

Desirable Skills:

• Associate Level and Professional Level AWS Certification

• Cloud Databases - AWS Aurora, Singlestore, RedShift, Snowflake

• Orchestration Tools - Apache Airflow

Qualifications:

• Bachelor’s degree in computer science, engineering, or related field (Master’s degree is a plus)

• Demonstrated continued learning through one or more technical certifications or related

methods

• At least 5+ years of IT experience; minimum 3 years of experience on Cloud related projects

• 3+ years of AWS related project