Data Engineer

Job Description

We are seeking a highly experienced and passionate Data Engineer to join our growing data team. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining our robust data infrastructure, ensuring data quality, availability, and scalability. You will play a critical role in empowering our data-driven decision-making by building and optimizing data pipelines, ETL processes, and data warehousing solutions. This is a senior-level position requiring extensive experience with database technologies, data modelling, and cloud platforms. Responsibilities Data Pipeline Development & Maintenance: Design, build, and maintain end-to-end data pipelines for ingesting, processing, and transforming large datasets from various sources (e.g., relational databases, APIs, flat files, streaming data). ETL/ELT Process Optimization: Develop and optimize ETL/ELT processes using industry-standard tools and techniques, ensuring data accuracy, efficiency, and scalability. Database Design & Management: Design, implement, and manage relational databases, including schema design, indexing, performance tuning, and data governance. Data Warehousing: Design and implement data warehousing solutions to support business intelligence and reporting needs. Experience with star schema, snowflake schema, and other data modelling techniques is essential. Database Administration: Perform database administration tasks, including performance monitoring, capacity planning, backup and recovery, and security management. Data Quality & Governance: Implement data quality checks, validation rules, and data governance policies to ensure data accuracy and consistency. Cloud Platform Expertise: Leverage cloud platforms (e.g., AWS, Azure, GCP) for data storage, processing, and management. Collaboration & Communication: Collaborate with cross-functional teams (e.g., data scientists, business analysts, software engineers) to understand data requirements and deliver effective data solutions. Clearly communicate technical concepts to both technical and non-technical audiences. Mentoring & Knowledge Sharing: Mentor junior engineers and share knowledge and best practices within the team. Automation & Scripting: Automate data engineering tasks using scripting languages (e.g., Python, Bash). Stay Up to Date: Continuously research and evaluate new data technologies and techniques to improve our data infrastructure. Production Support: Monitoring the batch/jobs daily, no matter it?s weekdays or weekend. Production Release: Actively participation in release process. Qualifications Experience: Minimum of 6 years of experience as a Data Engineer or in a similar role. Database Expertise: Expert proficiency in SQL Server: Extensive experience with SQL Server, including database design, performance tuning, query optimization, and database administration. Expert proficiency in PostgreSQL: Deep understanding of PostgreSQL, including database design, performance tuning, query optimization, and database administration. Solid experience with other relational databases like MySQL, Oracle, etc. is a plus. ETL/ELT Tools: Proven experience with ETL/ELT tools (e.g., Apache Airflow, Informatica, Talend, SSIS, ADF etc.). Data Modelling: Strong understanding of data modelling principles and techniques (e.g., dimensional modelling, star schema, snowflake schema). Cloud Computing: Experience with cloud platforms (AWS, Azure, or GCP) and related data services (e.g., S3, Redshift, Snowflake, Azure Data Lake Storage). Programming & Scripting: Proficiency in scripting languages such as Python, Bash, or similar. Data Governance & Quality: Experience implementing data quality checks, data governance policies, and data validation rules. Problem-Solving & Analytical Skills: Excellent problem-solving and analytical skills with the ability to identify and resolve complex data-related issues. Communication & Collaboration: Excellent communication, collaboration, and interpersonal skills. Ability to work effectively in a team environment. Education: Bachelor?s or master?s degree in computer science, Information Technology, or a related field. Bonus Points Experience with data streaming technologies (e.g., Kafka, Spark Streaming). Experience with data visualization tools (e.g., Tableau, Power BI, Looker). Relevant certifications (e.g., AWS Certified Data Engineer, Microsoft Certified: Azure Data Engineer Associate).