GCP Data Engineer (Kafka + Java)

Job Description

Overview Overview: We are seeking a talented and motivated GCP Data Engineer with experience in real-time streaming technologies, specifically Apache Kafka, and proficiency in Java. The ideal candidate will be responsible for designing, developing, and maintaining data pipelines and streaming applications in a cloud environment. Responsibilities Key Responsibilities: Data Pipeline Development: Design and implement robust data pipelines using Google Cloud services (BigQuery, Dataflow, Pub/Sub) integrated with Kafka for real-time data processing. Streaming Applications: Develop and maintain real-time streaming applications using Kafka and Java, ensuring high availability and performance. Cloud Infrastructure: Utilize GCP technologies to build scalable and efficient data architectures, leveraging tools such as Cloud Storage, Cloud Functions, and Kubernetes. Monitoring and Optimization: Monitor data flow, performance metrics, and troubleshoot issues in production environments. Optimize data pipeline performance and reliability. Collaboration: Work closely with data scientists, analysts, and other engineers to understand data requirements and translate them into technical specifications. Documentation: Maintain comprehensive documentation of data pipelines, architecture, and processes to ensure knowledge sharing and compliance. Requirements Required Skills: Cloud Skills: Proficiency in Google Cloud Platform (GCP), including services like BigQuery, Dataflow, and Pub/Sub. Streaming Technologies: Strong experience with Kafka for real-time data streaming and messaging. Programming Language: Proficient in Java, with a solid understanding of object-oriented programming principles and best practices. Data Management: Familiarity with data modeling, ETL processes, and database management. Development Tools: Experience with version control systems (e.g., Git), CI/CD tools, and Agile methodologies. Problem-Solving: Strong analytical and troubleshooting skills, with the ability to work independently and as part of a team. Preferred Qualifications: Bachelor's degree in Computer Science, Information Technology, or a related field. Experience with other programming languages (Python, Scala) is a plus. Knowledge of additional GCP services or data visualization tools (e.g., Looker, Tableau) is advantageous.