Senior Data Engineer (GCP)

About the Role:

Pixeldust Technologies is looking for a highly skilled Senior Data Engineer to join our growing Data & AI team. You will be responsible for designing, building, and optimizing scalable data platforms, real-time data pipelines, and cloud-native data engineering solutions on Google Cloud Platform (GCP).

The ideal candidate should have strong expertise in Python, PySpark, SQL, GCP Data Engineering services, and modern microservices architecture. Experience with streaming pipelines, Graph Databases, and Agentic AI frameworks will be an added advantage.

This role offers the opportunity to work on cutting-edge AI, Analytics, and Data Engineering projects involving large-scale datasets and real-time processing.

Key Responsibilities:

Data Engineering & Architecture:

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Build robust ETL/ELT pipelines using Python, PySpark, and SQL.
• Develop high-performance data processing frameworks for structured and unstructured datasets.
• Design scalable cloud-native data architectures on Google Cloud Platform.

Google Cloud Platform:

Develop and manage solutions using:

• BigQuery
• Cloud Storage (GCS)
• Cloud Run
• Pub/Sub
• Dataflow
• Cloud Functions
• Cloud Scheduler
• Secret Manager
• IAM
• Cloud Monitoring
• Logging
• Artifact Registry

Optimize cloud infrastructure for scalability, reliability, and cost efficiency.

Streaming & Data Pipelines:

• Build streaming data pipelines using Pub/Sub, Kafka, and Dataflow.
• Develop batch ingestion pipelines from multiple enterprise data sources.
• Optimize pipeline performance and reduce execution time.
• Implement monitoring, retry mechanisms, and error handling.

Microservices Development:

Develop REST APIs and data services using:

• FastAPI
• Flask

Build scalable microservices supporting AI and Data Engineering workflows.

Data Modeling & Warehousing:

• Design dimensional and normalized data models.
• Build enterprise-grade data warehouses in BigQuery.
• Optimize partitioning, clustering, indexing, and query performance.
• Implement metadata management and data governance practices.

Performance Optimization:

• Optimize SQL queries and PySpark jobs.
• Improve data pipeline performance and throughput.
• Drive cloud cost optimization initiatives.
• Implement monitoring and alerting for production workloads.

AI & Emerging Technologies (Preferred)

Exposure to:

• Neo4j Graph Database
• Knowledge Graphs
• Model Context Protocol (MCP)
• Agentic AI
• Retrieval-Augmented Generation (RAG)
• Vector Databases

Experience working on AI-powered data engineering solutions is a plus.

DevOps & Deployment:

• Build CI/CD pipelines using GitHub Actions.
• Implement Infrastructure as Code (Terraform preferred).
• Deploy containerized applications using Docker and Cloud Run.
• Follow DevOps best practices for automated testing and deployments.

Leadership & Collaboration:

• Mentor junior engineers and conduct code reviews.
• Collaborate with Product Managers, Data Scientists, AI Engineers, and Architects.
• Participate in Agile ceremonies including sprint planning, stand-ups, retrospectives, and backlog refinement.
• Drive engineering best practices and technical excellence.

Required Skills:

Programming:

• Python (Expert)
• SQL (Expert)
• PySpark (Expert)

Google Cloud Platform

Strong hands-on experience with:

• BigQuery
• Cloud Run
• Cloud Storage
• Pub/Sub
• Dataflow
• Cloud Functions
• Cloud Scheduler
• IAM
• Monitoring & Logging

Frameworks:

• FastAPI
• Flask

Streaming Technologies:

• Kafka
• Pub/Sub
• Dataflow

Data Engineering:

• ETL/ELT Pipelines
• Batch Processing
• Streaming Pipelines
• Data Warehousing
• Data Modeling
• Data Integration

DevOps:

• Git
• GitHub Actions
• Docker
• CI/CD
• Deployment Strategies

Preferred Skills:

• Neo4j
• Knowledge Graphs
• MCP (Model Context Protocol)
• Agentic AI
• Vector Databases
• RAG Architecture
• Terraform
• Kubernetes
• Cloud Run
• AI/ML Pipelines

Qualifications:

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• 6+ years of experience in Data Engineering.
• Experience delivering enterprise-scale data platforms on GCP.

What We're Looking For:

We're looking for someone who:

• Thinks like a software engineer and an architect.
• Enjoys solving complex data engineering problems.
• Has experience building scalable cloud-native systems.
• Takes ownership and drives projects independently.
• Mentors teammates and promotes engineering excellence.
• Is passionate about modern AI, Data Engineering, and Cloud technologies.

Why Join Pixeldust Technologies?

• 🚀 Work on cutting-edge AI, Data Engineering, and GenAI projects.
• ☁️ Build cloud-native solutions using the latest GCP technologies.
• 📈 Opportunity to work on enterprise-scale data platforms.
• 🤝 Collaborative and innovation-driven work culture.
• 📚 Continuous learning and exposure to emerging technologies like Agentic AI, Knowledge Graphs, and MCP.
• 💡 High-impact role with opportunities for technical leadership and career growth.