NBBL- Lead Innovation

About the Role

We are seeking a highly skilled Lead Data Engineer to design and implement the data foundation for the Innovation vertical within BBPS. This role focuses on building scalable, high-performance data pipelines and data products that enable AI/ML-driven solutions and real-time analytics for BBPS and the broader ecosystem.

Responsibilities

• Pipeline Architecture: Design, build and optimize high-scale data pipelines (Batch & Streaming) using Spark/PySpark, Kafka, and Airflow/similar on prem tools.
• Ecosystem Innovation: Develop data products that provide real-time insights to entire ecosystem such as transaction success-rate predictors and automated reconciliation engines and so on.
• Data Lakehouse Management: Build and maintain a scalable Data Lakehouse to consolidate disparate data formats from across the ecosystem into a unified source of truth.
• Orchestration: Utilize enterprise level schedulers like Apache Airflow/Dagster and to create models using build tools like DBT, to automate complex workflows and ensure data quality validation across the pipeline.
• Performance Engineering: Optimize complex SQL queries and data storage formats to ensure sub-second latency for innovation-led dashboards.
• Security & Compliance: Ensure all data engineering practices strictly adhere to NPCI/RBI guidelines, including PII masking and data encryption.

Preferred skills and Qualifications:

• Bachelor’s/master’s degree in computer science, Data Science or related field.
• 6+ years of experience in software development, with at least 2 years in AI/ML projects.
• Certifications in Data Engineering/ AI/ML technologies are a plus.
• Proficiency in Python and PySpark with strong experience in Spark using Java or Scala for advanced data engineering and scalable pipeline development.
• Hands-on experience with AI/ML frameworks (TensorFlow, PyTorch, Scikit-learn).
• Good Knowledge on building and managing Data pipelines.
• Streaming: Proven track requires real-time processing using Kafka or Flink.
• Orchestration: Hands-on experience with Airflow or similar workflow tools.
• Knowledge of vector databases and RAG pipelines.
• Experience with containerization (Docker) and orchestration (Kubernetes) is a big plus.
• Familiarity with open-source AI tools and libraries.
• Good understanding of security and compliance for enterprise applications.
• Excellent problem-solving and debugging skills.
• Ability to work collaboratively in agile teams and aid in building auto scalable pipelines.

Equal Opportunity Statement:

As an Equal Opportunity Employer, NBBL fosters a workplace culture that values diversity and inclusion. It maintains zero tolerance for discrimination based on race, ethnicity, disability, gender identity, or sexual orientation, and actively supports the LGBTQ+ community.