Quality Engineer

Hiring: QA/Evaluation Engineer – LLM & Generative AI

📍 Location: Gurugram (On-site)

💼 Experience: 4+ Years

🕒 Employment Type: Full-time

About the Role

We are looking for an experienced QA/Evaluation Engineer to join our AI team and build robust evaluation frameworks for Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) systems, and Agentic AI applications.

In this role, you will be responsible for defining and maintaining the quality standards of AI systems by designing evaluation pipelines, validating model outputs, improving retrieval quality, and ensuring responses are accurate, grounded, safe, and production-ready.

Key Responsibilities

• Build and maintain evaluation frameworks for LLMs, RAG systems, and AI agents.
• Develop automated evaluation suites using frameworks such as Ragas, DeepEval, Promptfoo, or custom evaluation harnesses.
• Design and execute retrieval evaluation using metrics like precision, recall, answer relevance, groundedness, and faithfulness.
• Evaluate end-to-end AI agent pipelines and multi-agent workflows.
• Design labeled datasets, golden datasets, and benchmark evaluation suites.
• Validate model performance before and after fine-tuning (SFT & LoRA).
• Perform regression testing, drift detection, and threshold calibration.
• Implement AI guardrails for hallucination detection, defamation prevention, and safety validation.
• Conduct AI red-teaming and structured output validation.
• Integrate evaluation pipelines into CI/CD workflows.
• Collaborate with AI Engineers, ML Engineers, and Knowledge Management teams to continuously improve AI quality.

Required Skills

• 4+ years of experience in QA, AI/ML Quality, ML Evaluation, or Applied Machine Learning.
• Strong proficiency in Python.
• Hands-on experience with LLMs, Generative AI, and RAG applications.
• Experience with evaluation frameworks such as Ragas, DeepEval, Promptfoo, DSPy, GEPA, or MIPROv2.
• Strong understanding of LLM-as-a-Judge, groundedness, faithfulness, hallucination detection, and rubric-based evaluation.
• Experience evaluating AI agents and end-to-end agentic workflows.
• Knowledge of SFT and LoRA model validation.
• Experience with regression testing, drift detection, statistical analysis, and threshold calibration.
• Familiarity with AI guardrails, red-teaming, and structured output validation.
• Experience integrating evaluation frameworks into CI/CD pipelines.

Preferred Qualifications

• Experience working on production-grade LLM applications.
• Familiarity with LangChain, LangGraph, LlamaIndex, or similar AI orchestration frameworks.
• Experience in high-trust domains such as Finance, Healthcare, Legal, or Consulting.
• Knowledge of vector databases and retrieval optimization techniques.

Why Join Us?

• Work on cutting-edge Generative AI and Agentic AI solutions.
• Build evaluation systems for enterprise-scale LLM applications.
• Collaborate with highly skilled AI and Machine Learning professionals.
• Contribute to production-ready AI systems used in real-world, high-impact environments.