Software Quality Assurance Engineer

Job Details

• Location: Gurugram
• Work Mode: Hybrid (3 days WFO)
• Experience: 7+ Years
• Employment Type: Full-Time

What You’ll Work On

• Design and execute LLM evaluation frameworks (LangSmith, DeepEval, etc.)
• Measure and improve metrics like F1 score, execution accuracy, latency, token cost
• Build and maintain golden datasets for AI regression testing
• Validate semantic search & retrieval systems (precision/recall, embeddings)
• Perform deep failure analysis across AI pipelines
• Develop API automation (Python + Pytest) and UI automation (Playwright)
• Monitor systems using Grafana & structured logs to detect hallucinations and bottlenecks
• Implement AI guardrails, validation rules & security checks for safe query execution

Tech Stack & Skills

• AI/LLM Testing: RAG pipelines, NLP systems, evaluation frameworks
• Languages: Python (must-have), JavaScript/TypeScript (good to have)
• Automation: REST API testing, Pytest, Playwright
• Data & Search: Vector DBs (Pinecone, Milvus), embeddings, hybrid search
• SQL: Strong query validation & data verification skills
• Tools: Git, Docker, CI/CD, Grafana/Prometheus

Domain Expertise (Preferred)

• BFSI (Banking, Financial Services, Insurance)
• Data security, compliance, enterprise-grade systems

Nice to Have

• Experience with LangGraph, Neo4j, or graph-based systems
• Exposure to OpenAI / Anthropic / Google LLMs
• Understanding of multi-service architectures & system reliability