AI QA Engineer

Job Summary

We are seeking an AI QA Engineer to ensure the quality, accuracy, and performance of our enterprise-grade Natural Language to SQL (NL2SQL) pipeline. You will be responsible for validating a complex, multi-stage AI architecture—including semantic routing, LLM-based disambiguation, and query generation—ensuring it securely and accurately translates user intent into valid queries within the BFSI domain.

Experience: 7+ Years

Location: Gurugram

Work Mode: Hybrid - 3 Days WFO

Employment Type: Full-Time

Key Responsibilities

• LLM & Pipeline Evaluation: Design and execute automated evaluations for a 4-stage NL2SQL pipeline using LangSmith. Monitor metrics such as structural F1, execution accuracy, latency, and token cost.
• Dataset Management: Create, curate, and maintain benchmark/golden datasets for continuous regression testing of LLM prompts and model outputs.
• Search & Retrieval Testing: Validate precision and recall trade-offs in semantic search and schema discovery, ensuring optimal candidate selection for downstream query generation.
• Failure Analysis & Debugging: Perform root cause analysis across pipeline stages (routing, disambiguation, query generation, execution), identifying issues such as schema mismatches, type/coercion errors, runtime incompatibilities, and query structure failures.
• E2E & API Automation: Develop automated test scripts using Python (Pytest) for backend API testing and Playwright for the React frontend, validating end-to-end user workflows.
• Observability & Debugging: Utilize Grafana and structured JSONL logs to identify pipeline bottlenecks, LLM hallucinations, or prompt degradation.
• Compliance & Security: Ensure the AI pipeline meets strict BFSI data security standards, validating execution safety mechanisms (e.g., runtime capability probing, injection prevention); Ability to design validation rules and guardrails for AI pipelines to prevent invalid query generation and runtime failures.

Required Skills

• AI/LLM Testing: Experience testing LLM applications, RAG (Retrieval-Augmented Generation) pipelines, or NLP models. Familiarity with AI evaluation frameworks (e.g., LangSmith, DeepEval, or similar).
• Languages: Strong proficiency in Python 3.12+ (crucial for integrating with the existing AI backend and Pytest suite). Secondary experience with JavaScript/TypeScript.
• Test Automation: Expertise in API testing (REST) and optional UI automation using Playwright.
• Data & Search: Understanding of Vector Databases (e.g., Milvus, Pinecone) and semantic search concepts (embeddings, hybrid search).
• Data & SQL Validation: Solid understanding of SQL and data validation techniques to verify correctness of complex query outputs.
• Tools & Infrastructure: Git, Docker, CI/CD pipelines, and observability tools (Prometheus/Grafana).

Education

• BE / BTech / MCA / BSc in Computer Science, Data Science, or a related field.

Nice to Have

• Familiarity with Graph Databases (Neo4j) and LangGraph orchestration.
• Experience evaluating foundational LLM models (OpenAI, Anthropic, Google).
• Prior exposure to query languages like SQL or PURE or any other functional programming language.
• Experience testing workflows across multiple services or pipelines, with an understanding of failure handling, retries, and system reliability concepts.
• Experience in Banking, Financial Services, or Insurance domains
• Understanding of data security, compliance, and enterprise database schemas