Senior AI Platform Engineer – Eval / QA / Model Governance / FinOps

Gurgaon | Hybrid

About neurogent.ai 

neurogent.ai builds agentic AI solutions that help organizations automate workflows and improve customer experience across Banking, Healthcare, Insurance, and Financial Services. We develop intelligent AI agents, chatbots, and voice assistants that provide 24/7 support, streamline operations, and drive measurable efficiency gains. Using modern architectures such as RAG and LangGraph, we help teams reduce costs, improve productivity, and accelerate growth.

The Role 

We are seeking a Senior AI Platform Engineer to own the quality, safety, and cost governance layer of an enterprise Agentic AI Platform built on Microsoft Foundry. This specialist role spans four interconnected disciplines: agent evaluation & QA, model governance, responsible AI enforcement, and AI FinOps. You will build the frameworks, pipelines, and dashboards that ensure every agent deployed to production meets defined standards for accuracy, safety, compliance, and cost efficiency.

Key Responsibilities

Evaluation & QA

• Design and deploy the end-to-end agent evaluation framework using Microsoft Foundry Eval SDK and custom harnesses
• Define evaluation templates for quality (coherence, fluency, relevance), RAG performance (groundedness, completeness, retrieval accuracy), and safety (jailbreak resistance, PII leakage, content policy)
• Build an automated quality gate that blocks agent promotion to production if evaluation scores fall below defined thresholds
• Operate the AI Red Teaming Agent with 20+ attack strategies; track Attack Success Rate (ASR) per agent
• Configure nightly and continuous evaluation runs on sampled live production traffic; monitor groundedness score drift, safety regressions, and hallucination rates
• Implement A/B testing pipelines for prompt and model changes with checkpointing and rollback capability

Model Governance

• Build and maintain a versioned prompt registry with auto-promote/demote policies, change detection, and audit trails
• Maintain the approved model catalog (Azure OpenAI models, OSS models via Managed Online Endpoints)
• Implement model watchdog - automated checks for model deprecation, version drift, and fallback orchestration
• Configure and tune Foundry Guardrails across 12 risk categories and 4 intervention points
• Build ML Continuous Testing pipelines that validate model behavior on golden datasets before and after model or prompt updates

FinOps

• Implement per-agent, per-project, per-department cost tracking using APIM custom dimensions + App Insights + KQL queries
• Manage PTU (Provisioned Throughput Unit) allocation across projects; implement PAYG overflow policies
• Build chargeback/showback reports via Power BI or Azure Monitor Workbooks
• Define and enforce token rate limits per project; establish monthly FinOps review cadence

Technical Requirements

• Evaluation & Testing: LLM evaluation methodology, experimental design, statistical significance testing. Hands-on with Microsoft Foundry Eval SDK, DeepEval, RAGAS, or custom harnesses. Adversarial/red-team testing for LLMs
• Prompt Engineering: Prompt versioning, registry management, optimization techniques, and auto-rollback patterns
• Model Governance: Model catalog management, versioning, deprecation tracking, fallback orchestration, Azure OpenAI model lifecycle
• Responsible AI: Content safety frameworks (Foundry Guardrails or equivalent), PII detection, task adherence, prompt injection defense
• FinOps: Azure Cost Management, APIM token metrics, KQL queries, Power BI or Azure Monitor Workbooks, PTU vs PAYG optimization
• AI Platform: Microsoft Foundry (Agent Service, Control Plane, Guardrails, Eval), Azure OpenAI, Azure AI Search
• DevOps / CI: Azure DevOps Pipelines, Python scripting, CI integration for eval runs and prompt validation gates

What We're Looking For

• 3+ years in software engineering or ML engineering, with at least 2 years focused on LLM/AI platform quality, evaluation, or MLOps
• Has designed and operated an LLM evaluation harness in production - including defining thresholds, building quality gates, and preventing defective deployments
• Has implemented cost attribution for AI/ML workloads with chargeback reporting to business stakeholders
• Hands-on experience configuring content safety filters, PII detection, or prompt injection defenses for production AI systems
• Strong foundation in statistical analysis of evaluation results, drift detection, and anomaly identification
• Certifications preferred: Microsoft Certified Azure AI App and Agent Developer Associate (AI-103), FinOps Certified Practitioner, or equivalent