Lead Quality Engineer

As a Senior Quality Engineer, you will own quality across AI-powered products, data pipelines, and agent-based systems. Your typical day will involve validating AI/LLM outputs against authoritative data sources, building and maintaining automation frameworks for data pipelines, and defining quality benchmarks for AI-generated content. You will focus on developing comprehensive testing strategies that align with business objectives, ensuring that quality is embedded throughout the application development process. Your role will also include monitoring key metrics to assess the effectiveness of testing efforts and driving continuous improvement initiatives within the team.

• Validate LLM and AI agent outputs against authoritative data sources, scoring for accuracy, completeness, and hallucination – and make production signoff decisions based on defined thresholds.
• Test end-to-end data pipelines across cloud-native architectures, covering schema validation, transformation logic, timelines, lineage, and cross-source consistency.
• Build and maintain automation frameworks spanning validation, anomaly detection, regression, and transformation modules – scalable across various product streams.
• Design and execute performance test packs for data agents and AI inference endpoints, establishing baselines and catching regressions before release.
• Own QA signoff cadence aligned to weekly releases, managing timelines from configuration fixes through new dataset onboarding, enforcing code freeze and versioning discipline.
• Validate compliance controls for AI products – information barriers, data redaction, monitored workflows, and audit trail completeness.
• Perform model comparison testing across LLM providers and versions, evaluating output quality shifts and informing go/no-go decisions for model upgrades.
• Coordinate UAT with product owners and business stakeholders, maintaining full traceability from requirements through to test execution evidence.
• Defect management and reporting.

• 5+ years of relevant quality engineering experience, with at least 2 years focused on data-intensive or AI/ML systems.
• Hands-on data pipeline testing experience (e.g., Snowflake, Databricks, Spark, SQL) – validation of transformations at scale, not just surface-level checks.
• Direct experience testing LLM/generative AI outputs – accuracy scoring, prompt-based evaluation, and hallucination detection.
• Python for building automation frameworks – not just individual scripts but maintainable, modular test infrastructure.
• Git-based workflows, CI/CD integration, and automation regression suites.
• Cloud platform experience (e.g., Azure, Databricks, Kubernetes) and API testing (REST, streaming/real-time APIs).
• Advanced communication skills: ability to effectively communicate with stakeholders, present findings, and write functional/technical documentation.
• Critical and analytical thinking: capability to analyze complex problems, prioritize requirements, and propose innovative solutions.

• Exposure to retrieval-augmented generation (RAG) patterns and multi-agent systems – including an understanding of how to validate tool-calling behavior, reasoning chains, and context window handling.
• Familiarity with compliance requirements – information barriers, data classification, and audit trail requirements.
• Working knowledge of test management and project management tools (e.g., qTest, Jira, Confluence).
• Experience with data profiling, anomaly detection frameworks, and statistical validation of datasets.
• Understanding of agile frameworks (Scrum, Kanban) and related project management practices.
• Stakeholder management: ability to identify, analyze, and manage the expectations and requirements of stakeholders. Establish and maintain positive relationships with stakeholders to ensure their support and cooperation.