AI QA Engineer

Quick Take

As an AI QA Engineer, you’ll drive the quality, reliability, and safety of the AI solutions we deliver to our Global 2000 clients, including GenAI and agentic applications, RAG pipelines, and ML models used across our products. This role is for an engineer with a solid data science/AI background who understands how models behave and is skilled in LLM evaluation, end-to-end AI testing, and ML model validation, helping us ship production-grade AI with confidence.

The Work

· Own the test strategy for our AI products, covering LLM features, agentic workflows, RAG pipelines, and ML models.

· Write and execute test plans, cases, and reports; manage the defect lifecycle; and perform exploratory testing of AI features.

· Define acceptance criteria and quality metrics for non-deterministic outputs (accuracy, groundedness, hallucination rate, relevance, cost) using LLM-as-a-judge and human evaluation.

· Build and maintain golden datasets, ground-truth labels, and regression suites with subject-matter experts, reflecting real scenarios and edge cases.

· Develop automated evaluation pipelines (DeepEval, Ragas, promptfoo, LangSmith) that run in CI/CD on every prompt, model, retrieval, or code change.

· Test agentic behavior—multi-step planning, tool calling, integrations, error handling, and fallbacks—to verify agents’ complete tasks correctly and stay within guardrails.

· Validate ML models and data pipelines for data quality, segment performance, bias/fairness, and drift, using statistical methods to compare versions.

· Perform adversarial and safety testing: prompt injection, jailbreaks, PII leakage, and compliance checks.

· Monitor production AI quality through observability tooling, triage degradations, lead root-cause analysis, and partner with Data Science, AI, and engineering teams on fixes and release sign-off.

The Must-Haves

· B.S./M.S. in Computer Science, Data Science, ML/AI, Statistics, or related field.

· 2–3 years in AI/ML, data science, or AI-focused QA, including 1+ year of hands-on test automation, ideally for production ML systems.

· Working knowledge of ML model building, evaluation metrics, and LLM behavior (prompting, context, RAG).

· Experience evaluating GenAI applications (RAG pipelines, agentic workflows, function calling) with DeepEval, Ragas, or similar.

· Strong Python with PyTest, API testing, and CI/CD integration.

· Grounding in statistics and experimentation (sampling, significance testing, model A/B comparison), plus SQL and data validation tools such as Great Expectations or Evidently.