Senior Software Development Engineer in Test - AI Test

We are looking for an experienced SDET - AI Testing to drive quality engineering initiatives for GenAI and Agentic AI applications. The ideal candidate should have strong expertise in traditional test automation along with hands-on experience evaluating Large Language Models (LLMs), AI Agents, RAG systems, and conversational AI platforms. You will be responsible for building scalable AI testing frameworks, defining evaluation strategies, automating AI quality checks, and ensuring reliability, accuracy, safety, and performance of AI-powered products.

The candidate will have responsibilities across the following functions: AI Testing and Evaluation:

• Design and execute testing strategies for LLM, GenAI, RAG, and Agentic AI systems.
• Build automated evaluation pipelines for AI applications using DeepEval, OpenAI Evals, and custom evaluation frameworks.
• Define and monitor AI quality metrics including: Faithfulness, Hallucination Detection, Context Relevance, Answer Relevancy, Toxicity, Bias & Safety.

Latency and Cost:

• Create golden datasets and benchmark suites for continuous model evaluation.
• Validate AI agent workflows involving planning, reasoning, memory, and tool usage.

Automation and Quality Engineering:

• Develop robust automated test frameworks for APIs, backend services, and AI applications.
• Build CI/CD-integrated testing pipelines for AI systems.
• Automate regression, integration, performance, and end-to-end testing.
• Design synthetic test data and evaluation datasets for AI workflows.
• Implement observability and monitoring mechanisms for AI applications in production.

Agentic AI Validation:

• Test multi-agent and autonomous workflows.
• Validate tool-calling, function-calling, memory management, and orchestration logic.
• Assess agent reliability, consistency, and failure recovery mechanisms.
• Evaluate prompt effectiveness and prompt regression issues.

Performance and Reliability:

• Conduct load and stress testing for AI-powered systems.
• Measure response quality, latency, throughput, and cost efficiency.
• Perform root cause analysis for AI failures and model performance degradation.

The core requirements for the job include the following:

AI Testing and GenAI:

• Hands-on experience testing LLM and Generative AI applications.
• Strong experience with: DeepEval, OpenAI Evals, RAG Evaluation Frameworks, Prompt Testing, Agent Evaluation Frameworks.
• Understanding of: LLM architectures, RAG systems.

Embeddings and Vector Databases: