Fullstack AI Quality Engineer – Applied & Agentic AI Systems
- Compensation
- Not disclosed
- Location
- Hyderabad
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- 1+ years
- Category
- Fullstack
- Posted
Skills
Must have
- Hold B.Sc., B.Eng., M.Sc., M.Eng. in Computer Science, Software Engineering, or equivalent degree with strong background in testing methodologies.
- Be passionate about quality assurance and testing, particularly in the context of AI and GenAI applications and agents.
- Have strong analytical and problem-solving skills with attention to detail.
- Utilize critical thinking and problem-solving when using AI prompts (in e.g. NLP, data distributions and edge cases).
- Be proactive in identifying potential issues and suggesting improvements.
- Comprehensive Full-Stack Testing Mastery
- Core Languages & Testing Frameworks: Advanced command of Python and standard testing frameworks (e.g., pytest, allure), alongside practical familiarity with specialized AI evaluation tools. High compe
- End-to-End Automation & CI/CD: Proven expertise in architecting and maintaining end-to-end automated testing suites that bridge traditional UI/API validation with complex, asynchronous AI/LLM evaluati
- Pipeline Integration: Hands-on experience integrating these comprehensive suites within GitHub Actions or modern CI/CD pipelines to ensure continuous, reliable quality checks throughout the entire dev
- Advanced AI Quality & Evaluation Mastery
- AI Evaluation Frameworks: Expertise in implementing evaluation harnesses (e.g., RAGAS, DeepEval, Promptfoo) to quantify LLM performance, hallucination rates, and answer relevancy.
- Agentic Workflow Validation: Ability to test autonomous agentic loops, including tool-use accuracy, step-by-step reasoning verification, and goal-directed performance metrics.
Full description
At Roche you can show up as yourself, embraced for the unique qualities you bring. Our culture encourages personal expression, open dialogue, and genuine connections, where you are valued, accepted and respected for who you are, allowing you to thrive both personally and professionally. This is how we aim to prevent, stop and cure diseases and ensure everyone has access to healthcare today and for generations to come. Join Roche, where every voice matters.
##
The Position
External Job Ad Text
Job description
We are seeking a highly skilled AI Quality Engineer specializing in testing Generative (and non-Generative) AI applications and agentic systems to join our team. The ideal candidate will have expertise in both traditional software testing and specialized testing methodologies for Large Language Models (LLMs)-based and AI-based systems.
As a fullstack Quality Engineer within a lean, agentic software team, you will cover the entire spectrum of quality assurance. You will span manual exploratory testing, traditional automated UI/API/MCP testing (across our Python and TypeScript stack), classical machine learning validation, and cutting-edge generative and agentic AI evaluation. This role requires a deep understanding of LLM, agentic and ML systems behaviors, and testing frameworks to ensure the reliability, fairness, and effectiveness of our AI-powered solutions.
Description of the area
At Roche Digital Technology, we are advancing the boundaries of Applied AI. The Applied AI Engineering Team focuses on architecting, building, and operating high-value AI solutions and services to solve complex business challenges in healthcare.
In the 2026 tech landscape, we operate in small, highly autonomous agile teams (e.g., ~9 members) powered by advanced coding agents (like Claude Code) to develop and ship solutions faster than ever before. In this highly regulated environment, quality cannot be an afterthought. You will be the foundational pillar ensuring our rapidly developed agentic workflows and AI, GenAI, and agentic applications are safe, compliant, and robust before they reach the clinical or enterprise user.
Job Responsibilities
- Generative & Agentic AI Testing Strategy: Design and implement comprehensive testing strategies specifically tailored for LLM-based applications and agentic systems, including evaluation of model responses, RAG pipelines accuracy, and overall system reliability. You will expand this to evaluate autonomous multi-agent behaviors and tool-use accuracy.
- Full-Stack Test Automation: Design and implement automated testing solutions for continuous evaluation of LLM applications, including integration tests, performance tests, and specialized AI behavior tests. This spans both our Python backend services and modern TypeScript/React/Angular frontends.