Senior AI QA Automation Engineer
Senior QA automation engineer who builds AI evaluation and test-automation frameworks (Python, Playwright/Selenium) for AI agents, RAG solutions, and prompts, implementing CI/CD quality gates and monitoring production AI quality. Remote role open to candidates in Costa Rica, Peru, Colombia, or Bolivia; requires 5+ years of experience.
- Compensation
- Not disclosed
- Location
- Bolivia
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Not specified
- Experience
- 5+ years
- Category
- Qa
- Posted
Skills
Must have
- 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering
- Strong Python skills with experience building reusable automated testing frameworks
- Hands-on experience with Playwright and/or Selenium
- Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation
- Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing
- Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds
- Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation
- Experience implementing CI/CD quality gates and production monitoring
- Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria
- Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related field
Nice to have
- Experience testing Generative AI, RAG solutions, copilots, or AI agents
- Experience with LLM evaluation frameworks and advanced prompt testing
- Experience developing agent-powered test generation, workflow validation, or intelligent UAT assistants
- Experience with adversarial testing and Responsible AI testing
- Experience monitoring AI quality, drift, reliability, and failure patterns in production environments
Full description
Job Title: Senior AI QA Automation Engineer
Key Skills: Python, AI Testing, Test Automation, Playwright, Selenium, API Testing, Azure AI Foundry Evaluations, RAG Evaluation, Prompt Testing, CI/CD, Monitoring
Experience: 5+ YOE.
Location: Costa Rica, Peru, Colombia, Bolivia.
Mode: Remote
We at Coforge are hiring Senior AI QA Automation Engineer (#006fu00000bWsOCAA0-4) with the following skill set.
Key Responsibilities
-
Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
-
Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
-
Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.
-
Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.
Required Skills & Qualifications
-
5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
-
Strong Python skills with experience building reusable automated testing frameworks.
-
Hands-on experience with Playwright and/or Selenium.
-
Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
-
Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
-
Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.
-
Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
-
Experience implementing CI/CD quality gates and production monitoring.
-
Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria.
-