Ather Energy Limited
AI Test Engineer
- Compensation
- Not disclosed
- Location
- Bengaluru, Karnataka, India
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- 1+ years
- Category
- Qa
- Posted
Skills
Must have
- Design and deployment of AI/LLM evaluation frameworks such as Ragas or Deepeval.
- Advanced test automation scripting using Python with libraries like PyTest, Selenium, or Playwright.
- In-depth understanding of LLM parameters, prompt engineering techniques, agent state machines, and JSON API payloads.
- Software testing methodologies, regression testing automation, and bug lifecycle management in Agile/Scrum environments using JIRA.
Nice to have
- Hands-on experience with CI/CD tools (GitLab CI, GitHub Actions) and containerized systems (Docker, Kubernetes).
- Cloud deployment knowledge on GCP alongside monitoring and logging tools like Grafana.
- Understanding of vector databases (Qdrant, Milvus) and ML tracking platforms (Langfuse, MLflow).
- Testing experience on web or mobile applications featuring integrated AI or conversational interfaces.
Full description
Role: AI Test Engineer (AI Applications & Agents)Location: IBC Knowledge Park, BangaloreReporting to: Senior AI engineerTeam name: Artificial Intelligence(DSAI), Digital Operation & EngineeringAbout the Team and the Role
This role is for an AI Test Engineer who will focus on the end-to-end validation of AI applications, Large Language Models (LLMs), and autonomous AI agents. You will be responsible for creating specialized validation frameworks for conversational systems, multi-agent workflows, and decision-making logic. By combining deep technical skills in AI testing methodologies, automation scripting, and system validation with a passion for quality, you will help ensure our AI agents operate safely, reliably, and accurately before deployment.
Responsibilities
- Design, develop, and execute automated test plans and validation scripts for AI-powered applications, LLMs, and autonomous AI agents.
- Formulate testing strategies covering model output accuracy, response relevance, hallucination detection, and prompt injection risks.
- Validate multi-agent system workflows, focusing on task routing, state transition logic, and pipeline error handling.
- Establish benchmarking frameworks to measure AI model accuracy, latency, and performance under stress and load conditions.
- Create mock environments, test datasets, and simulation frameworks to test edge cases and complex agent behaviors.
- Perform continuous regression testing within CI/CD pipelines as AI models are updated or fine-tuned.
- Identify, document, and isolate complex bugs, collaborating with Data Science, Dev, and Product teams to resolve root logic issues.
- Set up real-time automated reporting dashboards to track model quality and system reliability metrics.
Experience and Qualifications
- Bachelor’s degree in Computer Science, Software Engineering, or equivalent technical discipline.
- 1 to 3 years of experience in software engineering, testing or test automation, with at least 6 months – 1 year of experience specifically focused on validating AI agents, LLM systems, or conversational agents.
Must-Have Competencies
- Design and deployment of AI/LLM evaluation frameworks such as Ragas or Deepeval.
- Advanced test automation scripting using Python with libraries like PyTest, Selenium, or Playwright.
- In-depth understanding of LLM parameters, prompt engineering techniques, agent state machines, and JSON API payloads.
- Software testing methodologies, regression testing automation, and bug lifecycle management in Agile/Scrum environments using JIRA.