Senior Software Development Test Engineer (SDET), AI & Automation
Senior SDET embedded in Keeper Security's AI team, building automated test frameworks and CI/CD quality gates for AI-powered products. Day to day: define eval criteria for non-deterministic LLM outputs, build regression/API/E2E automation, and track quality using Python, Java, JS/TS or Rust with GitHub Actions, Jenkins, or GitLab CI.
- Compensation
- Not disclosed
- Location
- Remote, US
- Work setup
- Remote
- Seniority
- Senior
- Employment
- Full time
- Experience
- 5+ years
- Category
- Qa
- Posted
Skills
Must have
- 5+ years of experience in SDET, software development, test automation or another code-intensive quality engineering role
- Hands-on experience testing AI/ML, LLM-powered or model-integrated software systems
- Strong understanding of the differences between deterministic software testing and non-deterministic AI/ML validation
- Experience defining evaluation criteria, expected behaviors, tolerances or scoring approaches for AI-driven functionality
- Strong programming skills in Python, Java, JavaScript/TypeScript, Rust or another general-purpose programming language
- Significant experience designing, developing and maintaining automated test frameworks
- Experience building automated functional, integration, API and end-to-end test coverage
- Experience testing REST APIs and distributed application workflows using tools such as Postman, REST Assured or similar
- Experience integrating automated tests into CI/CD pipelines using GitHub Actions, Jenkins, GitLab CI or similar
- Experience working within an Agile software development team with close collaboration between QA, Engineering and Product
- Strong understanding of shift-left testing, continuous quality and testability principles
- Experience with TestRail or similar test management platforms
- Strong debugging and troubleshooting skills using logs, APIs, application telemetry and development tools
- Experience with Git-based development workflows, code review and modern software engineering practices
- Familiarity with AI-assisted development and testing tools such as Claude, ChatGPT, GitHub Copilot or similar
Nice to have
- Experience testing LLMs, agentic systems, retrieval-augmented generation, tool calling or prompt-driven workflows
- Experience building automated model evaluation, benchmarking or regression frameworks
- Familiarity with AI evaluation metrics and techniques for measuring quality across non-deterministic outputs
- Experience with Rust in production or test automation environments
- Experience with performance, reliability, security or adversarial testing of AI-enabled systems
- Experience with AI-assisted generation of automated tests or synthetic test data
- Familiarity with Docker, Kubernetes or cloud-based testing environments
- Experience testing cybersecurity, identity, PAM or other security-sensitive software
- Experience building shared automation frameworks or internal testing tools used across engineering teams
Full description
Description
Keeper Security is hiring a Senior Software Development Engineer in Test (SDET), AI & Automation to join our AI team and help build a modern, automated approach to quality across AI-powered products and services. This is a 100% remote position, with an opportunity to work a hybrid schedule for candidates based in the El Dorado Hills, CA or Chicago, IL metro areas.
Keeper’s cybersecurity software is trusted by millions of people and thousands of organizations globally. Keeper is published in 23 languages and sold in over 150 countries. Join one of the fastest-growing cybersecurity companies and help establish scalable quality engineering practices for AI-driven systems operating in security-sensitive environments.
About Keeper
Keeper Security is one of the fastest-growing cybersecurity software companies that protects thousands of organizations and millions of people in over 150 countries. Keeper is a pioneer of zero-knowledge and zero-trust security built for any IT environment. Its core offering, KeeperPAM®, is an AI-enabled, cloud-native platform that protects all users, devices and infrastructure from cyber attacks. Recognized for its innovation in the Gartner Magic Quadrant for Privileged Access Management (PAM), Keeper secures passwords and passkeys, infrastructure secrets, remote connections and endpoints with role-based enforcement policies, least privilege and just-in-time access. Learn why Keeper is trusted by leading organizations to defend against modern adversaries at KeeperSecurity.com.
About the Job
As a Senior SDET, AI & Automation, you will serve as an embedded quality engineering partner within Keeper’s AI team. You will work alongside engineers and Product throughout design and development to define test strategy, establish acceptance criteria and build automated coverage as AI-powered capabilities are being developed.
This role requires experience testing both traditional deterministic software systems and non-deterministic AI/ML systems. Traditional application behavior can often be validated against consistent expected outputs, while AI models may produce different but acceptable results across models, prompts and inputs. You will help design testing approaches that account for this variability while still establishing measurable quality, reliability and safety standards. You will build automation frameworks, develop deterministic and probabilistic validation strategies, integrate testing into CI/CD workflows and help Keeper move toward continuous quality engineering rather than end-of-cycle QA.
Responsibilities
- Partner directly with AI engineers, Software Engineers and Product Managers during design and development to define test strategy, acceptance criteria and automation requirements
- Design and maintain automated testing frameworks for AI-powered applications, backend services, APIs and supporting software systems