Member of Technical Staff, Head of Quality
Hands-on quality lead at an applied AI research company building training environments and datasets: the person reviews human data and environment outputs, builds automated evals, graders and QA agents, defines quality metrics, owns delivery acceptance, and manages a growing quality team. Fully onsite in San Francisco.
- Compensation
- $180K–$280K / yr
- Location
- San Francisco, California, United States
- Work setup
- Onsite
- Seniority
- C level
- Employment
- Not specified
- Experience
- 1+ years
- Category
- Software engineering
- Posted
Skills
Must have
- 1 to 5 years of relevant LLM evaluation, QA engineering, or ML data-quality experience
- Experience building automated quality systems for ML data, agents, or large-scale pipelines
- Technical understanding of what makes training data and environments useful for learning
- Hands-on work with LLM evaluations, annotation, or human-in-the-loop review
- Ability to build graders, judge models, telemetry, and dashboards
- Strong attention to detail and willingness to spend substantial time reviewing data manually
Nice to have
- RL environment, verifier, or reward-design experience
- Founder or zero-to-one product-building experience
- A STEM degree
Full description
About the company
Our client is an applied AI research company developing training environments and datasets. Its technical teams build the systems that help AI models learn from complex tasks and data.
The role
Raydar is recruiting for this role on behalf of our client. Own quality across environments and datasets as a Member of Technical Staff, Head of Quality. This is a hands-on evaluation and data-review role with responsibility for automation and a growing quality team.
What you'll do
- Review human data and environment outputs in detail.
- Build automated evaluations, graders, and QA agents.
- Detect reward hacking, broken verifiers, and distribution mismatches.
- Define measurable quality metrics and acceptance criteria.
- Own delivery acceptance for customer projects.
- Hire, onboard, and manage quality engineers and reviewers.
Requirements
What we're looking for
- 1 to 5 years of relevant LLM evaluation, QA engineering, or ML data-quality experience.
- Experience building automated quality systems for ML data, agents, or large-scale pipelines.
- Technical understanding of what makes training data and environments useful for learning.
- Hands-on work with LLM evaluations, annotation, or human-in-the-loop review.
- Ability to build graders, judge models, telemetry, and dashboards.
- Strong attention to detail and willingness to spend substantial time reviewing data manually.
Bonus points
- RL environment, verifier, or reward-design experience.
- Founder or zero-to-one product-building experience.
- A STEM degree.
Benefits
Compensation and benefits
- Base salary: USD 180,000 to 280,000 per year.
- Equity: Competitive equity.
Location and work model
- San Francisco, California, United States.
- Fully onsite role; exact weekly office-day count is not specified.
- Visa transfers are supported.