Machine Learning Engineer (Evals and Voice Models)
- Compensation
- Not disclosed
- Location
- San Francisco Office
- Work setup
- Not specified
- Seniority
- Not specified
- Employment
- Contract
- Experience
- 3+ years
- Category
- Ml / AI
- Posted
Skills
Must have
- BS in Computer Science, Machine Learning, Statistics, or related field
- 3+ years of experience in ML Engineering or Applied ML with 8+ years of overall experience
- Strong experience in evaluating supervised, unsupervised, LLMs and deep learning models.
- Hands-on experience in failure analysis and evaluating LLMs
- Experience building automated evaluation systems
- Strong communication skills to articulate complex technical concepts across technical and non-technical audiences
- Hands-on experience training or fine-tuning voice/speech models (TTS, ASR, or speech-to-speech), including data pipeline construction and experimentation.
Nice to have
- MS / PhD in Computer Science, Machine Learning, Statistics, or related field
- Experience evaluating LLMs or agentic systems (e.g., LLM-as-a-judge, RAG evaluation)
- Experience with synthetic data generation and prompt engineering
- Experience training or fine-tuning voice models at scale, with familiarity in synthetic data generation, model distillation, or low-latency inference optimization for production voice agents.
Full description
Aircall is a unicorn AI-powered communications platform transforming how teams win with customers.
For over a decade, we’ve believed voice builds the strongest business relationships. We bring voice, SMS, WhatsApp, and 250+ integrations together on one seamless workspace—so every customer conversation drives revenue, resolves issues faster, and builds trust. Our AI Voice Agents handle routine interactions while your team leads the conversations that matter most. Used by 24,000+ businesses worldwide.
We’re 600+ people from 40+ nationalities across Paris, London, Madrid, Berlin, New York, San Francisco, Seattle, Sydney, and Mexico City. We’re ambitious, product-driven, and execution-focused—built for visible impact, fast decisions, and real growth.
Come join us!
Aircall's AI suite includes an AI Voice Agent and AI Messaging Agent that autonomously handle calls, WhatsApp, and SMS, plus AI Assist, which delivers real-time coaching, call summaries, and CRM automation for sales and support teams. We are looking for someone that can build out the evaluation foundation across all of these products and other agentic products. You'll work on voice models, agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together by establishing shared metrics, test sets, and tooling to measure accuracy, resolution quality, and safety consistently across products. You will set up repeatable pipelines for regression testing and benchmarking as models and features evolve so teams can ship confidently without re-inventing evaluation methodology for each product.
Key Responsibilities
- Design and document comprehensive evaluation frameworks for Aircall’s AI agents across voice, chat and messaging.
- Train and fine-tune voice models (TTS, ASR, speech-to-speech) using production and synthetic data, iterating on architecture, data mix, and training strategy to improve accuracy, naturalness, and latency.
- Assess AI generated solutions across training pipelines, experimentation setups, debugging processes, and optimization strategies.
- Analyze system design decisions and identify strengths, weaknesses, and potential failure points.
- Design annotation guidelines and workflows for human-labeled evaluation data, and calibrate LLM-as-judge systems against human raters to ensure automated evals stay trustworthy over time.
- Build and maintain live quality monitoring for deployed AI agents, tracking accuracy, resolution rate, and safety signals in production, and flagging model or data drift before it impacts customers.
- Own the metric contract for every published AI metrics, including definition, population, grain, rollup, validity window.