Machine Learning Engineer – Evals

Salary: $220K–$300K + competitive equity
📍 Location: New York, NY
🏢 Work Model: 5 days/week on-site
💼 Employment: Full-time
🛂 Visa: US Citizens & Green Card holders only — no visa sponsorship
👤 Hiring: 1 position

About the Role

We're looking for a Machine Learning Engineer – Evals to own the evaluation of Honcho, the system that tells a research-driven team whether its identity representations are actually improving.

You'll work on a challenging problem: defining what "better" means for representations that evolve over time, often without traditional ground truth.

This is an opportunity for someone who enjoys building evaluation systems from scratch, analyzing real-world traces, identifying what is actually broken, and shipping the solution themselves.

What You'll Do

🔹 Design evaluations
Define evaluation metrics and determine what "better" means as representations evolve.

🔹 Build evaluation pipelines & harnesses
Create the infrastructure for data collection, labeling, versioning, reruns, and evaluation judges.

🔹 Run evaluations & generate insights
Analyze traces and results to identify meaningful problems—not simply what is easiest to measure—and turn those insights into product improvements.

🔹 Build simulation agents at scale
Develop systems capable of modeling and evaluating large numbers of entities simultaneously.

🔹 Own the entire evaluation loop
Take ownership from the initial research question through pipeline development, evaluation, analysis, reruns, and final recommendations.

Must-Have Qualifications

✅ 3+ years of experience designing and owning LLM evaluations
✅ Proven experience designing, building, and running LLM evaluations end-to-end
✅ Production-quality Python — including testing, maintenance, and release cycles
✅ Strong experience with PyTorch
✅ Strong experience with Hugging Face

Nice-to-Have Experience

⭐ Memory, identity, personalization, agent evaluation, or agentic optimization
⭐ Data pipeline tooling such as Hydra, Kafka, or SQL
⭐ Published research as a first/co-author at NeurIPS, ICML, ICLR, or equivalent
⭐ High-quality open-source research/work comparable to academic publications
⭐ Degree in Computer Science, Mathematics, Statistics, or another quantitative field

Who We're Looking For

This role is ideal for someone who doesn't just run existing benchmarks but can answer:

"How should we measure this, and how do we know our measurement is actually telling us something useful?"

If you've built LLM evaluation systems from the ground up and enjoy working at the intersection of ML research, agents, evaluation, and production engineering, we'd love to hear from you.

Location: New York, United States

Skills required for this job:

• AI evaluation
• Data pipelines
• Hugging Face
• LLM evaluation
• PyTorch
• Python
• SQL