Senior AI / LLM Evaluation Engineer
Senior engineer, placed by Ruzora with a US startup, who quantifies whether LLM features (RAG, agents, tool use, generation) got better or worse: designing eval sets and metrics, running CI-integrated eval pipelines, LLM-as-judge graders, and A/B model comparisons. Stack: Python, OpenAI/Anthropic APIs, eval frameworks, SQL, CI/CD.
- Compensation
- Not disclosed
- Location
- Remote, Argentina; Belize; Bolivia; Brazil; Chile; Colombia; Costa Rica; Dominican Republic; Ecuador; El Salvador; Guatemala; Honduras; Mexico; Nicaragua; Panama; Paraguay; Peru; Uruguay
- Work setup
- Remote
- Seniority
- Senior
- Employment
- Full time
- Experience
- 1+ years
- Category
- Ai engineering
- Posted
Skills
Must have
- 4+ years of software or ML engineering in Python
- 1+ years building or evaluating LLM applications in production
- Experience with eval tooling (OpenAI Evals, promptfoo, LangSmith, Braintrust, Ragas or your own)
- Solid statistics: sampling, confidence intervals, inter-rater agreement
- Comfortable with the OpenAI, Anthropic or similar model APIs
- Excellent written and spoken English (B2 or higher)
Nice to have
- Building labeled datasets and annotation workflows
- Red-teaming or AI safety work
- Fine-tuning or reward modeling
- Observability for LLM apps (tracing, cost and latency dashboards)
Full description
Senior AI / LLM Evaluation Engineer (Full-time, remote from LatAm)
Ruzora places senior LatAm engineers full-time with US startups. We're hiring a Senior AI / LLM Evaluation Engineer to join a partner company shipping LLM products. Your job is to tell the team, with numbers, whether the model, prompt or agent got better or worse, and to build the tooling that makes that answer fast.
What you'll do
- Design evaluation sets and metrics for LLM features: RAG, agents, tool use, classification, generation
- Build eval pipelines that run in CI and on every model or prompt change
- Set up LLM-as-judge graders and check them against human labels
- Run regression and A/B comparisons across models and providers
- Find failure modes (hallucinations, refusals, prompt injection) and turn them into tests
- Report results clearly to engineers and product owners
What we're looking for
- 4+ years of software or ML engineering in Python
- 1+ years building or evaluating LLM applications in production
- Experience with eval tooling (OpenAI Evals, promptfoo, LangSmith, Braintrust, Ragas or your own)
- Solid statistics: sampling, confidence intervals, inter-rater agreement
- Comfortable with the OpenAI, Anthropic or similar model APIs
- Excellent written and spoken English (B2 or higher)
Nice to have
- Building labeled datasets and annotation workflows
- Red-teaming or AI safety work
- Fine-tuning or reward modeling
- Observability for LLM apps (tracing, cost and latency dashboards)
Stack: Python, LLM APIs, eval frameworks, SQL, CI/CD
What you get
- Full-time, 100% remote from anywhere in Latin America
- Competitive salary in USD
- Flexible hours with overlap with US time zones
- Professional development budget, health insurance stipend, equipment allowance
Full role details: https://www.ruzora.com/jobs/ai-llm-evaluation-engineer
Location: Remote - Argentina, Belize, Bolivia, Brazil, Chile, Colombia, Costa Rica, Dominican Republic, Ecuador, El Salvador, Guatemala, Honduras, Mexico, Nicaragua, Panama, Paraguay, Peru, Uruguay
Skills required for this job:
- Artificial intelligence security
- CI/CD
- Dataset annotation
- Evaluation methods
- LLM evaluation
- Machine learning
- Model fine-tuning
- Observability
- Python
- SQL
- Software engineering