Software Engineering Director, Agentic Evaluations
Directs the core team building G2's agent evaluation system, which validates the performance of software vendors' AI agents. Day to day they own the full evaluation stack (admin, API, workflow dispatch), design reusable eval primitives, and lead/mentor engineers, using backend stacks like Python/Java/Kotlin/TypeScript/Go plus LLM-as-a-judge tooling.
- Compensation
- $260K–$320K / yr
- Location
- Remote (US)
- Work setup
- Remote
- Seniority
- Not specified
- Employment
- Not specified
- Experience
- 2+ years
- Category
- Software engineering
- Posted
Skills
Must have
- 10+ years of professional programming experience in backend or full-stack environments
- 2+ years of experience directly managing engineers
- Expert-level backend development in Python, Java/Kotlin, TypeScript/JavaScript, or Go, with strong proficiency in backend frameworks like FastAPI
- Direct experience creating evaluations (evals) against a customer-facing agent using agent trajectory trace data and rubrics to measure task completion, accuracy, correctness, or policy adherence
- Direct experience using frontier models from OpenAI, Anthropic, or Google in LLM-as-a-judge applications
- Regular use of coding agent harnesses like Claude Code, Codex, Opencode, or Pi in the daily development workflow
- BS/BA degree in a related field
Nice to have
- Experience with agent tool use via direct integration or as mediated by MCP servers
- Knowledge of STATE-Bench, tau2-bench, or similar benchmark frameworks
- Experience using Playwright, browser-use, Chrome DevTools MCP, or similar to drive user flows in the browser
- Experience with durable execution frameworks & workflow solutions (Temporal, DBOS, Cloudflare/Vercel Workflows) or agent sandboxing (AWS E2B, Daytona, Cloudflare/Vercel Containers)
Full description
About G2 - The Company
G2 is the world's largest and most trusted software marketplace. When you join G2, you’re joining the industry’s leading team that helps businesses reach their peak potential by powering decisions and strategies with trusted insights from real software users.
Now, we have joined forces with Capterra, SoftwareAdvice, and GetApp to create the largest source of online data and software insights to fuel intelligent buying in the age of AI. With 200M+ combined annual visitors and 6M verified reviews, we are now the centralized place to enable software buyers to make better and faster decisions with confidence.
And we are just getting started! We are setting out to transform the global B2B software industry and become the most trusted data foundation for buyers and sellers of software for the age of AI.
Does that sound exciting to you? Come join us as we try to reach our next PEAK!
About G2 - Our People
At G2, everything we are and what we do is grounded in our PEAK values— (Performance + Entrepreneurship + Authenticity + Kindness. Working at G2 means you are part of a value-driven, growing global community that climbs PEAKs together. We cheer for each other’s successes, learn from our mistakes, and support and lean on one another during challenging times. With ambition and entrepreneurial spirit we push each other to take on challenging work, which will help us all to grow and learn.
You will be part of a global, diverse team of smart, dedicated, and kind individuals - each with unique talents, aspirations, and life experiences. At the heart of our community and culture are our people-led ERGs, which celebrate and highlight the diverse identities of our global team. As an organization, we are intentional about our DEI and philanthropic work (like our G2 Gives program) because it encourages us all to be better people.
About The Role
Summary of Responsibilities:
As a Software Engineering Director, Agentic Evaluations, your responsibilities will center around delivery of useful and credible evaluations of agents offered by software vendors. You’ll enhance the system that performs validation of agent performance, and lead the core team behind it. You will leverage “agent-first” ways of working, applying agentic engineering techniques to deliver your work to production.
Detailed Responsibilities:Build and enhance agent evaluations that run against live agents [80%]:
- Help scope out the feasibility and effort involved in evaluating an agent offered by a software vendor, identifying the most practical path to reaching a credible evaluation
- Own the full stack of the core evaluation system, from admin & API surface to eval workflow dispatch
- Design evaluation system primitives that generalize across software verticals, so onboarding a new category benefits from reuse