Senior Software Engineer, Agentic Coding Evaluation (Python, C++) (Remote, Contract)
Senior engineers review and score long-horizon, multi-step coding work produced by AI agents in real repositories, write reference solutions and tests in Python or C++, and judge code quality, correctness, and security against a rubric. Remote, part-time contract role evaluating AI coding agents.
- Compensation
- $80–$120 / hr
- Location
- Chicago
- Work setup
- Not specified
- Seniority
- Senior
- Employment
- Part time
- Experience
- 5+ years
- Category
- Software engineering
- Posted
Skills
Must have
- 5+ years of professional software engineering
- Expert in Python or C++
- Comfortable reading unfamiliar codebases and writing tests
- Fluent written English
Nice to have
- Open-source maintainer or contributor record
- Systems, performance or security engineering
- Code review lead experience
Full description
Compensation: No equity
Senior engineers review and grade long-horizon, multi-step coding work by AI agents in real repositories.
Pay: $80 – $120 / hour (USD), paid weekly. Rate set after a short paid assessment.
Contract: independent contractor, flexible part-time (10–40 hrs/week), ongoing and project-based. Remote, worldwide.
Work is part-time and asynchronous. A paid coding assessment comes before any project work.
What you'll do
- Review agent trajectories on multi-file, long-horizon coding tasks and score each step.
- Write reference solutions and tests in Python or C++.
- Judge code quality, correctness, security and maintainability against a rubric.
- Explain the root cause when an agent's fix fails.
Requirements
- 5+ years of professional software engineering.
- Expert in Python or C++.
- Comfortable reading unfamiliar codebases and writing tests.
- Fluent written English.
Nice to have
- Open-source maintainer or contributor record.
- Systems, performance or security engineering.
- Code review lead experience.
How it works
- Create a free profile and upload your CV
- Video interview, about 20 minutes, in English, on a laptop or desktop with camera and microphone
- ID and credential check
- Matched to paid projects as they open. Paid weekly in USD.
Free to apply. Vettedlance never charges candidates.
About Vettedlance
Vettedlance is a vetted network of domain experts who train and evaluate AI models for AI labs and data vendors. Experts apply once, get verified and are matched to paid projects.
Apply here (required, applications are only reviewed on our site): https://vettedlance.com/jobs/senior-software-engineer-agentic-coding-eval?utm_source=wellfound&utm_medium=job_board&utm_campaign=recruit_d1