Prompt Engineer / AI Quality Lead (LLM Testing and Evals)
Owns the base prompt across 35+ client AI DM-setting bots at SettyAI: builds automated regression/eval testing for multi-turn LLM conversations, rolls out prompt updates without breaking client-specific behaviour, and diagnoses reasoning-level LLM failures. Stack: Claude and other LLMs, prompt engineering, eval tooling, Claude Code.
- Compensation
- Not disclosed
- Location
- Europe
- Work setup
- Remote
- Seniority
- Lead
- Employment
- Contract
- Experience
- Not specified
- Category
- Ai engineering
- Posted
Skills
Must have
- Production prompt engineering on multi-turn LLM systems
- Built automated evals/tests for LLM outputs
- High agency: owns outcomes without being managed
- Fluent English (C1+)
- Heavy daily Claude Code (or similar) user
Nice to have
- High-ticket sales, appointment setting or coaching industry knowledge
- n8n, Hermes or agent frameworks
Full description
We’re hiring fast. The process closes Friday 23 Oct: apply → paid test → final call → offer. Start Monday 26 Oct.
Stack: LLMs (Claude and others), prompt engineering, automated LLM testing/evals, Claude Code, our internal tools. Bonus: n8n, Hermes/agent frameworks.
About us
SettyAI builds AI DM setters for high-ticket coaching and info-business operators. Our bots qualify leads and book sales calls in Instagram DMs for 35+ clients, and we’re doubling that by end of November.
What you’ll own
Changes don’t create new problems. When we change a client’s bot, nothing else may break. Today that sometimes happens: we add one question and the bot starts repeating questions the lead already answered.
Clients need fewer changes. Feedback that applies to every client goes into our base prompt, so the same request never comes in twice.
You decide how. We measure results: fewer change requests, zero regressions, faster high-quality bots.
Responsibilities
- Own the base prompt across all client bots
- Build and extend our automated testing/regression system for multi-turn LLM conversations
- Design how base updates roll out across many client bots without breaking client-specific behaviour
- Diagnose LLM failures at the reasoning level and fix root causes
- Report weekly on change volume, regressions and quality
Requirements
- Production prompt engineering on multi-turn LLM systems
- Built automated evals/tests for LLM outputs
- High agency: you own outcomes without being managed
- Fluent English (C1+)
- Heavy daily Claude Code (or similar) user
Nice to have
- High-ticket sales, appointment setting or coaching industry knowledge
- n8n, Hermes or agent frameworks
Terms
Full-time contractor, remote, long-term. Starts Monday 26 Oct.
Process (done by Friday)
Apply + Loom (see control questions)
Paid written test, 24h turnaround
Top 2–3: 45-min call + paid 90-min live session
Offer Friday
On the Loom: we want depth, not buzzwords. Name specific causes, walk through exactly what you'd check, and describe the system in enough detail that we could build it. Generic answers ("tweak the prompt", "add more testing") are rejected.
Start your cover letter with the word “Rootcause” so we know you read this. Applications without a Loom won’t be reviewed.
.
..