BraintrustCoding & software eval
Senior Coding Annotator / LLM Evaluation Engineer (Contract / BYO)
up to $85/hr
About this role, in Braintrust's words
This is a contracting engagement - initially 6 months - with potential for long term engagement.
What you'll do
- Create high-quality coding prompts and reference answers (benchmark-style, e.g. SWE-Bench-like problems).
- Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
- Identify and document model failures, edge cases, and reasoning gaps.
Show all 6 itemsHide the other 3 in What you'll do
- Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
- Build or configure coding environments to support evaluation and reinforcement learning (RL).
- Follow detailed annotation and evaluation guidelines with high consistency.
Candidate background
What We’re Looking For
- 5+ years of professional software development experience .
- Strong Python skills (required).
- Knowledge of at least one additional programming language (bonus).
- 1+ year of coding annotation and/or LLM evaluation experience (part-time OK) for a major frontier AI lab or AI infrastructure company.
- Prior code reviewer experience is a plus.
- Proven ability to apply structured evaluation criteria and write clear technical feedback.
- Fluent in English (written and spoken) .
- Team lead or mentoring experience is a strong plus.
Quoted from selected sections of Braintrust's posting. The full posting may state other conditions. Read the full posting
A closer look
More context, when you need it.
Full job descriptionThe original posting, in Braintrust's words
Posted by Braintrust, reproduced here so you can judge the role before clicking. Original posting ↗
This is a contracting engagement - initially 6 months - with potential for long term engagement.
Location: Paris-based preferred; alternatively Europe remote for strong candidates
We are building and evaluating state-of-the-art large language models (LLMs) and are looking for experienced software engineers to join our evaluation and annotation team. This role sits at the intersection of real-world software engineering, model evaluation, and applied AI , and is critical to improving model reliability, reasoning, and code quality.
You will design challenging coding tasks, evaluate model outputs against rigorous benchmarks, identify failure modes, and contribute to reinforcement learning and model improvement workflows.
This is not a junior annotation role. We are looking for practitioners with deep hands-on coding experience who can think like both an engineer and an evaluator.
What You’ll Do
• Create high-quality coding prompts and reference answers (benchmark-style, e.g. SWE-Bench-like problems).
• Evaluate LLM outputs for code generation, refactoring, debugging, and implementation tasks.
• Identify and document model failures, edge cases, and reasoning gaps.
• Perform head-to-head evaluations between private LLMs (Mistral-based) and leading external models.
• Build or configure coding environments to support evaluation and reinforcement learning (RL).
• Follow detailed annotation and evaluation guidelines with high consistency.
What We’re Looking For
• 5+ years of professional software development experience .
• Strong Python skills (required).
• Knowledge of at least one additional programming language (bonus).
• 1+ year of coding annotation and/or LLM evaluation experience (part-time OK) for a major frontier AI lab or AI infrastructure company.
• Prior code reviewer experience is a plus.
• Proven ability to apply structured evaluation criteria and write clear technical feedback.
• Fluent in English (written and spoken) .
• Team lead or mentoring experience is a strong plus.
Why This Role
• Work hands-on with cutting-edge LLMs.
• Apply real-world engineering judgment to model evaluation and improvement.
• High-impact, technical work with a focused, senior team.
Published location regions: Europe
Working with BraintrustPay, getting started and work, from Braintrust's own pages
What the platform publishes in general. These are not extra terms confirmed for this particular role.
- Getting paid
- Through a processor such as Stripe or Wise. On hourly contracts you invoice through Braintrust and payment starts within 30 days, or later if the client has not paid. Its AI training page says gig tasks are paid weekly. usebraintrust.com/site-service-fees-terms, usebraintrust.com/ai-training
- Before your first paid task
- A profile and a short AI interview, usually under an hour, then ID verification and skills validation. Certification takes 1 to 2 days, and matching starts after it. usebraintrust.com/for-talent
- How much work
- No guaranteed minimum or regular pay. Its terms say Braintrust does not introduce talent to clients or find projects for them. usebraintrust.com/talent-terms, usebraintrust.com/terms
- Contract
- Independent contractor. Each project is a contract directly between you and the client, under terms with Freelance Labs, Inc. usebraintrust.com/talent-terms, usebraintrust.com/terms
Read on the pages linked above on 23 Sep 2026. Platforms change these terms, so check the linked page before you rely on one.
How does the pay compare?Across Coding & software eval roles, not this role alone
From our own daily reads of every open listing on the platforms we follow, not from Braintrust's posting. How we count.
Pay across the field: Coding & software eval
Across 393 roles in this field with a stated hourly rate in US dollars, on 13 platforms, the middle half pays $65 to $120 an hour at the top of the range, and the median is $95. This role's top rate, $85, is in the middle half. Of the 140 distinct rates we count, 59 are lower and 79 are higher.
| Platform | Roles | Top of range, per hour |
|---|---|---|
| Alignerr | 137 | $50 to $400 · median $100 |
| micro1 | 46 | $35 to $350 · median $110 |
| Handshake AI | 43 | $47 to $125 · median $80 |
| Meridial (Invisible) | 34 | $22 to $150 · median $60 |
| Mercor | 33 | $60 to $200 · median $105 |
Sources and listing historyWhere this information comes from
This listing's history
- First seen here on 13 Sep 2026, and still on Braintrust's list when we last read it on 13 Oct 2026.
- The listed pay has not changed since then.
- Posted by Braintrust
- First seen by AIPayList
- Last seen live
- Listing ID
- 17323
Pay as the source states it: “USD 70–85 hourly”.
Platforms sometimes change a rate after posting. Every listing here is re-read at least once a day and corrected when its source moves, so a difference you see on the platform is usually a change made since the last read. How freshness works.
Keep exploring
More like this
Related field, different requirements. Check each role's own page before you apply.