BraintrustCoding & software eval
Software Engineer – AI Code Evaluation
up to $105/hr
About this role, in Braintrust's words
Review Python code, tests, and execution logs from AI coding tasks to determine why a solution failed and document the evidence behind your conclusion.
What you'll do
- Read unfamiliar code and trace an AI coding agent’s changes through the task requirements, test suite, and run logs.
- Reproduce failures and identify what caused them. Check whether the implementation, test assertion, task specification, or environment is responsible.
- Diagnose test harness and environment failures using Docker configuration, dependency information, and execution logs.
Show all 6 itemsHide the other 3 in What you'll do
- Determine whether the issue changed the task’s pass/fail result. Write a concise finding that cites the relevant code, test, requirement, or log and explains how you verified it.
- Join daily discussions with research scientists to compare findings and apply consistent evaluation criteria.
- A typical investigation takes 30–60 minutes .
Candidate background
What We’re Looking For
- 2+ years of professional experience writing, reviewing, and debugging software , or a completed STEM master’s degree or PhD with intensive programming and computational research experience.
- Hands-on experience reading and debugging Python in unfamiliar or complex codebases. You can walk through a real failure you investigated, explain what evidence isolated the cause, and describe how you verified the fix.
- Ability to read at least one additional language, such as JavaScript or TypeScript, Go, C++, or Rust .
- Practical experience with Git, Docker, and software build or test environments .
- Experience with a test framework such as pytest, unittest, Jest, or Mocha . You can read test assertions and assess whether they match the documented behavior.
- Clear written and spoken English for explaining technical findings and the evidence behind them.
Nice to Have
Show all 3 itemsHide these 3 items in Nice to Have
- Experience evaluating LLMs or coding agents, including reviewing their code changes, test results, or execution traces.
- Familiarity with coding benchmarks such as SWE-bench or CodeContests.
- Experience with AI reliability evaluation or collaborating with ML researchers.
Quoted from selected sections of Braintrust's posting. The full posting may state other conditions. Read the full posting
A closer look
More context, when you need it.
Full job descriptionThe original posting, in Braintrust's words
Posted by Braintrust, reproduced here so you can judge the role before clicking. Original posting ↗
About the Opportunity
Review Python code, tests, and execution logs from AI coding tasks to determine why a solution failed and document the evidence behind your conclusion.
You’ll work with a leading AI research organization to evaluate how coding agents perform on software engineering tasks. A failing test might point to a bug in the agent’s code, an overly restrictive assertion, an unclear requirement, or a problem with the test environment. Your job is to trace what happened and explain whether it affected the result.
What You’ll Do
• Read unfamiliar code and trace an AI coding agent’s changes through the task requirements, test suite, and run logs.
• Reproduce failures and identify what caused them. Check whether the implementation, test assertion, task specification, or environment is responsible.
• Diagnose test harness and environment failures using Docker configuration, dependency information, and execution logs.
• Determine whether the issue changed the task’s pass/fail result. Write a concise finding that cites the relevant code, test, requirement, or log and explains how you verified it.
• Join daily discussions with research scientists to compare findings and apply consistent evaluation criteria.
A typical investigation takes 30–60 minutes .
What We’re Looking For
• 2+ years of professional experience writing, reviewing, and debugging software , or a completed STEM master’s degree or PhD with intensive programming and computational research experience.
• Hands-on experience reading and debugging Python in unfamiliar or complex codebases. You can walk through a real failure you investigated, explain what evidence isolated the cause, and describe how you verified the fix.
• Ability to read at least one additional language, such as JavaScript or TypeScript, Go, C++, or Rust .
• Practical experience with Git, Docker, and software build or test environments .
• Experience with a test framework such as pytest, unittest, Jest, or Mocha . You can read test assertions and assess whether they match the documented behavior.
• Clear written and spoken English for explaining technical findings and the evidence behind them.
Nice to Have
• Experience evaluating LLMs or coding agents, including reviewing their code changes, test results, or execution traces.
• Familiarity with coding benchmarks such as SWE-bench or CodeContests.
• Experience with AI reliability evaluation or collaborating with ML researchers.
Engagement Details
• Location — Remote, United States only
• Expected commitment — 35 hours per week
• Start — Immediate
• Compensation — $75–$105 USD per hour, depending on relevant skills and experience
Interview Process
Shortlisted candidates will need to complete a 15-minute AI interview.
Working with BraintrustPay, getting started and work, from Braintrust's own pages
What the platform publishes in general. These are not extra terms confirmed for this particular role.
- Getting paid
- Through a processor such as Stripe or Wise. On hourly contracts you invoice through Braintrust and payment starts within 30 days, or later if the client has not paid. Its AI training page says gig tasks are paid weekly. usebraintrust.com/site-service-fees-terms, usebraintrust.com/ai-training
- Before your first paid task
- A profile and a short AI interview, usually under an hour, then ID verification and skills validation. Certification takes 1 to 2 days, and matching starts after it. usebraintrust.com/for-talent
- How much work
- No guaranteed minimum or regular pay. Its terms say Braintrust does not introduce talent to clients or find projects for them. usebraintrust.com/talent-terms, usebraintrust.com/terms
- Contract
- Independent contractor. Each project is a contract directly between you and the client, under terms with Freelance Labs, Inc. usebraintrust.com/talent-terms, usebraintrust.com/terms
Read on the pages linked above on 23 Sep 2026. Platforms change these terms, so check the linked page before you rely on one.
How does the pay compare?Across Coding & software eval roles, not this role alone
From our own daily reads of every open listing on the platforms we follow, not from Braintrust's posting. How we count.
Pay across the field: Coding & software eval
Across 411 roles in this field with a stated hourly rate in US dollars, on 18 platforms, the middle half pays $60 to $120 an hour at the top of the range, and the median is $80. This role's top rate, $105, is in the middle half. Of the 135 distinct rates we count, 85 are lower and 48 are higher.
| Platform | Roles | Top of range, per hour |
|---|---|---|
| Alignerr | 97 | $50 to $200 · median $100 |
| Anyone AI | 79 | $35 to $150 · median $65 |
| micro1 | 42 | $50 to $400 · median $100 |
| AfterQuery | 37 | $45 to $200 · median $100 |
| Meridial (Invisible) | 35 | $22 to $150 · median $50 |
Sources and listing historyWhere this information comes from
This listing's history
- First seen here on 21 Nov 2026.
- The listed pay has not changed since then.
- Posted by Braintrust
- First seen by AIPayList
- Last seen live
- Listing ID
- 18169
Pay as the source states it: “USD 75–105 hourly”.
Platforms sometimes change a rate after posting. Every listing here is re-read at least once a day and corrected when its source moves, so a difference you see on the platform is usually a change made since the last read. How freshness works.
Keep exploring
More like this
Related field, different requirements. Check each role's own page before you apply.