MercorCoding & software eval
Applied Engineer, Evaluations Role
up to $120/hr
About this role, in Mercor's words
Mercor is seeking software engineers to build and refine evaluations. You’ll turn completed pull requests into engineering tasks and use our in-house evaluation framework to multiple frontier coding agents, including Claude Code, Codex and more.
A closer look
More context, when you need it.
Full job descriptionThe original posting, in Mercor's words
Posted by Mercor, reproduced here so you can judge the role before clicking. Original posting ↗
Mercor is seeking software engineers to build and refine evaluations. You’ll turn completed pull requests into engineering tasks and use our in-house evaluation framework to multiple frontier coding agents, including Claude Code, Codex and more.
We work alongside an in-house research team and have produced industry leading benchmarks to compare the performance of frontier large language models.
In this role, you will:
• Identify repositories with enough substantive work to support challenging evaluations, and select suitable completed PRs.
• Investigate each problem, its reference solution, and the repository’s architecture, tests, and conventions.
• Write task prompts and build evaluations using automated tests, shell commands, and LLM grading prompts.
• Validate evaluations against reference solutions and deliberately flawed implementations. Find missing checks, incorrect grades, and criteria that unnecessarily constrain how a problem can be solved.
• Run evaluations repeatedly across multiple models and harnesses. Investigate whether failures come from the agent’s solution, the environment, or the grading, and establish that tasks expose meaningful weaknesses in agent performance.
• Refine evaluations through repeated testing and review. Document findings and grading decisions, and work through feedback.
You’ll need:
• Strong programming fundamentals and practical experience working in substantial and complex codebases. Languages we create evals for include TypeScript/JavaScript, Python, Java, Kotlin, Go, Ruby, PHP, C++ and Rust
• The ability to understand unfamiliar code, investigate subtle behavior, and assess whether different implementations solve the same problem correctly.
• Experience writing meaningful tests, including edge cases and regression coverage.
• Attention to detail and patience for repeated investigation and refinement.
• Practical experience using AI coding tools, with the judgment to verify their output and catch mistakes.
• Confidence using Git, test runners, and CI tooling.
• Clear and fluent written and verbal communication with the ability to own a task independently while raising questions when requirements are ambiguous.
Relevant experience can come from open-source, private, or enterprise repositories. Familiarity with a particular language or ecosystem is helpful; the ability to learn the repository and make sound engineering judgments matters more than its popularity or your public contribution history.
We provide onboarding to the evaluation framework and ongoing review feedback. After onboarding, you’ll be expected to own task selection, evaluation development, and iteration without step-by-step direction.
Working with MercorPay, getting started and work, from Mercor's own pages
What the platform publishes in general. These are not extra terms confirmed for this particular role.
- Getting paid
- Hourly work is paid every Wednesday for the previous week, through Stripe Connect or Wise, whichever Mercor assigns to your region. Stripe holds the first payout for 7 days. talent.docs.mercor.com/how-to/payments, talent.docs.mercor.com/policies/contracts
- Before your first paid task
- An AI interview of about 20 minutes, reused for other roles that need it, then any assessments the role asks for. Before logging hours: ID verification through Persona, a tax form, the contract documents, the Insightful time tracker, and a background check the onboarding guide says usually takes one to two weeks. talent.docs.mercor.com/support/ai-interview, talent.docs.mercor.com/how-to/apply, talent.docs.mercor.com/policies/contracts, talent.docs.mercor.com/how-to/id-verification, talent.docs.mercor.com/working/project-onboarding
- How much work
- Hearing back after the AI interview usually takes 2 to 4 weeks. When a project ends you are not moved to a new one automatically. talent.docs.mercor.com/support/ai-interview, talent.docs.mercor.com/support/project
- Contract
- Independent contractor with Mercor.io Corporation of San Francisco, which deducts no taxes. talent.docs.mercor.com/policies/taxes-work-authorization, talent.docs.mercor.com/support/legal
Read on the pages linked above on 23 Sep 2026. Platforms change these terms, so check the linked page before you rely on one.
Alex's experience on MercorA personal account, not a promise about your project
Alexruns this site and has worked on Mercor himself. Written 31 Aug 2026.
- Every interview I took used an AI interviewer. They always checked my microphone and camera and asked me to share my screen, and some included a timed exercise of around 25 minutes.
- Many later applications needed no new interview from me. With the required assessments complete and no other outstanding steps, applying could be close to one click.
- The application I checked did not ask for a rate. My offer matched my minimum in Work Preferences. That does not establish that Mercor always caps higher offers at your preference.
- Onboarding and setup came to about 4 hours, with more security and tool setup. Of the three platforms I worked on, it was the most complicated for me. That is an estimate from my projects, not a platform average.
- Mercor required the Insightful tracker on my project. I observed app and website usage monitoring, and I am unsure which screenshot settings applied, so check the configuration.
- I was paid for hours logged, once every 14 days, and I have not experienced late or incorrect payments. That was my arrangement. Mercor's current documentation describes weekly payout cycles, so check your own offer.
One person's applications, not a promise about yours.
What happens after you apply, first-handHow does the pay compare?Across Coding & software eval roles, not this role alone
From our own daily reads of every open listing on the platforms we follow, not from Mercor's posting. How we count.
Pay across the field: Coding & software eval
Across 393 roles in this field with a stated hourly rate in US dollars, on 14 platforms, the middle half pays $65 to $120 an hour at the top of the range, and the median is $95. This role's top rate, $120, is in the top quarter. Of the 141 distinct rates we count, 97 are lower and 31 are higher.
| Platform | Roles | Top of range, per hour |
|---|---|---|
| Alignerr | 137 | $50 to $400 · median $100 |
| micro1 | 46 | $35 to $350 · median $110 |
| Handshake AI | 43 | $47 to $125 · median $80 |
| Meridial (Invisible) | 34 | $22 to $150 · median $60 |
| Mercor (this platform) | 32 | $60 to $200 · median $105 |
Sources and listing historyWhere this information comes from
This listing's history
- First seen here on 19 Aug 2026, and still on Mercor's list when we last read it on 13 Oct 2026.
- 1 recorded pay change is not shown.
A change can also be a correction to how we read Mercor's feed. The figure at the top of this page is always the latest read.
- Posted by Mercor
- First seen by AIPayList
- Last seen live
- Listing ID
- list_AAABlzI3p6G_LDaGWuNNEa6l
Pay as the source states it: “100–120 USD HOUR”.
Platforms sometimes change a rate after posting. Every listing here is re-read at least once a day and corrected when its source moves, so a difference you see on the platform is usually a change made since the last read. How freshness works.
Keep exploring
More like this
Related field, different requirements. Check each role's own page before you apply.