Jobs / Mercor

SWE-Bench Task Auditor

up to $90/hr

source wording: “70–90 USD HOUR”

Platform: Mercor
Posted: 12d ago · seen live 20 min ago
Eligibility: Worldwide
Except 74 countries Mercor names. Show them
Mercor says it cannot hire residents of 86 countries and regions. The full list is on our Mercor page. Checked 2026-09-05. Check the current list.
To start: Account and CV, then an AI interview of about 20 minutes. Mercor says replies take 2 to 4 weeks.Want to know exactly what the process looks like? Step by step for Mercor.

Apply on Mercor ↗

What happens after you apply →

This site earns a referral commission from Mercor. That commission keeps the site running and pays for new features.

Platforms sometimes change a rate after posting. Every listing here is re-read at least once a day and corrected when its source moves, so a difference you see on the platform is usually a change made since the last read. How freshness works.

What this role asks for

Python
TypeScript
Java
C++

Read from the posting's own words: show the exact sentences

Python: “…answer leakage / reward hacking • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++) Preferred Qualifications •…”
TypeScript: “…across common ecosystems (Python and at least one of Java / Go / TypeScript / C++) Preferred Qualifications • Familiarity with SWE-Bench (Verified) or similar…”
Java: “…• Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++) Preferred Qualifications • Familiarity with SWE-Bench…”
C++: “…ecosystems (Python and at least one of Java / Go / TypeScript / C++) Preferred Qualifications • Familiarity with SWE-Bench (Verified) or similar repository…”

The role, as Mercor describes it

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.

Basic Qualifications

• 3+ years professional software engineering

• Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles)

• Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking

• Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)

Preferred Qualifications

• Familiarity with SWE-Bench (Verified) or similar repository benchmarks

Read the rest of the description (2 more paragraphs)

• Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.)

• Prior code-review or task-grading experience

Posted by Mercor, reproduced here so you can judge the role before clicking. Original posting ↗

Apply on Mercor ↗What happens after you apply →

Source: platform job feed · first seen 9d ago · ID list_AAABoEpzuF806Dv0eAZAV5rw

SWE-Bench Task Auditor

The role, as Mercor describes it

More like this