Anyone AICoding & software eval
Senior Software Engineer – Open Source & SWE-Bench Evaluation
$65/hr
About this role, in Anyone AI's words
Anyone AI is recruiting experienced Software Engineers for a specialized project focused on reviewing and evaluating real-world software engineering tasks derived from open-source repositories.
What you'll do
WHAT YOU’LL WORK ON
- You’ll review software engineering tasks involving:
- Real-world bug fixes and feature implementations
- Open-source repositories and pull requests
- Unit tests and test coverage
Show all 10 itemsHide the other 6 in WHAT YOU’LL WORK ON
- Repository setup and dependency management
- Reproducibility and environment configuration
- Task difficulty and complexity
- Multi-file and cross-module code changes
- Technical feedback and quality assessment
- You’ll determine whether tasks are clearly specified, technically solvable, supported by sufficient tests, and free from issues such as flaky tests, missing dependencies, ambiguous requirements, or environment-specific behavior.
Candidate background
WHAT WE’RE LOOKING FOR
- 3+ years of professional software engineering experience
- Strong experience working with large, multi-file codebases
- Experience reviewing pull requests, debugging issues, and maintaining production code
- Strong understanding of unit testing and test coverage
- Ability to evaluate whether tests correctly validate a solution without unnecessarily restricting implementation approaches
- Experience with dependency management, environment setup, and reproducibility
- Strong understanding of Git and GitHub-based development workflows
- Ability to analyze complex technical problems and provide clear written feedback
NICE TO HAVE
Show all 7 itemsHide these 7 items in NICE TO HAVE
- Contributions to or maintenance of open-source projects
- Experience with SWE-Bench, SWE-Bench Verified, or similar coding benchmarks
- Experience with major Python open-source projects such as Django, Flask, scikit-learn, SymPy, matplotlib, requests, or pytest
- Experience with Docker, CI/CD, pip, conda, or dependency pinning
- Knowledge of test fixtures, test isolation, or property-based testing
- Experience designing technical assessments or reviewing coding challenges
- Experience with AI/ML evaluation, data curation, RLHF, or benchmark development
Quoted from selected sections of Anyone AI's posting. The full posting may state other conditions. Read the full posting
A closer look
More context, when you need it.
Full job descriptionThe original posting, in Anyone AI's words
Posted by Anyone AI, reproduced here so you can judge the role before clicking. Original posting ↗
Anyone AI is recruiting experienced Software Engineers for a specialized project focused on reviewing and evaluating real-world software engineering tasks derived from open-source repositories.
The work involves assessing whether coding tasks based on real GitHub issues and pull requests are technically sound, reproducible, appropriately tested, and representative of the kinds of problems professional software engineers solve every day.
WHAT YOU’LL WORK ON
You’ll review software engineering tasks involving:
- Real-world bug fixes and feature implementations
- Open-source repositories and pull requests
- Unit tests and test coverage
- Repository setup and dependency management
- Reproducibility and environment configuration
- Task difficulty and complexity
- Multi-file and cross-module code changes
- Technical feedback and quality assessment
You’ll determine whether tasks are clearly specified, technically solvable, supported by sufficient tests, and free from issues such as flaky tests, missing dependencies, ambiguous requirements, or environment-specific behavior.
WHAT WE’RE LOOKING FOR
- 3+ years of professional software engineering experience
- Strong experience working with large, multi-file codebases
- Experience reviewing pull requests, debugging issues, and maintaining production code
- Strong understanding of unit testing and test coverage
- Ability to evaluate whether tests correctly validate a solution without unnecessarily restricting implementation approaches
- Experience with dependency management, environment setup, and reproducibility
- Strong understanding of Git and GitHub-based development workflows
- Ability to analyze complex technical problems and provide clear written feedback
NICE TO HAVE
- Contributions to or maintenance of open-source projects
- Experience with SWE-Bench, SWE-Bench Verified, or similar coding benchmarks
- Experience with major Python open-source projects such as Django, Flask, scikit-learn, SymPy, matplotlib, requests, or pytest
- Experience with Docker, CI/CD, pip, conda, or dependency pinning
- Knowledge of test fixtures, test isolation, or property-based testing
- Experience designing technical assessments or reviewing coding challenges
- Experience with AI/ML evaluation, data curation, RLHF, or benchmark development
WHAT YOU’LL BE RESPONSIBLE FOR
- Reviewing coding tasks derived from real GitHub issues and pull requests
- Assessing whether problem statements and success criteria are clear and complete
- Evaluating unit tests for correctness, coverage, and robustness
- Identifying flaky tests, missing dependencies, version conflicts, and environment issues
- Determining whether tasks can be reliably reproduced across environments
- Assessing the real-world difficulty and complexity of each task
- Providing clear recommendations on whether tasks should be accepted, improved, or excluded
ENGAGEMENT
Work Type: Remote Engagement: Part-time, project-based consulting Focus: Software engineering, open-source code review, testing, and technical evaluation
This role is a strong fit for experienced engineers who enjoy debugging complex codebases, reviewing pull requests, working with open-source software, and evaluating what makes a software engineering problem well designed.
How does the pay compare?Across Coding & software eval roles, not this role alone
From our own daily reads of every open listing on the platforms we follow, not from Anyone AI's posting. How we count.
Pay across the field: Coding & software eval
Across 407 roles in this field with a stated hourly rate in US dollars, on 18 platforms, the middle half pays $60 to $120 an hour at the top of the range, and the median is $80. This role's top rate, $65, is in the middle half. Of the 133 distinct rates we count, 42 are lower and 85 are higher.
| Platform | Roles | Top of range, per hour |
|---|---|---|
| Alignerr | 97 | $50 to $200 · median $100 |
| Anyone AI (this platform) | 78 | $35 to $150 · median $65 |
| micro1 | 40 | $50 to $400 · median $100 |
| AfterQuery | 37 | $45 to $200 · median $100 |
| Meridial (Invisible) | 35 | $22 to $150 · median $50 |
Sources and listing historyWhere this information comes from
This listing's history
- First seen here on 15 Nov 2026, and still on Anyone AI's list when we last read it on 20 Nov 2026.
- The listed pay has not changed since then.
- Posted by Anyone AI
- First seen by AIPayList
- Last seen live
- Listing ID
- e80c2c62-6895-41f5-bf9d-89ae5207d2bf
Pay as the source states it: “$65 per hour”.
Platforms sometimes change a rate after posting. Every listing here is re-read at least once a day and corrected when its source moves, so a difference you see on the platform is usually a change made since the last read. How freshness works.
Keep exploring
More like this
Related field, different requirements. Check each role's own page before you apply.