AlignerrCoding & software eval
Senior Software Engineer: AI Evaluation & Benchmarks
up to $100/hr
About this role, in Alignerr's words
What if the code you write could determine how smart the next generation of AI truly is? We're looking for experienced Software Engineers to design and build the coding benchmarks and data pipelines used to evaluate frontier AI models — the systems that decide whether an AI can actually reason, debug, and write production-quality software.
What you'll do
- Design and implement coding benchmarks used to evaluate frontier AI models across real-world programming tasks
- Build and maintain scalable data pipelines for AI evaluation workflows
- Analyze AI-generated code for correctness, reliability, and edge-case failures
Show all 8 itemsHide the other 5 in What you'll do
- Create structured evaluation scenarios that rigorously test reasoning, debugging, and code quality
- Work with large code repositories and multi-language environments
- Collaborate on systems that improve how AI models understand and generate software
- Provide detailed technical feedback on model performance and failure patterns
- Contribute to the design of evaluation frameworks that set industry standards
Candidate background
Who You Are
- 4+ years of professional software engineering experience — this is non-negotiable
- Experience working at a high-growth tech company or top-tier software organization
- Expert proficiency in Python — you write clean, performant, well-tested Python code
- Hands-on experience with code repositories and working in large, complex codebases
- Proven experience designing and implementing LLM coding benchmarks and data pipelines
- Track record of working in high-performance engineering environments with large-scale products or platforms
- Strong command of version control systems (Git) and modern development workflows
- Bilingual or native English speaker with strong written communication skills
- Self-directed, technically rigorous, and comfortable operating with autonomy
Quoted from selected sections of Alignerr's posting. The full posting may state other conditions. Read the full posting
A closer look
More context, when you need it.
Full job descriptionThe original posting, in Alignerr's words
Posted by Alignerr, reproduced here so you can judge the role before clicking. Original posting ↗
Senior Software Engineer — AI Evaluation & Benchmarks
About the Role
What if the code you write could determine how smart the next generation of AI truly is? We're looking for experienced Software Engineers to design and build the coding benchmarks and data pipelines used to evaluate frontier AI models — the systems that decide whether an AI can actually reason, debug, and write production-quality software.
This is high-impact, technically demanding work at the intersection of software engineering and AI research. You'll work with large codebases, multiple programming languages, and scalable infrastructure to create evaluation systems that push the boundaries of what AI can do.
This is a fully remote contract role. If you thrive in fast-paced engineering environments and want your work to directly shape the trajectory of AI — this is the role.
- Organization: Alignerr - Type: Hourly Contract - Location: Remote - Contract Length: 3 Months - Commitment: Full-time availability preferred
What You'll Do
* Design and implement coding benchmarks used to evaluate frontier AI models across real-world programming tasks * Build and maintain scalable data pipelines for AI evaluation workflows * Analyze AI-generated code for correctness, reliability, and edge-case failures * Create structured evaluation scenarios that rigorously test reasoning, debugging, and code quality * Work with large code repositories and multi-language environments * Collaborate on systems that improve how AI models understand and generate software * Provide detailed technical feedback on model performance and failure patterns * Contribute to the design of evaluation frameworks that set industry standards
Who You Are
* 4+ years of professional software engineering experience — this is non-negotiable * Experience working at a high-growth tech company or top-tier software organization * Expert proficiency in Python — you write clean, performant, well-tested Python code * Hands-on experience with code repositories and working in large, complex codebases * Proven experience designing and implementing LLM coding benchmarks and data pipelines * Track record of working in high-performance engineering environments with large-scale products or platforms * Strong command of version control systems (Git) and modern development workflows * Bilingual or native English speaker with strong written communication skills * Self-directed, technically rigorous, and comfortable operating with autonomy
What Makes a Perfect Match
Candidates with these additional qualifications have the highest chance of success:
* Senior or Lead-level engineering profiles with a history of technical ownership * Bachelor's or Master's degree in Computer Science, Machine Learning, or a related field — or equivalent professional experience * Proficiency in one or more additional languages: JavaScript, Go, C++, or other relevant languages * Experience with CI/CD pipelines and writing robust unit tests (pytest, Mocha, JUnit) * Background in security engineering or significant open-source contributions * Familiarity with AI/ML evaluation methodologies or model benchmarking
Why Join Us
* Work on cutting-edge AI evaluation projects alongside world-class research teams * Fully remote — work from anywhere with a reliable internet connection * Your benchmarks directly influence how the most advanced AI systems in the world are measured and improved * Freelance autonomy with meaningful, high-stakes engineering work * Collaborate with a global community of elite engineers and researchers * Potential for contract extension and ongoing engagement as new evaluation challenges emerge
Working with AlignerrPay, getting started and work, from Alignerr's own pages
What the platform publishes in general. These are not extra terms confirmed for this particular role.
- Getting paid
- Billed weekly, Monday to Sunday, with approved work paid by the following Friday, according to Labelbox's help center. Payment goes through Stripe Connect or PayPal depending on your location, with one provider per country. intercom.help/labelbox/en/articles/13600616-alignerr-paym..., alignerr.com/process
- Before your first paid task
- Sign up, apply, verify your identity, set up billing and onboard. Some jobs add a Zara AI interview or domain assessments. alignerr.com/process
- How much work
- Time to your first project depends on demand and your skills, and there is no minimum number of hours. alignerr.com/faqs
- Contract
- Independent contractor, under terms with ALIGNERR LLC. alignerr.com/faqs, alignerr.com/privacy
Read on the pages linked above on 24 Sep 2026. Platforms change these terms, so check the linked page before you rely on one.
How does the pay compare?Across Coding & software eval roles, not this role alone
From our own daily reads of every open listing on the platforms we follow, not from Alignerr's posting. How we count.
Pay across the field: Coding & software eval
Across 393 roles in this field with a stated hourly rate in US dollars, on 14 platforms, the middle half pays $65 to $120 an hour at the top of the range, and the median is $95. This role's top rate, $100, is in the middle half. Of the 140 distinct rates we count, 73 are lower and 54 are higher.
| Platform | Roles | Top of range, per hour |
|---|---|---|
| Alignerr (this platform) | 136 | $50 to $400 · median $100 |
| micro1 | 46 | $35 to $350 · median $110 |
| Handshake AI | 43 | $47 to $125 · median $80 |
| Meridial (Invisible) | 34 | $22 to $150 · median $60 |
| Mercor | 33 | $60 to $200 · median $105 |
Sources and listing historyWhere this information comes from
This listing's history
- First seen here on 27 Aug 2026, and still on Alignerr's list when we last read it on 13 Oct 2026.
- The listed pay has not changed since then.
- Posted by Alignerr
- First seen by AIPayList
- Last seen live
- Listing ID
- 51c15c59-a780-4d32-a969-b4e5ea7ee6a3
Pay as the source states it: “$80-100/hr”.
Platforms sometimes change a rate after posting. Every listing here is re-read at least once a day and corrected when its source moves, so a difference you see on the platform is usually a change made since the last read. How freshness works.
Keep exploring
More like this
Related field, different requirements. Check each role's own page before you apply.