WorkadaCoding & software eval
Software Engineer – AI Agent Evaluation
$100/hr
Candidate background
You're a Great Fit If, 8 of 16 shown
- You hold, or are completing, a PhD in computer science or software engineering or a closely related field, or you have 5+ years of professional experience in it.
- You've done technical or computational work in your field and are comfortable in a Linux terminal: shell commands, scripting in Python or Bash, and installing packages.
- You've used Docker, or can pick it up quickly. We share templates and worked examples.
- You can draw on your own research, projects, code, or files as the source material for your Tasks.
- You have a precise, adversarial eye; you enjoy finding the exact point where a workflow breaks and explaining it cleanly.
- You're honest about uncertainty and write instructions that leave no room for misreading.
- You're based in the US or Canada.
- Academics and practitioners are equally welcome. No AI or machine-learning background is needed. Your expertise and your comfort working in a terminal are what we're after.
The other 8 may also decide whether you qualify.
Show all 16 itemsHide the other 8 in You're a Great Fit If
- Materials You Can Use
- Almost any piece of your work can seed a Task, for example:
- Repositories, schemas, and failing test suites
- CI configs and build scripts
- Data pipelines and the datasets behind them
- Profiler traces, logs, and incident write-ups
- Pull requests and bug fixes from your own work
- Bugs, edge cases, and failure modes you've had to debug yourself
Quoted from selected sections of Workada's posting. The full posting may state other conditions. Read the full posting
A closer look
More context, when you need it.
Full job descriptionThe original posting, in Workada's words
Posted by Workada, reproduced here so you can judge the role before clicking. Original posting ↗
The race conditions, flaky builds, and 2am pipeline failures you've chased down are exactly where AI agents still get stuck. Turn them into tests that the best models can't pass, on your own schedule, from anywhere. Workada connects skilled contributors with meaningful AI work. Your expertise makes AI systems smarter, safer, and more useful for everyone. We're building a community of experts, both PhDs and seasoned practitioners, to test how far today's AI agents can go on real problems, and we want you in it. What You'll Do As a contributor, you'll take problems from your own work and turn each one into a self-contained Task that an AI agent has to solve in a Linux terminal. For every Task, you'll write a clear brief, set up the environment with the files and tools the agent needs, build a reference solution, and write automated checks on the final result. The strongest Tasks are the ones today's best agents still get wrong. Your firsthand knowledge of where real work in your field goes wrong is what makes them valuable. That might be a race condition that only shows up under load, a migration that silently drops rows, a compiler pass that miscompiles an edge case, a consensus protocol that splits on a network partition, or a CI build that flakes because of test ordering. Specialties We're Looking For We're building depth across:
• Algorithms and data structures
• Distributed and operating systems , including concurrency and consensus
• Databases and SQL , including migrations and query planning
• Data engineering : pipelines, ETL, and streaming
• Compilers and programming languages
• Frontend and web
• DevOps, CI/CD, and build systems You're a Great Fit If
• You hold, or are completing, a PhD in computer science or software engineering or a closely related field, or you have 5+ years of professional experience in it.
• You've done technical or computational work in your field and are comfortable in a Linux terminal: shell commands, scripting in Python or Bash, and installing packages.
• You've used Docker, or can pick it up quickly. We share templates and worked examples.
• You can draw on your own research, projects, code, or files as the source material for your Tasks.
• You have a precise, adversarial eye; you enjoy finding the exact point where a workflow breaks and explaining it cleanly.
• You're honest about uncertainty and write instructions that leave no room for misreading.
• You're based in the US or Canada. Academics and practitioners are equally welcome. No AI or machine-learning background is needed. Your expertise and your comfort working in a terminal are what we're after. Materials You Can Use Almost any piece of your work can seed a Task, for example:
• Repositories, schemas, and failing test suites
• CI configs and build scripts
• Data pipelines and the datasets behind them
• Profiler traces, logs, and incident write-ups
• Pull requests and bug fixes from your own work
• Bugs, edge cases, and failure modes you've had to debug yourself
How does the pay compare?Across Coding & software eval roles, not this role alone
From our own daily reads of every open listing on the platforms we follow, not from Workada's posting. How we count.
Pay across the field: Coding & software eval
Across 411 roles in this field with a stated hourly rate in US dollars, on 17 platforms, the middle half pays $60 to $120 an hour at the top of the range, and the median is $80. This role's top rate, $100, is in the middle half. Of the 135 distinct rates we count, 74 are lower and 51 are higher.
| Platform | Roles | Top of range, per hour |
|---|---|---|
| Alignerr | 97 | $50 to $200 · median $100 |
| Anyone AI | 79 | $35 to $150 · median $65 |
| micro1 | 42 | $50 to $400 · median $100 |
| AfterQuery | 37 | $45 to $200 · median $100 |
| Meridial (Invisible) | 35 | $22 to $150 · median $50 |
Sources and listing historyWhere this information comes from
This listing's history
- First seen here on 20 Nov 2026, and still on Workada's list when we last read it on 21 Nov 2026.
- The listed pay has not changed since then.
- Posted by Workada
- First seen by AIPayList
- Last seen live
- Listing ID
- 02a430f9-7974-4488-9293-64563a91279e
Pay as the source states it: “{"@type":"MonetaryAmount","currency":"USD","value":{"@type":"QuantitativeValue","value":100,"unitText":"HOUR"}}”.
Platforms sometimes change a rate after posting. Every listing here is re-read at least once a day and corrected when its source moves, so a difference you see on the platform is usually a change made since the last read. How freshness works.
Keep exploring
More like this
Related field, different requirements. Check each role's own page before you apply.