WorkadaRating & model evaluation
AI Agent Evaluation Expert
$100/hr
Candidate background
You're a Great Fit If, 8 of 21 shown
- You hold, or are completing, a PhD in one of the fields above or a closely related field, or you have 5+ years of professional experience in one of them.
- You've done technical or computational work in your field and are comfortable in a Linux terminal: shell commands, scripting in Python or Bash, and installing packages.
- You've used Docker, or can pick it up quickly. We share templates and worked examples.
- You can draw on your own research, projects, code, or files as the source material for your Tasks.
- You have a precise, adversarial eye; you enjoy finding the exact point where a workflow breaks and explaining it cleanly.
- You're honest about uncertainty and write instructions that leave no room for misreading.
- You're based in the US or Canada.
- Academics and practitioners are equally welcome. Outside of the machine learning domain, no AI or machine-learning background is needed. Your expertise and your comfort working in a terminal are what we're after.
The other 13 may also decide whether you qualify.
Show all 21 itemsHide the other 13 in You're a Great Fit If
- Materials You Can Use
- Almost any piece of your technical work can seed a Task, for example:
- Scripts, code, and pipelines from your research or practice
- Datasets and the processing behind them
- Results, reports, or deliverables you can reproduce from raw sources
- Sequences, instrument data, and simulation configs
- Repositories, schemas, and failing test suites
- Model checkpoints, training code, and evaluation scripts
- Binaries, packet captures, and disk or memory images
- HDL designs, testbenches, and CAD models
- Ledgers, transaction exports, and SQL databases
- Audio, MIDI, notation, and design files
- Bugs, edge cases, and failure modes you've had to debug yourself
Quoted from selected sections of Workada's posting. The full posting may state other conditions. Read the full posting
A closer look
More context, when you need it.
Full job descriptionThe original posting, in Workada's words
Posted by Workada, reproduced here so you can judge the role before clicking. Original posting ↗
Put your expertise to work at the frontier of AI. Bring the expertise you know best and help train AI agents that do real technical work, on your own schedule, from anywhere. You'll turn problems from your own research or professional practice into hands-on challenges an AI agent has to solve at the command line, find exactly where it breaks, and set the expert standard it's measured against. Workada connects skilled contributors with meaningful AI work. Your expertise makes AI systems smarter, safer, and more useful for everyone. We're building a community of experts across science, engineering, business, and creative fields, both PhDs and seasoned practitioners, to test how far today's AI agents can go on real problems, and we want you in it. What You'll Do As a contributor, you'll take problems from your own work and turn each one into a self-contained Task that an AI agent has to solve in a Linux terminal. For every Task, you'll write a clear brief, set up the environment with the files and tools the agent needs, build a reference solution, and write automated checks on the final result. The strongest Tasks are the ones today's best agents still get wrong. Your firsthand knowledge of where real work in your field goes wrong is what makes them valuable. That might be a numerical solver that quietly loses precision, a migration that silently drops rows, a data loader that leaks the test split, a stripped binary hiding its real logic, a clock-domain crossing that passes simulation but fails in silicon, a reconciliation that's off by a currency conversion, or a color profile that shifts on conversion. Specialties We're Looking For If your background is in one of the fields below, there's likely a place for you. We're building depth across seven domains (among others):
• Natural sciences and engineering: biology, chemistry, physics, earth sciences, robotics, math, and linguistics
• Software engineering: algorithms, systems, databases, data engineering, frontend, and programming languages
• Machine learning: training, inference, evaluation, and kernels
• Security: cryptography, reverse engineering, forensics, and application security
• Hardware: CAD and mechanical design, and RTL and digital design
• Business and finance: finance, logistics, supply chain, claims, compliance, and marketing
• Music and design: composition, music technology, and audio production, and graphic, UX/UI, motion, and product design You're a Great Fit If
• You hold, or are completing, a PhD in one of the fields above or a closely related field, or you have 5+ years of professional experience in one of them.
• You've done technical or computational work in your field and are comfortable in a Linux terminal: shell commands, scripting in Python or Bash, and installing packages.
• You've used Docker, or can pick it up quickly. We share templates and worked examples.
• You can draw on your own research, projects, code, or files as the source material for your Tasks.
• You have a precise, adversarial eye; you enjoy finding the exact point where a workflow breaks and explaining it cleanly.
• You're honest about uncertainty and write instructions that leave no room for misreading.
• You're based in the US or Canada. Academics and practitioners are equally welcome. Outside of the machine learning domain, no AI or machine-learning background is needed. Your expertise and your comfort working in a terminal are what we're after. Materials You Can Use Almost any piece of your technical work can seed a Task, for example:
• Scripts, code, and pipelines from your research or practice
• Datasets and the processing behind them
• Results, reports, or deliverables you can reproduce from raw sources
• Sequences, instrument data, and simulation configs
• Repositories, schemas, and failing test suites
• Model checkpoints, training code, and evaluation scripts
• Binaries, packet captures, and disk or memory images
• HDL designs, testbenches, and CAD models
• Ledgers, transaction exports, and SQL databases
• Audio, MIDI, notation, and design files
• Bugs, edge cases, and failure modes you've had to debug yourself
How does the pay compare?Across Rating & model evaluation roles, not this role alone
From our own daily reads of every open listing on the platforms we follow, not from Workada's posting. How we count.
Pay across the field: Rating & model evaluation
Across 156 roles in this field with a stated hourly rate in US dollars, on 18 platforms, the middle half pays $30 to $75 an hour at the top of the range, and the median is $45. This role's top rate, $100, is in the top quarter. Of the 69 distinct rates we count, 60 are lower and 8 are higher.
| Platform | Roles | Top of range, per hour |
|---|---|---|
| Meridial (Invisible) | 44 | $15 to $65 · median $30 |
| micro1 | 28 | $15 to $140 · median $45 |
| Prolific | 18 | $20 to $216 · median $40 |
| Alignerr | 11 | $120 |
| Innodata | 11 | $15 |
Sources and listing historyWhere this information comes from
This listing's history
- First seen here on 5 Nov 2026, and still on Workada's list when we last read it on 21 Nov 2026.
- The listed pay has not changed since then.
- Posted by Workada
- First seen by AIPayList
- Last seen live
- Listing ID
- 384c86f4-3b3c-41de-9a53-69f9e486dfba
Pay as the source states it: “{"@type":"MonetaryAmount","currency":"USD","value":{"@type":"QuantitativeValue","value":100,"unitText":"HOUR"}}”.
Platforms sometimes change a rate after posting. Every listing here is re-read at least once a day and corrected when its source moves, so a difference you see on the platform is usually a change made since the last read. How freshness works.
Keep exploring
More like this
Related field, different requirements. Check each role's own page before you apply.