AlignerrCoding & software eval
Senior Software Engineer – AI Evaluation
up to $120/hr
About this role, in Alignerr's words
What if your engineering skills could directly determine whether the world's most advanced AI systems are actually working? We're looking for Senior Software Engineers to design, build, and scale the evaluation infrastructure that measures AI performance — the critical layer between raw model output and real-world trust.
What you'll do
- Design and build scalable evaluation pipelines and frameworks for assessing AI model performance across diverse tasks and domains
- Develop automated testing harnesses, scoring systems, and benchmarking tools for large language models and other AI systems
- Write clean, production-quality code to process, analyze, and visualize evaluation datasets at scale
Show all 9 itemsHide the other 6 in What you'll do
- Create and maintain APIs, dashboards, and internal tools that enable research teams to run, track, and compare evaluations efficiently
- Collaborate with AI researchers and data scientists to translate evaluation methodologies into reliable, repeatable software
- Identify edge cases, failure modes, and reliability issues in AI outputs through systematic engineering approaches
- Optimize system performance, data processing speed, and infrastructure costs
- Contribute to the architecture and technical direction of the evaluation platform
- Write clear documentation and participate in code reviews to maintain high engineering standards
Candidate background
Who You Are
- 4+ years of professional software engineering experience, with a track record of building and shipping production systems
- Strong proficiency in Python — including experience with data processing libraries (pandas, NumPy) and web frameworks (FastAPI, Flask, or Django)
- Solid understanding of software architecture, design patterns, and engineering best practices
- Experience working with large datasets and building data pipelines
- Comfortable with cloud infrastructure (AWS, GCP, or Azure) and containerized deployments
- Familiarity with version control (Git), CI/CD workflows, and testing frameworks
- Strong problem-solving skills and the ability to work through ambiguity independently
- Excellent written communication skills — you can document your work clearly and collaborate asynchronously
- Self-motivated and reliable when working independently in a remote environment
Nice to Have
Show all 7 itemsHide these 7 items in Nice to Have
- Experience with ML/AI evaluation, benchmarking, or model testing
- Familiarity with LLMs, prompt engineering, or AI safety and alignment concepts
- Background in building developer tools, internal platforms, or data infrastructure
- Experience with distributed systems, message queues, or workflow orchestration (Airflow, Prefect, etc.)
- Knowledge of statistical methods for measuring and comparing model performance
- Prior experience in a remote-first or async-first engineering culture
- Contributions to open-source projects related to AI, ML, or evaluation tooling
Quoted from selected sections of Alignerr's posting. The full posting may state other conditions. Read the full posting
A closer look
More context, when you need it.
Full job descriptionThe original posting, in Alignerr's words
Posted by Alignerr, reproduced here so you can judge the role before clicking. Original posting ↗
Senior Software Engineer – AI Evaluation
About the Role
What if your engineering skills could directly determine whether the world's most advanced AI systems are actually working? We're looking for Senior Software Engineers to design, build, and scale the evaluation infrastructure that measures AI performance — the critical layer between raw model output and real-world trust.
This is high-impact, technically challenging work at the intersection of software engineering and AI research. You'll build the tools, pipelines, and frameworks that help leading research teams understand what their models can do, where they fail, and how to make them better. If you love building robust systems and care deeply about quality and measurement, this role puts you at the center of the AI revolution.
- Organization: Alignerr - Type: Hourly Contract - Location: Remote - Commitment: 20–40 hours/week
What You'll Do
* Design and build scalable evaluation pipelines and frameworks for assessing AI model performance across diverse tasks and domains * Develop automated testing harnesses, scoring systems, and benchmarking tools for large language models and other AI systems * Write clean, production-quality code to process, analyze, and visualize evaluation datasets at scale * Create and maintain APIs, dashboards, and internal tools that enable research teams to run, track, and compare evaluations efficiently * Collaborate with AI researchers and data scientists to translate evaluation methodologies into reliable, repeatable software * Identify edge cases, failure modes, and reliability issues in AI outputs through systematic engineering approaches * Optimize system performance, data processing speed, and infrastructure costs * Contribute to the architecture and technical direction of the evaluation platform * Write clear documentation and participate in code reviews to maintain high engineering standards
Who You Are
* 4+ years of professional software engineering experience, with a track record of building and shipping production systems * Strong proficiency in Python — including experience with data processing libraries (pandas, NumPy) and web frameworks (FastAPI, Flask, or Django) * Solid understanding of software architecture, design patterns, and engineering best practices * Experience working with large datasets and building data pipelines * Comfortable with cloud infrastructure (AWS, GCP, or Azure) and containerized deployments * Familiarity with version control (Git), CI/CD workflows, and testing frameworks * Strong problem-solving skills and the ability to work through ambiguity independently * Excellent written communication skills — you can document your work clearly and collaborate asynchronously * Self-motivated and reliable when working independently in a remote environment
Nice to Have
* Experience with ML/AI evaluation, benchmarking, or model testing * Familiarity with LLMs, prompt engineering, or AI safety and alignment concepts * Background in building developer tools, internal platforms, or data infrastructure * Experience with distributed systems, message queues, or workflow orchestration (Airflow, Prefect, etc.) * Knowledge of statistical methods for measuring and comparing model performance * Prior experience in a remote-first or async-first engineering culture * Contributions to open-source projects related to AI, ML, or evaluation tooling
Why Join Us
* Work on cutting-edge AI evaluation projects alongside world-class research labs * Directly influence how AI quality and safety are measured at scale — your code shapes the standard * Fully remote and flexible — work when and where you're most productive * Freelance autonomy with access to deeply meaningful, technically stimulating work * Collaborate with a global team of engineers and researchers pushing the boundaries of AI * Exposure to the latest developments in AI research, model capabilities, and evaluation science * Potential for ongoing work and contract extension as the platform and project scope grow
Working with AlignerrPay, getting started and work, from Alignerr's own pages
What the platform publishes in general. These are not extra terms confirmed for this particular role.
- Getting paid
- Billed weekly, Monday to Sunday, with approved work paid by the following Friday, according to Labelbox's help center. Payment goes through Stripe Connect or PayPal depending on your location, with one provider per country. intercom.help/labelbox/en/articles/13600616-alignerr-paym..., alignerr.com/process
- Before your first paid task
- Sign up, apply, verify your identity, set up billing and onboard. Some jobs add a Zara AI interview or domain assessments. alignerr.com/process
- How much work
- Time to your first project depends on demand and your skills, and there is no minimum number of hours. alignerr.com/faqs
- Contract
- Independent contractor, under terms with ALIGNERR LLC. alignerr.com/faqs, alignerr.com/privacy
Read on the pages linked above on 24 Sep 2026. Platforms change these terms, so check the linked page before you rely on one.
How does the pay compare?Across Coding & software eval roles, not this role alone
From our own daily reads of every open listing on the platforms we follow, not from Alignerr's posting. How we count.
Pay across the field: Coding & software eval
Across 391 roles in this field with a stated hourly rate in US dollars, on 14 platforms, the middle half pays $65 to $120 an hour at the top of the range, and the median is $95. This role's top rate, $120, is in the top quarter. Of the 140 distinct rates we count, 96 are lower and 31 are higher.
| Platform | Roles | Top of range, per hour |
|---|---|---|
| Alignerr (this platform) | 136 | $50 to $400 · median $100 |
| micro1 | 44 | $60 to $350 · median $120 |
| Handshake AI | 43 | $47 to $125 · median $80 |
| Meridial (Invisible) | 34 | $22 to $150 · median $60 |
| Mercor | 33 | $60 to $200 · median $105 |
Sources and listing historyWhere this information comes from
This listing's history
- First seen here on 27 Aug 2026, and still on Alignerr's list when we last read it on 13 Oct 2026.
- The listed pay has not changed since then.
- Posted by Alignerr
- First seen by AIPayList
- Last seen live
- Listing ID
- cfa052f0-e721-4ec5-8d38-d697e534081b
Pay as the source states it: “$60-120/hr”.
Platforms sometimes change a rate after posting. Every listing here is re-read at least once a day and corrected when its source moves, so a difference you see on the platform is usually a change made since the last read. How freshness works.
Keep exploring
More like this
Related field, different requirements. Check each role's own page before you apply.