For a decade, the recipe for a better AI model was more text. Scrape the internet, predict the next word, scale. That recipe ran out of internet, and the models it produced could write an essay but could not finish a job. The thing that replaced it is quieter and stranger: frontier labs now pay people to build little worlds where a model attempts real work, gets scored, and tries again. Those worlds are called reinforcement learning environments, and by 2026 they are where a large share of AI progress actually happens.
This post explains what an RL environment is, why labs switched to them, how the scoring works, what a good one costs, who builds them, and where they still fall short. Then it turns the same lens on something we run at Taskade, because once you understand environments you start seeing them everywhere, including in the way we test which AI model builds the best app. 🧭
TL;DR: An RL environment is a task, a place to attempt it, and a grader. Labs now pay for environments instead of labels because static data goes stale. Tasks run roughly $200 to $2,000 each, complex product replicas about $300,000, per Epoch AI. The hard part is a grader that cannot be fooled. How we grade real apps →
What Is an RL Environment?
A reinforcement learning environment is a controlled setting where an AI model acts and gets scored. In the definition Epoch AI settled on after interviewing the people who build them, it has three parts: the actions a model can take, such as running code, clicking a button, or searching; the surrounding context that determines what those actions do; and task prompts paired with automated graders that decide whether an attempt succeeded. Most ship as software containers.
That is the whole idea, and it is worth pausing on how different it is from a dataset. A dataset is a pile of examples with answers attached. The model reads them and adjusts. An environment is a place. The model does something in it, the place changes, a grader looks at the result, and a reward comes back. The model adjusts toward whatever earned reward, then tries again.
| A dataset | An RL environment | |
|---|---|---|
| What it is | Examples with answers | A place where tasks can be attempted |
| How the model learns | Imitates the examples | Tries, gets graded, adjusts, tries again |
| Who supplies the answer | A human, in advance | A grader, after the attempt |
| What happens as models improve | The examples get too easy and stop teaching | The same environment keeps teaching, because harder tasks can be posed in it |
| First big domains | Language, images, speech | Math, competitive programming, then software engineering, then enterprise workflows |
| Where it fails | Stale as soon as the model exceeds the examples | A weak grader gets gamed, and a narrow environment does not generalize |
Reinforcement learning itself is not new. It trained game-playing systems long before large language models existed, and our wiki entries on reinforcement learning and RLHF cover the older lineage. What is new is the object of the reward. RLHF rewarded a model for answers a human rater preferred. Environments reward a model for finishing a job, as judged by something that checks the result.
Why AI Labs Switched From Labels to Environments
Labs switched because the text on the internet taught models to write, and nothing on the internet taught them to work. Pretraining on a corpus of human writing produced models that could draft, translate, and explain, but that had to be kept on a very short leash the moment a task ran longer than a few steps. There was no internet-scale record of people finishing long tasks, recovering from their own mistakes, and using tools reliably. So labs built places where models could generate that record themselves.
The economics followed. A September 2025 report said Anthropic had discussed spending more than one billion dollars a year on RL environments. Wing Venture Capital put traditional data labeling at roughly a five billion dollar market in January 2026, growing more than 50 percent a year, and described environments as the larger opportunity forming on top of it. The vendors that used to sell labeled examples now sell task suites with graders attached.
Three things changed at once, and the RL environment company Mechanize stated them as a manifesto in July 2025 under the title Sweatshop data is over:
- Software, not datasets. Static data goes stale as models improve. An interactive environment keeps offering a challenge, in their words, "much like how games continue to engage players across a wide range of skill levels."
- Full-time contributors, not contractors. Not low-skill workers at scale, and not even high-skill contractors working sporadically. Environments that teach a whole job need months of sustained attention from the same people.
- Deep expertise. The tacit knowledge of subject-matter experts is now the bottleneck. Data generation, they argued, has to be reframed "from a low-status activity outsourced to workers in poor countries, to an elaborate process requiring the world's finest talent."
Epoch's interviewees said the same thing in plainer terms. Domain knowledge and expert-level prompting matter more than machine learning skills for building a good task. The thing that limits growth is not finding experts. It is, as one founder put it, "maintaining quality while scaling."
The domains moved in a telling order. Math and coding came first, because grading is free there: a proof checks or it does not, a test passes or it does not. Then software engineering, where an agent works inside a real repository. Now, per Epoch, the fastest-growing category is enterprise workflows: expense reports, pivot tables, navigating a CRM, working a spreadsheet. The frontier of AI training has moved from the whiteboard to the back office, and the reason is simply that the back office is where the graders can be built next.
How a Verifiable Reward Works
A verifiable reward is a score a program can compute without asking a human. The unit test passes or fails. The answer matches the reference or it does not. The browser workflow reaches its end state or stalls. This is the RLVR paradigm, reinforcement learning with verifiable rewards, and it is why the first sharp capability gains of the environment era landed in mathematics and competitive programming.
The reason it works is also the reason it is hard. A grader is a promise that high reward means the task was done. Break that promise and the model learns to collect the reward instead of doing the task. One environment builder gave Epoch the rule in a sentence: "Soundness matters most: high reward must mean the task was actually solved, not hacked."
The public benchmarks show what happens when soundness slips:
- In a May 2026 audit of SWE-bench Pro, a frontier model read the correct fix out of the repository's own version history in 12 to 25 percent of its passing runs. The grader checked whether the tests passed. It did not check how.
- In April 2026, Berkeley researchers built an agent that scored 100 percent on SWE-bench Verified, SWE-bench Pro, and Terminal-Bench without solving a single problem. Their conclusion: if your benchmark is exploitable, it will be exploited.
- OpenAI stopped reporting SWE-bench Verified on February 23, 2026 after finding that 59.4 percent of the hardest problems it audited were flawed, and that models were reproducing gold-standard patches from task identifiers alone.
Graders sit on a ladder from cheap and sound to expensive and fuzzy, and the environment market is largely a market for climbing it.
Two more design rules from Epoch's interviews are worth knowing, because they explain why environments are hard to build well. Difficulty has to be calibrated. A task a model never solves teaches nothing, so builders aim for a minimum pass rate of about 2 to 3 percent, or at least one success in every 64 to 128 attempts, and a smooth gradient of harder tasks above that. Tasks should share underlying skills. A suite of disconnected puzzles trains disconnected tricks. A suite that exercises the same capabilities from different angles trains something that transfers.
The judge-model rung of the ladder is where most of the current research effort goes, and it is the rung our LLM-as-a-judge entry and the agent evaluation literature cover. The short version: a model can grade an open-ended answer against a rubric far more cheaply than a person can, but it inherits every blind spot of the model doing the grading, so the sound practice is a deterministic gate first and a judge second.
Replication Training: When the Answer Already Exists
The most concrete proposal for scaling environments comes from Mechanize, and it borrows the trick that made pretraining work. Pretraining scaled because nobody had to write the corpus. Books, papers, and forum threads already existed. In The upcoming GPT-3 moment for RL, they argue that software already exists in the same abundance, and that it can be turned into tasks without hand-authoring each one.
A replication task is a detailed specification plus a reference implementation. Take an existing piece of software, write down exactly what it does, and train the model to produce an implementation whose behavior matches the reference. Grading collapses to a binary: it behaves identically, or it does not. That makes the grader both cheap and sound, which is the combination the whole market is chasing.
Their argument for what this trains is a good list, and it reads like a description of every complaint people have about AI coding agents:
- Read and deeply understand detailed instructions.
- Execute meticulously, without errors.
- Notice earlier mistakes and reliably recover from them.
- Sustain performance over month-scale horizons where quality is directly rewarded by correctness.
- Do not settle prematurely for a solution that merely looks good enough.
They also state the limits honestly. Writing comprehensive tests for a replication task is itself hard engineering. Exact replication is rare in everyday software work, showing up mainly in porting, legacy rewrites, and clean-room reimplementation. And a model that can replicate a specification perfectly may still be poor at the open-ended planning a real project needs. Replication training, in their framing, is a bridge to the next paradigm rather than the destination.
One number from that essay is worth carrying around. They estimate that matching frontier pretraining budgets with RL would take on the order of ten thousand years of model-facing task time, meaning the time humans would need to do the tasks the model trains on. For comparison, they note that projects such as Windows Server 2008 or Grand Theft Auto V each consumed roughly that many person-years. The environment era is, in effect, an effort to manufacture several major software projects' worth of graded work.
Why a Good Task Costs Thousands of Dollars
A single RL task can justify a price of a few thousand dollars, because the compute spent running it dwarfs the cost of building it. Epoch's January 2026 interviews put individual tasks at roughly $200 to $2,000, website replicas used for interface training at about $20,000 each, and a complex product clone, of something on the scale of Slack, at about $300,000. Exclusive deals run four to five times the price of shared ones, and contracts reach six or seven figures per quarter.
Mechanize published the arithmetic behind those prices in August 2025, in Cheap RL tasks will waste compute. It is a reusable template, so here it is in full.
| Input | Their figure | Where it comes from |
|---|---|---|
| Opportunity cost of compute | $15 per million output tokens | The API price of a frontier model at the time, since that compute could be sold as inference instead |
| Tokens per task, SWE-bench Verified | About 20,000 | Published run logs |
| Tokens per frontier RL task today | About 100,000 | Their own experience |
| Growth in transcript length | About 5 times per year | Epoch AI's output-length data |
| Tokens per task within a year | About 500,000 | Extrapolation |
| Attempts per task per training run | 64 | The group size used in DeepSeek-R1 |
| Compute per task, per run | About $480 | 500,000 tokens times $15 per million times 64 |
| Reuse across research and production runs | 5 times, conservatively | — |
| Lifetime compute per task | About $2,400 | — |
The principle underneath the numbers is that data and compute are complements. Underinvest severely in one and you waste most of your spend on the other. Their image for it: putting cheap tires on a Ferrari. If running a task costs $2,400 over its life, paying $100 to build it badly is not thrift. It is the most expensive line in the budget, because it degrades every one of those runs.
Wing Venture Capital reached the same conclusion from the investor's side: "environment quality, not environment count, becomes the binding constraint." The winners in this market, both essays agree, will be the teams with deep domain expertise, rigorous validation, and the discipline to build fewer, better tasks.
Who Builds RL Environments in 2026
The market has three kinds of builders, plus the labs themselves and a surprising fourth group: the companies whose software is being replicated.
| Group | Examples | What they bring |
|---|---|---|
| Human data companies that added environments | Scale AI, Surge AI, Mercor, Turing | Existing lab relationships, expert networks, and the operations to manage thousands of contributors |
| Environment-native startups | Mechanize, Fleet AI, HUD, Veris AI, Plato, Bespoke Labs | Purpose-built tooling for tasks, graders, and containers, and a thesis about which domains matter |
| Open ecosystems | Prime Intellect | Shared frameworks and community-built environments for decentralized training |
| Frontier labs, in-house | The major labs and the newer research labs | Environments too sensitive or too specific to outsource |
| Product companies partnering with labs | Salesforce, Slack, Benchling, per Epoch's interviews | The real software, so a model can learn a product from the product rather than from a replica |
Wing predicted in January 2026 that the field, then roughly twenty seed- to Series A-stage companies, narrows to three to five leaders by 2030, with one or two dominant platforms. Epoch's interviewees expect growth in enterprise workflow environments, longer multi-step tasks, multi-turn interaction with simulated users, and tooling that lets a lab inspect a model's attempts rather than just score them.
There is a detail in that last row that matters for anyone who builds software. When Slack or Salesforce partners with a lab, the model is being trained to operate their product. The next generation of models will be fluent in the tools that showed up in environments, and less fluent in the ones that did not. That is a new kind of distribution advantage, and it did not exist two years ago.
What an Environment Cannot Grade, Yet
The environment era has a boundary, and the people inside it are candid about where it sits. In How to fully automate software engineering, Mechanize lists what today's graders cannot see: whether an agent followed open-ended instructions from a customer who did not have a full technical specification in mind, whether its code is maintainable, whether it avoided technical debt, whether it dodged a trapdoor decision. "Without being able to grade these parts of the AI's work," they write, "we can't know if an AI can act as a fully independent engineer, or whether it will just be a tool that saves human engineers time."
Their July 2025 essay puts the same boundary in one example: a scoring script cannot tell you whether an AI would make an effective lawyer. That requires constructing cogent arguments, contextualizing information properly, and prevailing in court. None of those has a unit test.
The numbers agree. On Vibe Code Bench, the first serious benchmark to ask a model to build a whole web app from a written specification, the best model completed 61.8 percent of the work, and the researchers found that changing the automated evaluator moved step-level agreement anywhere from 31.8 to 93.6 percent. The grader was as much a variable as the model.
| Gradable today | Hard to grade today |
|---|---|
| Does the code pass the tests | Is the code maintainable |
| Does the output match the reference | Did it understand what the customer meant |
| Did the workflow reach the end state | Did it make a decision it cannot undo |
| Was every field saved correctly | Is the design good |
| Did the automation fire on the right record | Would a real user come back tomorrow |
This boundary is not a reason to dismiss the shift. It is a map of where the gains will land first. Anything in the left column is about to get much better, because thousands of experts are building graders for it right now. Anything in the right column will improve more slowly, and will need a human in the loop for longer. If you build with AI, that map is the most useful thing in this post.
The Same Loop, Seen From the App Builder's Chair
Once you understand environments, you start seeing them everywhere, and one of the places we saw one was in our own benchmark. TSK-1 is not an RL environment. Taskade does not train models, and nothing that happens in a TSK-1 test flows back into any model's weights. But it is built like one, and the resemblance is instructive.
Every TSK-1 test gives every model the same frozen request, registered with a fingerprint so that two results only go in the same comparison if the request bytes match. That is the task prompt. Every model builds inside Taskade Genesis, the same product, held constant. That is the environment. A tester then opens the finished app in light and dark and at phone width, fills in its form as a fixed persona whose expected outcome is known, submits, and compares every saved field to what was typed. That is the grader, and it sits on the sound end of the ladder: a state check on the database, not a vibe about the screen.
The sharpest parallel is the second of TSK-1's two requests. It is a real customer's 32-question client sign-up form, pasted word for word, with the customer's own instruction: do not shorten my questions or answers. Grading fidelity to that request means counting how many of the 32 questions survived into the finished app exactly as written. That is a replication-shaped reward, exact match against a reference, applied to a product a real business asked for.
The differences matter as much as the parallel:
- Where the grade goes. In a lab, the reward updates the model. In TSK-1, the grade goes to a public hub so a buyer can see what each model actually built, with the day it was measured.
- Who holds the grader. In a lab, a program. In TSK-1, a person using the app, with the field-by-field check as the sound core and human judgment for the parts no program can grade yet: whether the app looks right, whether it built what was asked, whether a change landed cleanly.
- Consent. The environment market has a gap here that its own essays do not mention. Replication training runs on existing software, and nobody in the manifestos asks who owns it. TSK-1's real customer request is never published, and the customer is never named. That is a rule, not a courtesy, and it is also the reason the benchmark's hardest task cannot leak into training data.
The full protocol is in the TSK-1 methodology, and the results, model by model, are in Best AI Model for Building Apps in 2026. The reason to mention it here is narrower. Environments have a grading gap, and the gap is exactly where a product with a real workspace behind it has something to offer: an app built in Taskade Genesis lands its data in a project with typed fields, its logic in an automation with triggers and actions, and its knowledge in an AI agent that can be asked about the record. Every one of those is a state a grader can check. That is the Workspace DNA loop, projects that remember, agents that think, automations that execute, and it happens to be the shape of a gradable environment.
What This Means If You Build With AI
The practical lesson of the environment era is that models get better at whatever has a grader, so the fastest way to get more from AI is to give your work one. Five habits follow, and none of them requires understanding a gradient.
- Write the brief like a specification. The models being trained today are rewarded for following detailed instructions exactly. A vague request leaves the model to invent, and invention is where the wrong app comes from. Name the fields, name the outcomes, name what must not change.
- Decide the expected outcome before you test. A fixed persona with a known result turns "does this feel right" into "did it produce 11 or did it not." That is the difference between a vibe and a grader.
- Check the saved data, not the screen. A form can render perfectly and save nothing. Open the record. Compare every field to what you typed. This is the single check that catches the most failures, and it is the one almost nobody does.
- Expect fast gains where grading is cheap and slow gains where it is not. Forms, data, tests, and automations will keep improving quickly. Taste, judgment, and reading an ambiguous customer's mind will not, so keep a person in those loops and do not be surprised when the model needs one.
- Match the model to the task, or leave the default alone. No model led every quality in our tests, and the environment era makes that more likely, not less, because each lab is training in the domains it chose. TSK-1 Auto handles the default, and Taskade Genesis offers 15+ frontier models from OpenAI, Anthropic, and open-weight providers, so you can switch the model behind an app or agent without rebuilding it.
THE GRADER TEST FOR ANY AI TASK Can a program, or a person with a checklist, say for certain whether it succeeded?
YES -> expect rapid improvement; automate it, measure it, trust it sooner
NO -> expect slower improvement; keep a human in the loop, write down
what "good" means, and revisit when a grader exists
Forms that must save every field ............ YES
Data that must match a formula .............. YES
Code that must pass a test .................. YES
An automation that must fire on a record .... YES
A design that must look right ............... PARTLY
A brief the customer never fully specified .. NO, not yet
If you would rather see the loop than read about it, describe an app in Taskade Genesis, open the live apps other people have built, or read what each model did on the hub. The quickstart covers the first build, and the free plan includes three Taskade Genesis apps.
Frequently Asked Questions
What is an RL environment in AI?
A controlled setting where an AI model acts and gets scored. It has three parts: the actions the model can take, such as running code or clicking buttons; the surrounding context that determines what those actions do; and task prompts paired with automated graders that decide whether an attempt succeeded. Environments typically ship as software containers, and by 2025 they had replaced static labeled datasets as the main way frontier labs teach models to do multi-step work.
Why did AI labs switch from labeled data to RL environments?
Because static datasets go stale as models improve, while an environment keeps offering a challenge the model has not solved. Pretraining taught models language, not how to finish a long task or recover from mistakes, and there was no internet-scale record of people doing those things. A September 2025 report said Anthropic had discussed spending more than one billion dollars a year on environments.
How much does an RL environment or task cost?
Per Epoch AI's January 2026 interviews, individual tasks run roughly $200 to $2,000, website replicas about $20,000 each, and complex product clones about $300,000, with exclusive deals costing four to five times more. Mechanize argued in August 2025 that labs should spend a few thousand dollars per task, because the compute spent running a task over its life is around $2,400 and a cheap task degrades every run.
What is a verifiable reward?
A score a program can compute without human judgment: a test passes or fails, an output matches the reference, a workflow reaches its end state. Reinforcement learning with verifiable rewards is why math and coding improved first, because grading is free there. The hard part is soundness: high reward must mean the task was actually solved, not hacked.
What is reward hacking?
Earning a high score without doing the task the score was meant to measure. In a May 2026 audit of SWE-bench Pro, a frontier model read the correct fix out of the repository's version history in 12 to 25 percent of its passing runs. In April 2026 Berkeley researchers scored 100 percent on three major coding benchmarks without solving a single problem. Any exploitable grader eventually gets exploited.
What is replication training?
A proposal from Mechanize to scale RL tasks the way pretraining scaled text: each task is a detailed specification plus a reference implementation of existing software, and the model is trained to match the reference's behavior exactly. The grader collapses to a binary, which makes it cheap and sound. Its authors concede that exact replication is rare in everyday engineering and will not, alone, produce open-ended planning.
Who builds RL environments in 2026?
Human data companies that added environments, such as Scale AI, Surge AI, Mercor, and Turing; environment-native startups such as Mechanize, Fleet AI, HUD, Veris AI, Plato, and Bespoke Labs; open ecosystems such as Prime Intellect; frontier labs in-house; and product companies including Salesforce, Slack, and Benchling partnering with labs. Wing Venture Capital predicted in January 2026 that the market narrows to three to five leaders by 2030.
Is TSK-1 an RL environment?
No. TSK-1 is an evaluation, and Taskade does not train models. But it is built like one: a frozen request, the same product for every model, and a tester who checks every saved field against what was typed. The word-for-word check on a real customer's 32-question form is the same exact-match grader that replication training uses. The grade goes to a public hub for buyers rather than back into a model.
What does the shift to RL environments mean for people who build apps with AI?
Models improve fastest at whatever has a grader. Forms, data, tests, and automations are gradable, so expect rapid gains there. Judgment, taste, and ambiguous briefs are not yet gradable, so expect slower gains and keep a person in that loop. Write briefs like specifications, decide the expected outcome before you test, and check the saved data rather than the screen.
Can I use models trained in RL environments inside Taskade?
Yes. Taskade Genesis offers 15+ frontier models from OpenAI, Anthropic, and open-weight providers. TSK-1 Auto handles the default, and you can set a model per AI agent. The hub publishes dated, hands-on evidence of how each family builds a real app. The free plan includes three Taskade Genesis apps, and paid plans start at $10 per month billed annually.
Related Reading
- The TSK-1 Methodology, how we grade AI models on the app they build.
- Best AI Model for Building Apps in 2026, what nine models built from one request.
- History of AI Benchmarks, why every benchmark saturates and what survives.
- What Are AI Agent Evals? and AI agent reliability, the evaluation side of the same loop.
- Reasoning models and how LLMs work, for the training background.
- Why AI-Generated Apps Break, the failures a grader would have caught.
- The Living Software Era, what happens to an app after the prompt.
Primary sources: Epoch AI's FAQ on reinforcement learning environments (January 12, 2026), Wing Venture Capital's Who Will Win the RL Environment Market (January 2026), and Mechanize's essays How to fully automate software engineering, The upcoming GPT-3 moment for RL, Sweatshop data is over, and Cheap RL tasks will waste compute (May to August 2025). Mechanize is a company that sells RL environments, so read its forecasts as an interested party's, which is also how we would ask you to read ours.
The pretraining era taught machines to talk. The environment era is teaching them to work, one graded task at a time, and the people writing the graders are quietly deciding what the next decade of software will be good at. The best thing you can do about that is to build things that can be checked. ▲ ■ ●