RL environments for LLMs: how they're built, scored and scaled

Why environments matter

Reinforcement learning (RL) has become the main driver of capability gains for reasoning models and agents: it's where supervised fine-tuning hits a ceiling and RL keeps lifting performance past it. And its core piece is the environment: the place where a model practices, gets graded, and learns from interaction over long horizons.

To hit capability targets, environment counts have exploded:

• Qwen3 trained on roughly 20 general-domain tasks.
• Qwen3-Coder pushed that to 20,000 parallel environments on Alibaba Cloud.
• MiniMax's Forge trains M2.5 across hundreds of thousands of real-world environments.
• Qwen3.5 reports million-agent environments with progressively more complex task distributions.

The Qwen team is blunt about it in the Qwen3.5 release notes: most of the gain over Qwen3 comes from “extensive scaling of virtually all RL tasks and environments we could conceive”, deliberately raising their difficulty and generality instead of optimizing for narrow benchmarks.

So the bottleneck is no longer “can I set up an environment?” but “how do I run 100,000 of them, keep them honest and feed them into a training loop?”. Frameworks are emerging to standardize that, alongside catalogs of ready-made environments: the largest sits on Hugging Face Spaces, with over 4,000 MCP-compatible environments, and Prime Intellect's Environments Hub and openreward.ai add several thousand more.

The authors' conclusion, up front: each framework does the same thing its own way. The differences are about how the environment wires into the rest of training, not what it can ultimately do. The same environment can be replicated in all six, so you won't miss anything fundamental by picking one over another: what changes is convenience, and that depends on what's already in your stack. All the code lives in their RL_Envs_101 repo.

What an RL environment is in the LLM era

In classical RL (Atari, robotics), an environment is small and self-contained. The textbook example is CartPole: a cart on a track with a pole balanced on top. The agent observes four numbers (cart position and velocity, pole angle and angular velocity), picks one of two actions (push left or right), and the environment ticks one physics step forward. The reward is +1 for every step the pole stays upright. The environment is the physics simulator plus the reward rule.

With language models the picture gets more complex, and there's no canonical implementation. One common shape: the agent is the language model; the environment is a sandbox that runs shell commands or executes code; and the action space is whatever tools the framework exposes. Each rollout is a multi-turn conversation: the model writes, runs a tool, reads the output, decides what to try next, and eventually submits an answer. The environment scores the completed rollout and returns a reward, and each training step pulls in a group of rollouts per task to learn from.

Many other shapes exist (single-turn reasoning, web tasks, code-repo agents, multi-agent setups), but they're all variations on the same skeleton. What changes is the tools, the observations and the reward rule.

The five stages, and who covers what

Every RL training system for LLMs sits on the same five-stage spine: tasks to solve, a harness that lets the model interact, a reward that scores the behavior, a rollout collector that gathers complete episodes, and a trainer that turns those episodes into a policy update. What changes from framework to framework is which slice of that spine comes in the box:

OpenEnv gives you the harness (the tools the model can call) and the reward, but you write the trainer yourself. Verifiers, on its own, describes the task, the tools, the harness and the reward; the actual training is handled by Prime RL, which it pairs with. That's why a flat “X vs Y” comparison is hard: each was built to slot into a different ecosystem, and they don't live at the same level of the stack. For a complementary cut, sync versus async training, the authors point to Hugging Face's async RL training landscape.

The eleven components of an environment

After enough time inside all six frameworks, the same pieces kept showing up: the ones you actually have to think about when you build or pick an environment for an agent. Each answers a question:

Why frameworks exist, and the six they picked

Why have environment frameworks at all? Mostly for standardization. If there's an agreed protocol for how a trainer talks to an environment, like MCP for tools, any training loop can plug into any environment, researchers across different domains follow the same shape, and someone else's environments become reusable for your run instead of one-off scripts.

There's no standard protocol yet, so each framework answers four questions its own way:

1. What is an environment? For some, just a reward function; for others, also tools, state and the full multi-turn loop; others bundle the whole training pipeline.
2. Where does it run? As an HTTP server (Docker, HF Spaces), scaling independently from training, or in-process inside the training venv, with no network hop but no isolation either.
3. How much trainer comes with it? Some ship their own (Prime RL, NeMo RL, SkyRL); others need adapters to plug into external loops like TRL.
4. When does the reward fire? Per tool call, per step with a rubric, in a post-episode verify, or in an external scoring function.

Out of 16 they surveyed, they picked six to build the same environment in and compare head to head:

The 16 split into three tiers:

• Tier 1, pure task libraries: problems and verifiers, with no transport, tools or state (RLVE, Reasoning Gym).
• Tier 2, environment frameworks: they define how to build an environment and leave the trainer to you (OpenEnv, ORS, NeMo Gym, RL-Factory).
• Tier 3, environment and training bundled: environment definition, rollouts and training in one package (Verifiers, SkyRL Gym, GEM, Atropos, Harbor, RAGEN, rLLM, Open-Instruct, TextArena, LlamaGym).

They focused on tiers 2 and 3, the ones that support multi-turn tool-calling environments. The ones left out, and why:

What each one ships out of the box

This is the key table. Each framework ships a different subset of components: ✅ a first-class API for that component; ⚙️ it works, but you lean on a convention or a trainer hook; 🔧 bring your own.

• Most bundled: Verifiers ships dataset, tools, rubric, rollout harness and training. You define the problem and it handles the rest, with the trade-off that you live inside its API surface.
• Least bundled: OpenEnv ships a protocol (MCP), session management and a rubric system. Tasks and execution are yours: more setup, more control.
• In between: ORS and NeMo Gym ship a deployment protocol and a reward mechanism, but you bring execution and tasks. GEM ships built-in environments and the Gymnasium API, and you bring the trainer.

And one thing jumps out of the table right away: none of them ships the execution backend. The sandbox is always on you.

Same concept, six names

Reading two of these frameworks' docs back to back feels like reading two different APIs for the same concept, because that's basically what it is. A small Rosetta stone:

Two things stand out with everything in one view: every framework has something for the initial state and for tool definitions, because every multi-turn environment needs both, but tasks are only filled in by frameworks that bundle their own task source. And for episode state, the HTTP frameworks all use sessions while the in-process ones lean on plain Python.

Dimension 1: what code you write

The first thing you run into is the API surface: what you subclass, how you declare a tool and how much boilerplate sits around the actual logic. The same toy environment comes out six different ways:

To see the contrast, here's the environment in OpenEnv, with its MCP tools registered inside the class:

class MyEnv(MCPEnvironment):
    def __init__(self):
        mcp = FastMCP("my_env")

        @mcp.tool
        def do_action(input: str) -> str:
            """Execute an action. Returns observation."""
            return execute(input)

        super().__init__(mcp)

    def reset(self, **kwargs) -> Observation:
        self._state = fresh_state()
        return Observation(done=False, reward=None, metadata={"status": "ready"})

In Verifiers, plain Python functions and a rubric are enough:

def do_action(input: str) -> str:
    """Execute an action. Returns observation."""
    return execute(input)

async def correctness(completion, answer, **kwargs) -> float:
    return 1.0 if answer in completion[-1]["content"] else 0.0

env = vf.ToolEnv(
    tools=[do_action],
    rubric=vf.Rubric(funcs=[correctness]),
    dataset=my_dataset,
)

And in GEM everything goes through Gymnasium-style reset() and step(), with spawn() to fork the environment for parallel GRPO rollouts:

class MyGEM(gem.Env):
    def reset(self, seed=None):
        super().reset(seed)
        self._state = fresh_state()
        return "Your task: ...", {"suffix": "Begin."}

    def step(self, action):
        parsed = parse_action(action)
        result = execute(self._state, parsed)
        terminated = is_done()
        return result, score(result), terminated, False, {"suffix": "Continue."}

    def spawn(self, same_state=False, **kwargs):
        return MyGEM(...)

Dimension 2: HTTP server or in-process

The most fundamental split: does the environment run as a separate HTTP server or inside the training process? Almost everything else falls out of that single choice. It shows in three ways:

• Where it runs. A server lives on its own machine, often a cheap CPU box, or in a Hugging Face Space. The in-process kind shares the training GPU node.
• What you install on the trainer side. HTTP needs only an SDK or requests. In-process pulls the full framework package into the training venv.
• How it scales. HTTP adds server replicas behind a load balancer. In-process adds identical training workers, each with its own copy of the environment.

An important nuance: in practice, most environments don't do the heavy work themselves. They delegate it to a sandbox provider like E2B, Modal or a custom container backend. That changes the scaling story for both shapes: the HTTP server becomes a thin router holding sessions and forwarding tool calls into the sandbox, so adding replicas is cheap; and the in-process class becomes a thin client over the sandbox, so the per-worker footprint stays small even with hundreds of workers. The deployment shape still matters (HTTP isolates dependencies, in-process shares them), but the heavy compute lives behind whichever sandbox you picked.

Dimension 3: tools and actions

All six end up exposing the same thing to the model: a list of callable tools with names, descriptions and typed parameters. What differs is where the tool definition lives and how the trainer discovers it. HTTP frameworks define tools on the server and ship a discovery endpoint the client hits at runtime; in-process frameworks define tools as Python functions registered when the environment is constructed. After discovery, both shapes are identical: send a tool call, get an observation back.

• Schema portability. HTTP frameworks ship JSON Schema (MCP, ORS ToolSpec, OpenAPI), which any OpenAI-compatible model knows how to call: the same tool spec hops between models cleanly. In-process frameworks derive schemas from Python type hints: faster to write, harder to lift into another framework or a model that expects a different schema dialect.
• Authoring cost per tool. A @tool decorator on a function is one line; a FastAPI endpoint plus its Pydantic body model is closer to a small file. Multiply by your tool count.
• The multi-turn loop is universal. Once the tools are wired up, the model-side experience is identical: name and typed args in, observation out.

Dimension 4: the reward

Who decides how good the model's behavior was, and when? This is where frameworks differ most philosophically. There are four patterns:

The composable rubric supports composition (WeightedSum, Sequential, Gate), an LLM as judge (LLMJudge) and trajectory-level scoring.

What goes inside a reward

The four patterns are about when the reward is computed and who owns the code. What goes inside is a separate question, with three flavors:

• Procedural or verifiable. Deterministic checks: does the final answer match, did the unit test pass, does the JSON parse, does the math evaluate to the right number? This is the RLVR (reinforcement learning with verifiable rewards) pattern that DeepSeek-R1 scaled up: the model only earns a reward when its output passes a programmatic check, typically a 0/1 signal. Scalable, hard to game, well suited to math, code and logic. The catch: it needs a clear ground-truth oracle.
• LLM-as-judge (sometimes called RLAIF). For what code can't check: creative writing, open-ended reasoning, summary quality, agent coherence. Another model reads the trajectory and scores it, often guided by a rubric (“rate factual accuracy 1–5, clarity 1–5, weight them”). That's what LLMJudge and JudgeRubric do. The catch is reward hacking: the policy learns to produce outputs the judge over-rewards, even though a stricter gold-standard judge would penalize them. Decomposed rubrics and “thinking judges” (J1-style) that reason before scoring push back on this.
• Dense versus sparse. A separate axis. A sparse reward fires only at the end (pass/fail, final score); a dense one fires per step, per token or per sub-goal, giving the trainer a much richer credit-assignment signal. External and post-episode rewards are sparse by default; server-embedded and rubric rewards can be either, and the rubric is the most natural place to make them dense. Dense rewards train faster and more stably; sparse ones are simpler to define and harder to game.

In practice you almost always mix them: a procedural part for what you can verify (the test passed, the JSON is valid, the answer matches) and a judge for what you can't (was the explanation good, did the agent stay on task), composed with weights into a single scalar. The frameworks only differ on where you write that composition and when it fires.

Dimension 5: who drives the loop, and when it stops

A multi-turn rollout is a tight cycle: the model generates an action, the tool gets called, the observation goes back to the model, repeat. Two things must be decided: who runs that cycle and how it knows it's done.

In every framework they tested, the trainer drives: it generates, parses tool calls, executes them through the adapter and generates again. The environment is passive: it responds to tool calls but never initiates anything. So the trainer controls episode length, tool-call parsing and retries. Some frameworks they didn't test flip this: in Atropos, the environment owns its own vLLM instance and generates rollouts autonomously; in Verifiers' env.evaluate() mode, the framework manages the loop. It matters because it changes how you think about batching, parallelism and resource allocation.

The other half is the stop signal:

Two examples with a coding-agent environment using a Claude Code style tool set: file ops (read, write, edit, multi_edit, ls, glob, grep), shell (bash(command, timeout=30)), planning (todo_write(todos)) and a terminator, submit_solution().

• Tool-based control (trainer-driven). The environment just exposes the tools, and the episode ends when the model calls submit_solution() or max_completion_length cuts it off. All six support this out of the box and it ports almost identically across them. For most tool environments, it's the path of least resistance.
• Harness-controlled rollout (env-driven). Sometimes you don't want the trainer to drive at all: you want a real coding harness (OpenCode, OpenClaw, Aider, Claude Code style) running in the sandbox and talking to the model directly. The harness owns the conversation: it manages the file tree, compresses context, retries on tool failures and knows when the task is done; the environment just hosts it and hands back the trajectory. You get benchmark-grade fidelity, but collecting training data becomes its own problem: the harness must emit, on every turn, the action, observation, reward and logprobs your loss needs, and that's harness-specific work each time.

Dimension 6: tasks and datasets

Every rollout starts with a task: the prompt, the data the model works on and, for scoring, the expected answer or the test that decides success. After the reward, it's the most varied dimension:

• Verifiers integrates with Hugging Face Datasets.
• GEM has a registry of 24+ built-in environments.
• ORS keeps tasks on the server (list_tasks(split)).
• NeMo Gym preprocesses JSONL through a CLI (ng_prepare_data).
• OpenEnv and SkyRL Gym leave it to you.

Frameworks with a dataset abstraction also give you splits and versioning for free; with the others, you wire that yourself.

A task is more than a row

A row is one input plus one expected output. A task, especially for agents and coding, is a bundle of files that has to land in the sandbox before the rollout starts. The cleanest example is SETA, on OpenReward: 1,376 terminal-agent tasks on the Open Reward Standard, each shipping a task.yaml (prompt and metadata), the data files, a tests/test_outputs.py run after submission, and a weights.json that turns test outcomes into a scalar reward. SWE-Bench rows go further: each carries repo, base_commit, problem_statement and test_patch; the environment clones the repo at that commit, applies the test patch and grades the agent's diff with FAIL_TO_PASS and PASS_TO_PASS.

Those bundles usually live behind the dataset row (in S3, an HF dataset repo or a tarball): the environment pulls them on episode start, drops them into the sandbox, scores and tears down.

Coupled or decoupled: who owns the task spec

• Coupled. Verifiers expects a Dataset with a prompt column and optional answer or info; GEM ships its own loaders; ORS and NeMo Gym pin the schema on the server. The Environments Hub and OpenReward go further and standardize the whole package (layout, scoring contract, even wheel-based packaging), so any task that fits the spec runs in any environment that follows it.
• BYO. OpenEnv and SkyRL Gym leave the dataset to you: prompts come from anywhere, but every new source costs a little integration.

Coupling means the environment dictates the spec and you transform your data. For example: turning SWE-Bench JSONL into a Verifiers Dataset with a prompt column and the patch in info; fanning a Kaggle Q&A dump out into OpenReward-style folders, one per row, with task.yaml, data/ and tests/; or, for HumanEval or GSM8K, nothing at all, because the framework already ships a loader. You write the mapping once and, in return, the same environment runs unchanged on every task that fits the spec. If you care about one task, the spec is overhead; if you train across many or pull in benchmark suites, it's what makes it work.

Dimension 7: ecosystem and maturity

Dimension 8: what a new environment costs

How hard is it to go from “I have an idea for an environment” to “it's running in training”? It matters whether a human writes it or an agent does.

And they tested it with an agent: Claude Code scaffolded a complete Wordle environment across all six frameworks in a single session. The in-process ones (Verifiers, SkyRL, GEM) were done in minutes; the HTTP ones (ORS, NeMo Gym, OpenEnv) needed extra rounds for Dockerfiles and deployment configs. A single Python file, no Docker, no server and instant pytest feedback is exactly what suits an agent best.

Dimension 9: local and cluster setup

On a Slurm cluster, the question is who lives where. HTTP frameworks put the environment on its own node, often CPU-only, and point any number of GPU training nodes at it, with a load balancer fronting a pool of replicas you can grow. In-process frameworks put trainer and environment in the same venv on the same GPU node: simpler to operate, but scaling the environment is whatever orchestration story your training framework already has.

Dimension 10: scaling

RL generates several rollouts per prompt, ideally in parallel, so you're talking to many environments at once. In GRPO that's num_generations (typically 4-16) per prompt: with 64 prompts and num_generations=8, you have 512 concurrent environment instances per step.

In-process

Verifiers, SkyRL Gym and GEM create Python objects: startup takes under a millisecond, so 512, or 10,000, are no problem, and parallelism comes from the training framework (each worker gets its own instance). The real limit is orchestration, not architecture: if the environment does its own compute (heavy parsing, simulation), it shares CPU with training, and scaling it means scaling training nodes, paying for GPUs to do CPU work; if it delegates to a sandbox or remote service (E2B, Modal, a custom HTTP backend), the class is a thin client and the environment scales independently, but the orchestration is yours: picking the provider, managing replicas, failover, load balancing. HTTP frameworks bake that in; in-process ones leave it to you. Two things stay constant: environment dependencies must coexist with training dependencies (a framework pinning a conflicting numpy or torch can break training), and a crash in pure in-process environment code can take down the training worker, while sandboxed execution isolates failures naturally.

With a server: the benchmark

The openenv-scaling benchmark deployed an environment as a FastAPI app in a Docker container (the same image HF Spaces uses) across five infrastructures. OpenEnv, ORS and NeMo Gym all follow that shape (an app with per-session state in a container), so the numbers apply to all three. Maximum concurrent environments at ≥95% success:

And where does the time go at full load? The median of each phase (connect, reset and step, with a simulated 1 s tool wait) and the p99 total:

The multi-node p99 reflects connection queuing for 16,384 sessions through the Envoy proxy, even though its absolute throughput (518 requests per second) is the highest of any configuration. The five takeaways:

1. Docker adds no meaningful overhead: local Docker and uvicorn reach the same 2,048.
2. Load-balancer configuration matters: before fixing Envoy, the two nodes reached only 128; after, 16,384, a 128× improvement.
3. HF Spaces caps at about 128 sessions: enough for development and demos, and convenient because it's the largest catalog of pre-built environments.
4. The server is rarely the bottleneck: even a laptop handles 2,048 sessions. Execution (sandbox creation, tool execution) dominates per-step latency regardless of framework.
5. Horizontal scaling is a load-balancer config problem, not a protocol problem: the 128 → 16,384 jump came from tuning Envoy, not from changing the wire format. Sticky sessions, which WebSocket forces, make it harder; for thousands of environments, stateless requests with a session ID have fewer footguns.

The global table

Fold the ten dimensions into one table, and three things come out:

• No framework dominates every row. Each wins a column or two and loses elsewhere. The pick comes down to the two or three dimensions you actually care about: usually deployment shape, episode control and how strict the task spec is.
• The loudest split is still deployment. HTTP server (OpenEnv, ORS, NeMo Gym) versus in-process (Verifiers, SkyRL Gym, GEM) cuts through almost every other row. Pick that axis first and the table narrows fast.
• Tooling maturity is uneven. CLI scaffolding, dataset registries and observability are strong in some frameworks and absent in others, even ones that look identical on API surface. Worth checking before you commit.

The six, one by one

What stood out, and which one to pick

• The “environment” boundary is a design choice, not a standard. Some (OpenEnv, SkyRL Gym) give you a thin tool interface and leave dataset, reward and trainer wiring to you; others (Verifiers, GEM) bundle it all. Neither is wrong: it's control versus time-to-running.
• HTTP versus in-process is the loudest fork. Sandboxed execution (code, shell, browser) wants HTTP, so environment compute scales independently from training. Pure Python (games, math, text reasoning) wants in-process: zero RPC overhead, no infrastructure to babysit.
• Dataset coupling cuts both ways. Bundled (Verifiers, ORS), environment and dataset come as one unit and you can't swap one without the other. Decoupled (OpenEnv, SkyRL Gym), you wire datasets yourself, but any task fits any environment.
• Reward timing matters more than reward content. Per-call scoring is clean for per-step feedback and awkward when the reward only makes sense at the end; post-episode verification is clean for trajectory-level scoring but gives no in-episode signal.
• The execution backend dominates per-step latency, regardless of framework.

And if you just want a recommendation, their decision tree has four yes/no questions:

1. Where should the environment run? On its own server (OpenEnv, ORS, NeMo Gym) or inside the trainer (Verifiers, SkyRL Gym, GEM).
2. Do you want a bundled trainer? Yes (NeMo RL, Prime RL or SkyRL) or no, because you'll plug in your own (TRL or custom).
3. Do you need a large catalog of pre-built environments?
4. Do you need a strict Gymnasium-style API? The 5-tuple from step(), drop-in for classical RL code.

For example: with your own server and your own trainer, you're left with OpenEnv and ORS; with a server and a bundled trainer, NeMo Gym; and if you need strict Gymnasium, only GEM.

What bit them

It's a snapshot from May 2026. All six launched in 2025 and are moving fast: six frameworks doing the same thing with different APIs is what early-stage exploration looks like. The authors expect consolidation around fewer protocols (MCP, ORS) over the next year.

Two details that don't add up

Comparing the article's interactive figures against each other, two things don't quite match:

• GEM and the trainer. The tier map puts GEM in tier 3 (environment and training bundled), but the text says GEM “provides built-in environments + Gymnasium API but you bring the trainer”, and the ship matrix marks its training as partial (“multi-trainer”).
• NeMo Gym and training. In the five-stage diagram, NeMo Gym stops at rollout, with no trainer; in the ship matrix, though, it has native training (Megatron / NeMo RL), and the decision tree also counts it among the frameworks with a bundled trainer.

None of this changes the conclusions, but it's worth knowing if that cell is what you're choosing by.

Wrap-up

An RL environment for an LLM is tasks, tools, a reward and a loop, spread across eleven components that the six frameworks cover in different ways. The decision that orders everything is whether the environment lives on its own server or inside training; then come when the reward fires, who drives the loop and how strict the task format is. And when you scale, the server is rarely the bottleneck: the sandbox is, along with a well-configured load balancer.

What environment would you like to train?

Sources

• “The ultimate guide to RL environments: building and scaling them in the LLM era”, by Adithya S Kolavi, Lewis Tunstall, Leandro von Werra, Quentin Gallouédec, Amine Dirhoussi, Ben Burtenshaw and Sergio Paniego (Hugging Face), May 5, 2026.
• The code: RL_Envs_101, and the openenv-scaling benchmark.
• On reinforcement learning applied to agents: my post on NapMem, which trains an agent with GRPO to navigate its memory.
• The video, on my channels: English and Spanish.