Start several queries together
Imagine the agent needs to look up foods, exercises and workout routines to make a recommendation. It already knows what to search for in each case, and no query needs a result from the others. Your program can start all three together.
If each query takes 300 milliseconds, running them one after another takes about 900 milliseconds. If they spend most of that time waiting for a download, starting them together can bring the total closer to 300 milliseconds. The idea is to use the time spent waiting for one query to make progress on the others.
One after another
Total time: about 900 ms
All three together
Total time: about 300 ms
The model can request several tools in a single response, but your program executes them. Receiving three requests is not enough: the code must start the queries before it begins waiting for their results.
Decide what can run together
The first decision is to distinguish tools that read data from tools that change data. Then check their dependencies: a query can start alongside another if it already has all its arguments and does not need a pending result or change. Define these rules in code; do not depend on the model guessing which calls are safe.
For example, the agent requests a food search, an exercise search, saving a weight and reading the saved weight. We can organize those four operations like this:
- Search for foods and search for exercises.
These are two independent queries. Start them together and wait for both to finish.
- Save a weight of 75 kg.
Now we change a value. Run this operation on its own and wait until it finishes saving.
- Read the saved weight.
Start this query after saving, so it reads the 75 kg value. Starting it earlier could return the previous weight.
You also need to keep changes from overwriting each other. Imagine storage that saves the entire food list: two operations read the same empty list; one prepares a list with an apple, and the other a list with a yogurt. If they save those lists separately, the second save can erase what the first added. Running them one after another lets the second operation read the list that already contains the apple and add the yogurt.
Gymnasia makes a simple choice: it runs all changes one after another, even when they affect different data. A tool the program does not recognize also waits its turn. If an action needs user confirmation, wait for that decision and for the action to finish before continuing. A later query must read the updated data rather than reuse a copy from before the save.
How to implement it, step by step
You need a component that receives the model's requests and decides when to start each one. You can build it around these rules:
- Declare what each tool does.
Specify whether it reads or changes data and the conditions that allow it to run alongside others.
- Prepare each request.
Check the inputs to the tool, for example that a search includes the text it should look for. Also keep the identifier that arrives with the LLM request. When the tool finishes, send its result back to the LLM with that identifier so the model knows which request it answers.
- Limit active queries.
Gymnasia allows up to four. A fifth waits until one finishes. This is a starting point to adjust through measurement, not an optimal number for every app.
- Respect data changes.
Go through requests in their original order. Before changing data, wait for all earlier queries. Finish that change before starting the next requests.
- Collect the results.
When the tools finish, gather their results and send them to the LLM so it can continue the conversation and prepare the answer for the user. In this implementation we send them in the original request order, each with its identifier. This lets the model distinguish the food-search result from the exercise-search result.
This pseudocode shows the central decision. A “slot” is simply permission to start another query without exceeding the limit:
for each call in the order requested:
if the call can run independently:
wait until a read slot is free
start the call; save its result or error with its label
else:
wait for all running calls
run this call on its own and wait for it to finish
save its result or error with its label
wait for all running calls
send every result or error to the LLM in request order, with its labelThis lets consecutive queries start together, while keeping every query behind an earlier change. Decide how to represent errors too: if a query fails, keep that error with its request without losing the other responses. If the failure requires ending the turn, stop starting operations and wait for those already running. Avoid automatically repeating a change when you do not know whether it was saved.
Match each response to its request
Suppose the LLM requests these queries: A searches for foods, B searches for exercises, and C reads routines. C might finish first. Its result still belongs to the routines query, even if it is the first result your program receives.
Your program stores each result with its request identifier as soon as it arrives. Once all queries finish, it prepares the next message to the LLM with A and its foods, B and its exercises, and C and its routines. The model can use that data to continue the conversation and write the answer the user will see.
The label connects the LLM request to the tool result; it is not a number the user needs to see. Keep the identifier that arrives with each request. The tool name is not enough: the LLM could request two different food searches in the same turn.
Tests to check that the pattern works
First test the component that decides when to run tools using test functions: you control when they start, when they finish and what they return. This checks the rules without depending on a connection or a real model. Include these cases:
- Independent queries start together.
Keep two queries waiting and check that both have started before allowing either to finish. This proves that the second does not wait for the first.
- The operation limit is respected.
With a limit of four, prepare five queries. Check that four start, the fifth waits, and it can start when one of the earlier queries finishes.
- Saving finishes before the new value is read.
Use the example with two searches, saving 75 kg and reading the weight. The save waits for the searches; the read waits for the save and returns 75 kg.
- Results go back to the correct LLM request.
Make queries finish C, A and B. Check that the message sent to the LLM contains A with foods, B with exercises and C with routines, in that order and with their original identifiers.
- An error does not make other results disappear.
Make one query fail and two others succeed. If that error allows the turn to continue, the LLM should receive both results and the error matched to the failed request. Separately test that a failure requiring the turn to stop prevents more operations from starting.
- Two changes preserve what the earlier one saved.
Add an apple and then a yogurt. Read the saved list again and check that it contains both. The test checks the data that was saved as well as the operation order.
Then test the complete path with a simulated LLM: supply requests A, B and C, run the tools and inspect the next message your program would send to the model. It must contain the correct results, their identifiers and the chosen order. This test checks that scheduling and sending results back to the LLM fit together; measuring speed alone would not catch a result matched to the wrong request.
This also works on a phone
A phone can start several downloads while waiting for their responses. This technique uses those waits; it does not by itself spread local calculations across multiple cores. If tools do substantial computation, starting them together does not guarantee an improvement. Measure on the device and connection your app will use.
Check the improvement and the results
Compare the same queries before and after the change. To find out what this implementation contributes, measure from the start of the tools until their results are available. Measure the time the model takes to generate its final response separately.
Repeat the comparison when the app needs to download the data and when it already has the data stored. The improvement may be large in the first case and small in the second. Use several runs, not just one, and compare typical times and slower cases.
Also check that each response stays matched to its request, a query after saving returns the new value, active operations never exceed the limit, and errors do not make other results disappear. Speed is only useful if the app continues to give correct responses.
A measured example from Gymnasia
We compared three exercise searches in the web version of Gymnasia. A simulated model always supplied the same requests, and every download of search data had a controlled delay. When that data still needed downloading, median time fell from about 696 to 264 milliseconds.
The measurement included receiving requests, running searches and preparing their results to continue the conversation. It did not include a real model generating a response. This is a controlled web comparison; the improvement on a phone still needs measuring on the device. Full method and results.
A piece of the agent we are building
In this series we use a fitness app to explain how to build an agent that reads and changes data. Running tools in parallel adds a decision to the program that controls it: what can start together and what must wait. You can apply the same pattern to other agents, keeping the operation limit and the link between each request and its response.
Continue the series on building an agent with Gymnasia. To see the specific implementation behind this example, read the code change and its tests.