What Gymnasia is
Gymnasia is a mobile personal-training app built with React Native and Expo. Inside it there are two AI agents: a conversational coach and a vision subagent that estimates calories and macronutrients from food photos.
The interesting part, and what this post is about, is that both run entirely on the device. There is no backend of our own: orchestration, tools and storage live in App.tsx and in AsyncStorage, and the API keys are supplied by the user. The user's data never leaves the phone except for one single exception, explained below.
This is the main post of the series. Below are the two architecture graphs, and at the end the series index with the posts that cover each part of the development.
The general agent: Gymnasia Coach
Swipe the graph horizontally to see all of it.
Gymnasia Coach is the app's general conversational agent, the one in the AI Chat tab. It is a BYOK (Bring Your Own Key) agent: the user configures their own OpenAI, Anthropic or Google API key, and the app calls the chosen provider directly from the device. There is no backend of our own: all orchestration, tooling and storage logic lives in App.tsx and in local AsyncStorage.
Configuration and active provider
The user saves their API key
One of the 3 keys (OpenAI / Anthropic / Google) is flagged as is_active. Only that one is used for the general chat.
prompts/AGENTS.md on GitHub
Downloaded from raw.githubusercontent.com/maximofn/gymnasia, cached locally, and falling back to an embedded default prompt if the network fails.
Provider call
Each provider has its own request/response adapter, because tool formats, streaming and content blocks are different in all three. But they share the same agentic loop that comes next.
Agentic loop (tool use)
Model responds
Streamed text and/or tool_use / function_call blocks.
handleToolCall()
Runs 100% locally against the app state (AsyncStorage, JSON repos).
Result → model
Fed back as tool_result / function_call_output. Repeats until no more tools are requested.
The 12 tools
Storage and output
LocalStore (AsyncStorage)
Diet, routines, measurements and personal data live on the device. The food and exercise repos are static JSON bundled with the app.
GitHub Issues API
The only exit point to an external service other than the LLM provider: create_feature_issue.
apps/anthropic_proxy/cors-proxy.py) because the browser blocks the direct call to api.anthropic.com. OpenAI and Google work directly from the browser.The vision subagent: Food Estimator
Swipe the graph horizontally to see all of it.
The Food Estimator is the vision subagent that estimates calories and macros from food photos (Diet tab → AI Estimation). It is independent from the general chat: it has its own system prompt, its own tool and its own provider-selection policy.
Input
1–6 photos of the meal
Camera or gallery. It also accepts text (follow-up questions about the estimate), reusing the conversation context.
Priority-based provider selection
Unlike the general chat, the active provider is not used here: providers are tried in order until one has an API key configured.
Specialised system prompt
Visual nutritionist
Always estimates kcal, protein (g), carbs (g), fat (g) and total weight (g). Gives ranges when uncertain.
Classification
Determines whether it is a producto_comercial, a receta or a generic base food.
Structured output
If the user asks for "Devuelve json", it replies with JSON only: dish_name, calories_kcal, protein_g, carbs_g, fat_g.
Agentic loop with the barcode tool
Is there a barcode in the photo?
The prompt forces the model to use the tool if it detects an EAN/UPC in any of the images.
scan_barcode(barcode)
Calls OpenFoodFacts (public API) with the scanned code and returns exact nutrition data for the product.
Commercial product confirmed
If scan_barcode was used, the classification is always producto_comercial, with exact rather than estimated data.
As in the general agent, the tool result is fed back to the model and the loop repeats, up to 5 rounds, until a final answer is produced.
Persisting the result
User confirmation
When the user accepts the estimate, the final JSON block is requested and parsed.
add_meal_food
It is added to the local diet for the selected day and meal, in the same store the general agent uses.
Variant: manual estimation
The user describes a food in text
Conversational flow: the user names the food, the model asks for missing ingredients and quantities, computes values per 100 g or unit, the user confirms, and it returns the JSON to store in the food repo.
The series
This post is the cover. Each part of the agent's development gets its own post, and they are all linked from here.
- No posts in the series published yet. The first ones will be the planning post and the tool calling post.
In the meantime, the project page is at maximofn.com/en/gymnasia and the code is open source, on GitHub.
