A PRISMA review run by AI agents that audits itself (Hermes)

A PRISMA review run by AI agents that audits itself (Hermes)
6rules a review must meet for the author to trust it
3phases: corpus and screening, full text and extraction, synthesis and closing
4bibliographic sources: OpenAlex, Crossref, Semantic Scholar and arXiv
2reviewers running different models before the final gate
A PRISMA systematic review run by AI agents that audits itself (built with Hermes)
I also explain it in this video, on my channel Articulator EN. The figures in this post are frames from the video.

The problem: it sounds good but leaves no traillink

A systematic review is not a summary of what has been published on a topic. It is a method: you define a question, search in a documented way, decide with written criteria which studies get in and which don't, and keep a record of everything. PRISMA 2020 is the guideline for reporting that process (how studies are identified, selected and included), and PRISMA-S extends it to the search itself.

The first problem the author wanted to attack is how fragile a review is when a single model does it, whoever makes the model. A model summarizes, compares and proposes lines of argument very well, but that is not a serious review. If the process leaves no trail of the search, the inclusion criteria, the reasons for exclusion, proof that someone read the full text and a verifiable closing, the result can sound good and be methodologically weak.

The second problem is operational. A review is not solved with one call to the model. It needs a protocol, bibliographic acquisition, a DOI audit, screening, PDF retrieval, full-text reading, extraction, figures, manuscript writing, cross-review, packaging and syncing. If all of that lives only in the memory of a conversation, the system is fragile by definition.

That is why the system separates state, process and persistent artifacts. The line that sums it up: what matters is not that the agent «remembers», but that it reads again what it left written.

Before getting into it, the author adds a disclaimer: no interest in the university debate about whether these systems should be used; the point is only to show how this one is built inside so others can replicate or improve it. And a warning: the packaged agent is not released yet. That, the author says, will come in another article.

The six ruleslink

The system is not built around the question «what can an agent do in this research?», but around another one: «what minimum conditions must a review meet for me to trust it?». The answer is six rules:

  1. PRISMA 2020 as the reporting framework.
  2. PRISMA-S as the framework for search transparency.
  3. DOI-first: whenever it exists, the DOI (the unique identifier of each publication) is the key for each study.
  4. Mandatory full text for the final corpus.
  5. Counts are recomputed from the real files; nobody types them by hand.
  6. Editorial closing only when the final gate returns PASS, not on the subjective feeling that «it's good enough».
The six rules: PRISMA 2020, PRISMA-S, DOI first, full text, recomputed counts and closing only on PASS
The six rules. The sixth: it closes when the gate returns PASS, not when it «looks done».

These rules live in a file, the base skill prisma-systematic-review/SKILL.md, which works as an operational and methodological contract. The author insists on this: in many agent systems the interesting part sits in invisible prompts. Here the methodology is something you can version and audit.

The rule that costs the most tokens is the fourth. It demands readable full text for final inclusion, and a model that can read that kind of document well. An abstract helps to discard quickly, but it is not enough to declare that a study belongs to the corpus. In the author's words, it is a methodological boundary, not a style preference.

The architecture: four layerslink

The system is built with Hermes (Hermes Agent, according to the article's architecture figure) and rests on four layers:

  1. A minimal surface on Telegram. The author doesn't want operations to depend on a custom interface. Reviews are launched and checked with commands such as /prisma_systematic_review and /prisma_status.
  2. The runtime in Docker, with two services: hermes-agent runs the gateway and the main work, and hermes-prisma-watchdog monitors live reviews and resumes phases when it detects they have stalled.
  3. The model policy. The primary model is Qwen 3.6, served by NaN Builders. If it fails, there is a fallback chain on Ollama Cloud: DeepSeek v4-pro, Kimi K2.6, Qwen3-coder and Gemma4. It all lives in a config.yaml with its parameters.
  4. Orchestration, with three scripts: complete_review.py, publication_autopilot.py and publication_gate.py.

There is another layer that usually stays hidden: the skills. The author's installation has ten local research skills, and on the road to a publishable review five do most of the work: systematic review, status, academic review, integrity and revision roadmap. The underlying idea is that the architecture is not one super agent that does everything, but specialized layers with small contracts. That, the author says, keeps it maintainable for this use case.

According to the article's figures, the work is organized in three phases: corpus and screening; full text and extraction; synthesis and closing.

Three phases: corpus and screening, full text and extraction, synthesis and publication; the system stays alive thanks to saved state, the watchdog, Telegram and Obsidian
The three phases. If the user leaves, the system keeps going: saved state, watchdog, Telegram and Obsidian.

How a review startslink

One of the most useful ideas in the article: a review gets biased earlier at intake than later in the discussion. So the system forces you to freeze, as soon as possible, the question, the time window, the admissible type of study, the target number of final studies, the inclusion and exclusion criteria and the editorial framework. If the topic starts as «let's see what comes up», the rest of the system only automates ambiguity.

That phase leaves a trail from the first minute, in three files:

  • protocol/intake.md: the question and the scope.
  • protocol/eligibility-criteria.md: the inclusion and exclusion criteria.
  • protocol/search-strategy.md: the search strategy.

What matters is not only that they exist, but that someone else can open them and check whether the bias was born in the question, in the search or in the interpretation.

There is a detail the author likes to mention. If there is no specific target journal, the system falls back to a generic mode (generic-common-core), a template that would pass a first filter. That decision affects the LaTeX template, the editable package and the tone of the closing. It is a way of not pretending the work is aligned with a journal that hasn't been chosen yet.

Four commands, four uncertaintieslink

This is where the review stops being an intention. The question, the unit of analysis, the time window, the criteria and the quality threshold are fixed in files that condition everything that follows. Four pieces do it, and each one closes a different uncertainty:

PieceWhat it fixesWhat it does
bootstrap_topic_review.pyThe scopeIt doesn't just create folders: it materializes the starting point (the intake.md, the search strategy and the project structure). If the corpus later turns out biased, you can go back to that origin and see where the problem was born.
doi_audit.pyThe identity of the corpusIt separates records with an identifiable DOI, duplicates and ambiguous ones. It avoids one of the most common errors: believing there are more studies than really exist, or not knowing why a paper was merged or discarded.
complete_review.pyThe order of the processIt governs the whole sequence: acquisition, screening, full-text retrieval, extraction and the jump to the editorial layer. No extraction if the corpus isn't fixed, no study accepted without full text, no publication if intermediate artifacts are missing.
review_runtime_state.pyThe operational memoryIt keeps the state outside the conversation: what is done, what is missing, what is blocked and which files support each step. If the review is interrupted, the system reads it again and continues.
Four pieces and the uncertainty each one closes: scope, identity, order and memory
Each piece closes one uncertainty: scope, identity, order and memory.

The methodological boundary is written down before the first record is imported. From then on, the system no longer tries to guess which review the author wants: it runs the one that was specified.

From four sources to a DOI-first universelink

The system queries several sources because none is enough on its own: OpenAlex, Crossref, Semantic Scholar and arXiv, with direct HTTP calls. The result is not «a search in the abstract», but a set with a technical trail, spread over five files:

  • searches/search-log.csv: the query, the source, the date and how many records came back.
  • records/master-records.csv: the raw universe, normalized.
  • records/doi-index.csv: the canonical key of each study, by DOI.
  • records/duplicates.csv: the collisions detected.
  • records/missing-doi.csv: what cannot be resolved by DOI.

When a record has no DOI, the system generates a stable hash-based identifier for it (a RID). That way the same paper, arriving from two sources with its title written two ways, counts once. According to one of the article's figures, this approach «reduces noise, lowers cost and improves reliability»; that is the author's claim, not a measurement.

At this point, the author says, the review stops being a collection of searches and starts to look like a document base.

How it decides: OK, KO or needs morelink

The inclusion and exclusion logic is not a magic algorithm, and it is not asking a model «does this look relevant?». It is a sequence of four layers:

  1. Minimum rules. The most visible one is a minimum threshold of useful text: if the full text doesn't reach it, there is no basis for final inclusion. The article's text gives no number; one of its figures sets it at 4,000 useful characters.
  2. Model-assisted decision. The model classifies and justifies.
  3. Canonicalization. A function normalizes the variants (include, include_ft, exclude, «excluido», «excluir», maybe) into a consistent set of decisions.
  4. Audit with recomputation. The counts in the PRISMA diagram are not typed by hand: they are recomputed from those normalized decisions.

With that, each study ends in one of three exits:

OK

It passes the minimum rules, confirms the focus in the full text and leaves a traceable justification in a CSV.

KO

It fails those criteria or has a clear reason for exclusion.

Needs more

Quick screening is not enough to decide.

Three exits for each study: accepted, rejected or needs more, each with its reason
Three exits, and each one leaves its reason in writing.

For the author, this logic is closer to structured appraisal tools such as ROBIS or AMSTAR 2 than to a simple relevance ranking. That is an opinion about the design, not a formal equivalence.

Deep extractionlink

Knowing that a paper is «experimental» is not enough. The author wants to know which systems it names, which benchmark it uses, which sample it describes, which variables it employs, which method it reports and which theory it stands on. All of that goes into extraction/extraction-table.csv, with fields such as work_type, empirical_type, models_or_systems_studied, benchmark_dataset_or_corpus, dependent_variables or theoretical_framework.

When the first extraction falls short, refresh_extraction_depth.py comes in: it rereads the text extracted from the PDF and forces a deeper layer. It solves a problem the author says came up several times: papers that after a first pass were labeled only as «experimental study», when the full document did offer names, scales and metrics. The author flags this as very important for anyone building their own system.

Each study is organized in four blocks:

One record in four blocks: common core, empirical, technical, and theory and synthesis
The record for each study: common core, empirical block, technical block, and theory and synthesis block.

The conclusion of this part: comparability across studies comes from the extraction schema, not from a pretty table at the end.

Figures and visual evidencelink

In many topics, a figure concentrates the architecture, the experimental design or the relations between concepts that the abstract doesn't capture. So the system has an explicit layer of visual evidence, with models that can see and process images.

The key script, prepare_paper_figures.py, separates three things that are easy to mix up:

  • Scientific figures extracted from the PDFs (figures/extracted/).
  • Page renders for support and diagnosis (figures/page-renders/).
  • Own figures, generated for the manuscript.

Without that separation, a full rendered page can be mistaken for a reusable figure. Traceability lives in a manifest, a catalog and a ranking (figures/manifest.csv, figure-catalog.md and figure-ranking.csv). Own figures carry a signature (in this case, 686f6c61) to tell them apart from those of the corpus: mixing someone else's evidence with your own visual synthesis is, the author says, bad methodological practice and bad editorial practice.

From corpus to manuscriptlink

Once the corpus is consolidated, the challenge shifts from documentary to editorial. The central piece is publication_audit.py: it rebuilds sections, cleans up inconsistencies, normalizes citations and references, integrates tables and figures, detects obvious gaps and generates the manuscript. It leaves three versions of the same document in paper/manuscript/:

  • publication-ready.md: the editable version.
  • publication-ready.tex: the LaTeX output, to keep working in an academic setup.
  • publication-ready.pdf: the compiled version, ready to circulate.

On top of that come the BibTeX bibliography (references.generated.bib) and the generic template main-common-core.tex when there is no target journal.

Here the author imposes what are described as personal quirks: every relevant study must have more than a table row. The manuscript includes a short profile per study, one or two paragraphs before the compact table, and expanded blocks for discussion, theoretical implications, original contribution, conclusions and future work. «Publication-ready» is the name of the files and the author's goal; the article doesn't say that any of these reviews has been published.

Cross-review and the final gatelink

For the author, this is the layer that most sets the system apart from a conventional flow with several models. A manuscript is not accepted because a model wrote solid prose. It is accepted when it gets through a cumulative circuit:

  1. Cross-review (publication_peer_review.py). Reviewer A runs on Qwen 3.6 and reviewer B on DeepSeek v4-pro; Kimi K2.6 is the first editorial fallback. The assignment sits in a file, reviewer-models.csv.
  2. Integrity (check_manuscript_integrity.py): it hunts for weak claims, inconsistencies and gaps in support.
  3. Revision roadmap (build_revision_roadmap.py): it turns the review into a matrix of actionable changes.
  4. Final gate (publication_gate.py): it checks that the required artifacts exist and that the closing is legitimate.
  5. Autopilot (publication_autopilot.py): it repeats the circuit until it closes or documents a real blocker.
The circuit: integrity, revision roadmap and final gate, repeated by the autopilot until PASS or a documented blocker
The autopilot repeats the circuit until the gate returns PASS or a blocker is documented. The matrix is illustrative.

It is worth reading this precisely: the two «reviewers» are different models reviewing each other, not independent human reviewers and not a journal's peer review. What it avoids, according to the author, is one specific mistake: delivering something that looks finished when it hasn't passed the checks that justify calling it publishable.

Packages and memory outside the chatlink

When the gate returns PASS, the system doesn't end in a loose folder. It builds two packages, because not every reader needs the same thing: publication-package.zip, the general editorial package, and publication-latex-editable.zip, to keep editing in LaTeX.

At the same time, sync_review_to_obsidian.py syncs the review to an Obsidian vault, where a browsable version stays: study notes, overview, visual library, notes and the PRISMA trail.

Continuity doesn't depend on remembering a conversation. It depends on state files (notes/runtime-state.md and notes/runtime-state.json), on manifests, on the ZIPs and on a watchdog that knows how to reread and resume. The system's real memory is outside the chat. According to the author, from a FAIR point of view this brings the review closer to a findable, accessible and interoperable asset; the article doesn't claim it complies.

The limits the author admitslink

The article doesn't close pretending the system is perfect. It still depends on four things: the quality of the sources, whether the full text can be retrieved, whether the cloud models are available and whether complex figures or tables can be extracted without loss.

And even with a hardened gate, the author sees four areas to improve:

Room to improve: a formal risk of bias layer, better templates per journal, more control over meta-analysis and a finer step from publication-ready to submission-ready
What the author wants to improve.

The first one is telling: a formal risk of bias layer is still missing. The author says work continues toward publishing something people can install, use and improve.

How to read this articlelink

  • It describes a design, not results. There are no figures for accuracy, time or accepted reviews. «Publishable» and «defensible» are the author's goal, not something the article proves.
  • You can't install it yet. The package is not released; the author announces it for later.
  • It doesn't remove bias. What it does is leave a trail so someone else can audit each decision.
  • The reviewers are models. It is cross-review between different models, not independent human review.
  • Models and versions (Qwen 3.6, DeepSeek v4-pro, Kimi K2.6…) are those of the author's installation, as named in the article.

In shortlink

What I take from the article applies to any agent system, not only to scientific reviews: rules in a file instead of an invisible prompt, one file per step, memory outside the conversation, counts recomputed from the data, and a final gate that decides the closing instead of the feeling that «it's done».

In one sentence

A serious review leaves artifacts, not just a convincing conversation; quality is an exit condition, not an intention.

Would you trust a review made this way if you could open every file and follow every decision?

Sourceslink

Continue reading

Last posts -->

Have you seen these projects?

Gymnasia

Gymnasia Gymnasia
Expo
React Native
TypeScript
OpenAI
Anthropic

Fitness app with two agents that run entirely on the device, with no backend, so the user's data never leaves the phone. A BYOK conversational coach with local tools and a remote system prompt with offline fallback, plus a food estimator that pulls macronutrients out of a photo of the plate, with barcode scanning against OpenFoodFacts.

LangGraph Deep Researcher

LangGraph Deep Researcher LangGraph Deep Researcher
Python
LangGraph
FastAPI
React
TypeScript
Docker

Multi-agent research system built with LangGraph. A supervisor breaks your question down into topics and launches search sub-agents in parallel; each one compresses its findings before handing them to a writer agent that produces the final sourced markdown report. Live streaming over WebSockets, a configurable model per role and bring-your-own API keys that are never persisted server-side.

Tau

Tau Tau
Python
LangChain

Multi-agent tutoring system for secondary school students, with one agent per subject and course material written and validated by a team of teachers. It was used with real students at a private school in Spain and at a secondary school in Colombia.

View all projects -->
>_ Available for projects

Do you have an AI project?

Let's talk.

maximofn@gmail.com

Machine Learning and AI specialist. I develop solutions with generative AI, intelligent agents and custom models.

Do you want to watch any talk?

Last talks -->

Do you want to improve with these tips?

Last tips -->

Use this locally

Hugging Face spaces allow us to run models with very simple demos, but what if the demo breaks? Or if the user deletes it? That's why I've created docker containers with some interesting spaces, to be able to use them locally, whatever happens. In fact, if you click on any project view button, it may take you to a space that doesn't work.

Flow edit

Flow edit Flow edit

FLUX.1-RealismLora

FLUX.1-RealismLora FLUX.1-RealismLora
View all containers -->
>_ Available for projects

Do you have an AI project?

Let's talk.

maximofn@gmail.com

Machine Learning and AI specialist. I develop solutions with generative AI, intelligent agents and custom models.

Do you want to train your model with these datasets?

short-jokes-dataset

HuggingFace

Dataset with jokes in English

Use: Fine-tuning text generation models for humor

231K rows 2 columns 45 MB
View on HuggingFace →

opus100

HuggingFace

Dataset with translations from English to Spanish

Use: Training English-Spanish translation models

1M rows 2 columns 210 MB
View on HuggingFace →

netflix_titles

HuggingFace

Dataset with Netflix movies and series

Use: Netflix catalog analysis and recommendation systems

8.8K rows 12 columns 3.5 MB
View on HuggingFace →
View more datasets -->