Dominik Lukeš · Semantic Machines explainers

Beyond Deep Research

How are AI-scientist systems actually evaluated — and who checks the claims? Follow one inquiry through finding papers, organising references, and extracting the evidence.

The conversations, tables, and Zotero simulation use invented examples. The Elicit search example describes saved searches from research on AI-scientist evaluations.

Use Next step in each conversation to follow the question, the agent’s preparation, its use of the tools, and the saved results. Previous step goes back; Replay starts again.

Why run the inquiry yourself?

Deep research tools write reviews and reports from sources they chose. You get text with references, but checking how the answer was reached can be difficult. To understand the literature yourself, you need control over where you look, what you keep, and what you take from each paper.

Specialised tools such as Elicit and Consensus give you more control, but you still rely on their method. A desktop agent such as Codex or Claude Code, working with local files and skills, lets you direct the process. A skill is a set of instructions for a task, such as organising references or reading a paper. You give up some tested presets, and a provider's tool may sometimes do better. Services like Elicit can also be used by the agent directly as part of your inquiry.

Our question concerns systems intended to carry out parts of scientific research. We want to understand the evidence for what they can do. Start by finding that evidence, then decide how to examine it.

2. Analyse and organise

The refined question gives us a reason to group the references. System reports show us the claims. Papers about evaluation methods help us understand the measures. Possible independent tests give us checks to investigate. Zotero keeps these groups connected to papers and notes.

Step 1 of 6: Your question
AI-scientist evaluations Mock chat
Compare the saved results with my Zotero library. Show me duplicates and different versions of the same work. Group the references into system reports, evaluation methods, and possible independent tests. Keep a group for material we set aside, with a reason for each choice.

Different versions of the same work should be considered together. An announcement can be set aside from the comparison while we follow its link to a paper. Keep that reason in the record, so the next reading decision starts from what we already know.

The agent loads my public Zotero Wrangler skill and reads its instructions before working with the library. You describe the organisation you need; the skill tells the agent how to do the Zotero work.

Illustrative proposed Zotero collections organised around claims, measures, and possible checks
A proposed organisation for the example inquiry; no real library changes are shown.

Try the reference manager

From search results to a Zotero collection

Follow what changes as the agent organises the references. Select a collection or a paper to inspect it.

Illustrative simulation · All papers, passages, and decisions are invented. This page does not connect to your Zotero library.

Loaded skill · Zotero Wrangler

Before organising the references, the agent loads the Zotero Wrangler skill and reads its instructions. The researcher supplies the task; the skill tells the agent how to do the library work.

Step 1 of 5

1. Bring the search results together

Five records have arrived from the example searches. Keep where each came from, so we can return to the search that found it. Two records refer to the same paper; we will inspect that next.

Zotero · example library5 references

References

What the agent keeps in the project record

Open the Zotero simulation on its own

3. Extract what answers the question

For this inquiry, a general summary is too open. We need to know who ran each test, what the system did, what was measured, and how the result was checked. Decide these questions before asking the agent to read across the selected papers.

Put the passage behind each answer beside it. If the agent says a test was independent, we can examine the words that led to that interpretation. If the paper does not establish it, record that as an unanswered question.

Step 1 of 5: Your question
AI-scientist evaluations Mock chat
Find the selected papers in the project and Zotero. For each evaluation, extract who ran it, their relationship to the developers, the task, what was measured, and how the result was checked. Put each answer beside the supporting passage and its page or section. Keep the passage separate from your interpretation. Mark what you cannot establish.

My Academic PDF to Markdown skill helps turn a PDF into readable text with figures and tables. My Paper Reviewer skill gives the agent instructions for examining a paper's question, methods, and evidence. Your inquiry determines what to extract.

Illustrative extraction table placing each interpretation beside an invented supporting passage and section
In your inquiry, each passage would come from the linked paper. These passages are invented to demonstrate the comparison.

Keep a record and a visual overview

The record grows throughout the inquiry: which search found a paper, why we put it in a group, and which passage supports an answer. My Research Reproducibility skill helps the agent keep that account.

A table compares papers. A chart by year shows when the papers in our collection appeared. A literature map shows how the groups relate to our question. Open each group below to see what it helps us ask next.

How are AI-scientist systems evaluated — and who checks the claims?

Illustrative literature map

What is claimed?

Building an AI scientist

Read the system report's evaluation and compare it with the extraction table. Which claims need a check by another team?

Ask the agent

Map the literature we have kept. Link each group to its papers and our notes. Show what we still cannot answer and suggest where to search next. Keep the reasons for changes in the project record.

Illustrative map connecting system claims, evaluation measures, and possible independent checks
A visual overview helps us decide what to read or search for next.

Some results provide abstracts rather than full text. The results are a sample of the literature, not everything that exists. Check the paper and its supporting passage before citing it.

Researchers and students in control

Deep research was a first attempt at using agents to improve retrieval and synthesis. Now researchers and students can direct the process themselves: find and save the sources, analyse and organise the references, then extract what their question requires. The agent helps with the work and keeps it visible. You decide what the evidence means and where the inquiry goes next.

Start with your own question

I want to understand how AI-scientist systems are evaluated and who checks their claims. Search several scholarly services and the web, and look in the papers I already have. Save every result you retrieve and the history of your searches in one project folder. Show me a table of what you found and why each result might matter. Help me refine the question before we organise the references and read the selected papers.

Read the full guide

Cite this page

Lukeš, Dominik (2026). Beyond Deep Research. Semantic Machines explainers. https://explainers.semanticmachines.fyi/beyond-deep-research/