Building an AI scientist
Example Research Team · 2025 · System report
Found via: Broad evaluation search
Read the developers’ own evaluation to establish what they claim and how they measured it.
Collection: System reports
How are AI-scientist systems actually evaluated — and who checks the claims? Follow one inquiry through finding papers, organising references, and extracting the evidence.
The conversations, tables, and Zotero simulation use invented examples. The Elicit search example describes saved searches from research on AI-scientist evaluations.
Use Next step in each conversation to follow the question, the agent’s preparation, its use of the tools, and the saved results. Previous step goes back; Replay starts again.
Deep research tools write reviews and reports from sources they chose. You get text with references, but checking how the answer was reached can be difficult. To understand the literature yourself, you need control over where you look, what you keep, and what you take from each paper.
Specialised tools such as Elicit and Consensus give you more control, but you still rely on their method. A desktop agent such as Codex or Claude Code, working with local files and skills, lets you direct the process. A skill is a set of instructions for a task, such as organising references or reading a paper. You give up some tested presets, and a provider's tool may sometimes do better. Services like Elicit can also be used by the agent directly as part of your inquiry.
Our question concerns systems intended to carry out parts of scientific research. We want to understand the evidence for what they can do. Start by finding that evidence, then decide how to examine it.
At first, we do not know how the literature describes evaluation. Search several places: Semantic Scholar, Google Scholar or Google, Elicit, and papers already in the library. The variety helps us see what the question needs to include.
Ask the agent to save every result it retrieves and the history of the searches in one folder, with PDFs where available. We will want to know what we asked, where we looked, which limits we used, and why we changed direction. Keeping the results also lets us reconsider something we first set aside.
An API lets the agent send a research question directly to Elicit and collect the paper records it returns. You describe what you want to find. The agent searches for Elicit’s official guidance, finds the API documentation, and reads how to make the search work. If API access is needed, it asks you to authorise it. You do not have to supply the technical instructions for using Elicit.
The agent then prepares the searches from the guidance it found, uses Elicit’s API, and saves the results with material from other sources. Looking up how to use the service is part of its work. Elicit’s API guide describes this direct access.
The saved example uses two complementary searches. One asks broadly for empirical evaluations and independent checks of AI-scientist systems, including expert or experimental validation. The other focuses on particular life-science systems and benchmarks. Both look for papers from 2025–2026 and ask for up to 25 results. Each returned 25 records; the agent saved the questions and results and brought the 50 returned rows into one table.
The records include references, abstracts where available, and links to papers. A paper can appear in both searches. These records give us sources to investigate; deciding what to keep and extracting evidence from full papers are the next parts of the inquiry.
A prompt for that searchUse Elicit to look for empirical evaluations and independent checks of AI-scientist systems. Also look for evaluations of the life-science systems and benchmarks we are investigating. Find papers from 2025–2026, up to 25 for each question. Save both questions and all the returned results, with references, available abstracts, and paper links. Keep a note of which search found each paper.
Found Elicit’s official API guide and API documentation. Read how to request paper searches, choose the publication years and number of results, and collect the returned references.
Elicit provides an API I can use for these searches. Do you have Elicit API access I may use? I have found its guidance and can work out the search details from that.
Ran a broad search for evaluations and independent checks, and a focused search for life-science systems and benchmarks. Saved both questions and all returned paper records. Also searched the other scholarly sources, the web, and the project papers.
The first results suggest three things to look for: the developers’ own tests, ways of measuring research ability, and checks by other teams. The announcement is a lead to a paper; it has no evaluation detail. We can use those distinctions to refine the next search.
Separate system reports, evaluation methods, and possible tests by other teams. Show what you can tell from the titles and abstracts and what needs the full paper. Then search for independent tests of the systems we have found. Save the new results and explain how this changes our question.
We now have a more precise question: which claims have been checked by someone other than the team that built the system, and what did that check involve?

The refined question gives us a reason to group the references. System reports show us the claims. Papers about evaluation methods help us understand the measures. Possible independent tests give us checks to investigate. Zotero keeps these groups connected to papers and notes.
Read the Zotero Wrangler instructions for comparing references, keeping notes and search sources, and organising collections. Use those instructions to carry out the researcher’s request.
These groups let us compare claims, measures, and possible checks. Open each collection below. A possible independent test still needs reading: its title does not tell us enough about who did the work.
Different versions of the same work should be considered together. An announcement can be set aside from the comparison while we follow its link to a paper. Keep that reason in the record, so the next reading decision starts from what we already know.
The agent loads my public Zotero Wrangler skill and reads its instructions before working with the library. You describe the organisation you need; the skill tells the agent how to do the Zotero work.

Try the reference manager
Follow what changes as the agent organises the references. Select a collection or a paper to inspect it.
Illustrative simulation · All papers, passages, and decisions are invented. This page does not connect to your Zotero library.
Before organising the references, the agent loads the Zotero Wrangler skill and reads its instructions. The researcher supplies the task; the skill tells the agent how to do the library work.
Five records have arrived from the example searches. Keep where each came from, so we can return to the search that found it. Two records refer to the same paper; we will inspect that next.
Example Research Team · 2025 · System report
Found via: Broad evaluation search
Read the developers’ own evaluation to establish what they claim and how they measured it.
Collection: System reports
Example Research Team · 2025 · Same paper found again
Found via: Targeted follow-up search
The example provides the same paper identity for both records. Bring their search sources together in the next step.
Example Methods Group · 2025 · Evaluation methods
Found via: References in a known paper
Use this paper to ask what a test measures, and what evidence would answer our question.
Collection: Evaluation methods
Example Laboratory Team · 2026 · Evaluation study
Found via: Search for tests by other teams
Read the methods and the relationship to the developers before deciding whether this is an independent check.
Collection: Possible independent tests
“Our laboratory, which did not develop the system, repeated two of its proposed experiments.”
Location: Methods section · invented passage
Interpretation: A separate team checked selected experiments. Read the results before drawing a conclusion.
Example Announcement Team · 2026 · Announcement
Found via: Web search
This announcement may lead us to a paper, but gives no evaluation detail for the comparison.
Collection: Set aside / revisit
Decision: Set aside from the evaluation comparison. Keep the lead and return to it if it helps us find the underlying paper.
Five records have arrived from the example searches. Keep where each came from, so we can return to the search that found it. Two records refer to the same paper; we will inspect that next.
The example identifies two records as the same paper. Keep one reference with both search sources. We now have four references, and can still see how each was found.
Group the references by what we need to learn: the developers’ claims, evaluation methods, and possible independent checks. Select a collection to see its references. The announcement is still waiting for a decision.
The announcement gives no evaluation detail. Keep it in Set aside / revisit with that reason. It remains in the project collection, so setting it aside does not lose the reference.
The selected paper now has a note explaining who checked the work, with the passage and section behind that interpretation. Select a collection and paper to compare the notes.
Loaded the Zotero Wrangler skill and read its instructions, then saved the search questions, the five returned records, and their search sources.
Consolidated the example’s known duplicate into one reference; retained both search sources. Four references remain.
Added collections for system reports, evaluation methods, and possible independent tests. Recorded why each paper might help answer the question.
Kept the announcement in Set aside / revisit with the reason “no evaluation detail”. The full project collection still contains four references.
Attached an illustrative reading note to the evaluation paper, including the supporting passage, its section, and the interpretation to check.
Step {step} of {total}1 reference{count} referencesNo references in this collection at this step.For this inquiry, a general summary is too open. We need to know who ran each test, what the system did, what was measured, and how the result was checked. Decide these questions before asking the agent to read across the selected papers.
Put the passage behind each answer beside it. If the agent says a test was independent, we can examine the words that led to that interpretation. If the paper does not establish it, record that as an unanswered question.
One example reports the developers’ own scores on prepared tasks. The other describes a separate lab repeating selected experiments. Those tell us different things. The passages let us examine that distinction and decide what to read next.
My Academic PDF to Markdown skill helps turn a PDF into readable text with figures and tables. My Paper Reviewer skill gives the agent instructions for examining a paper's question, methods, and evidence. Your inquiry determines what to extract.

The record grows throughout the inquiry: which search found a paper, why we put it in a group, and which passage supports an answer. My Research Reproducibility skill helps the agent keep that account.
A table compares papers. A chart by year shows when the papers in our collection appeared. A literature map shows how the groups relate to our question. Open each group below to see what it helps us ask next.
Map the literature we have kept. Link each group to its papers and our notes. Show what we still cannot answer and suggest where to search next. Keep the reasons for changes in the project record.

Some results provide abstracts rather than full text. The results are a sample of the literature, not everything that exists. Check the paper and its supporting passage before citing it.
Deep research was a first attempt at using agents to improve retrieval and synthesis. Now researchers and students can direct the process themselves: find and save the sources, analyse and organise the references, then extract what their question requires. The agent helps with the work and keeps it visible. You decide what the evidence means and where the inquiry goes next.
I want to understand how AI-scientist systems are evaluated and who checks their claims. Search several scholarly services and the web, and look in the papers I already have. Save every result you retrieve and the history of your searches in one project folder. Show me a table of what you found and why each result might matter. Help me refine the question before we organise the references and read the selected papers.
Lukeš, Dominik (2026). Beyond Deep Research. Semantic Machines explainers. https://explainers.semanticmachines.fyi/beyond-deep-research/