AI Research Engine

Objective

Test whether an AI system can turn a strategic question into a brief that a decision-maker can verify without redoing the research. The measure of success is not fluency but traceability: how much of the brief is tied to a specific passage the reader can check.

Technical architecture

Retrieval. The four published papers are split into sections of about 2,000 characters. A BM25 keyword index ranks them against the question, and the six most relevant are kept.
Generation. The sections are passed to Claude as separate documents with citations enabled. The model writes the brief and returns, for each claim, the exact passage it relied on. When web search is switched on, the model can run up to four searches, and web sources are cited in the same way.
Server boundary. All model calls go through a server route on this site. The API key never reaches the browser, inputs are length-limited, and requests are rate-limited per connection.
Rendering. The brief is rendered with numbered citation markers. Each source is listed with the passages quoted from it, along with every section that was retrieved and its relevance score. Model output is escaped before rendering, so it cannot inject scripts or unsafe links.

Methodology

Evidence first. The model is instructed to use only the supplied documents and search results, to cite every factual claim, and to say plainly when the evidence does not answer the question.
A fixed structure (answer, key evidence, gaps and disagreements, next questions) makes briefs comparable and puts uncertainty on the page instead of hiding it.
Native citations. Citations come from the API's citation feature, not from the model writing references itself. Each one points to a span of a document actually supplied, which removes the risk of invented references.
Cited coverage, the share of the brief's text that carries a citation, is calculated for every brief.

Evaluation

Planned evaluation, once the live connection is switched on:

Measure	How it is assessed
Cited coverage	Calculated automatically for every brief
Citation accuracy	A reviewer checks whether each cited passage actually supports the claim it is attached to
Unsupported claims	Factual statements with no citation, counted by a reviewer
Appropriate refusal	Questions the papers do not cover, to check that the engine says so
Reviewer time	Time to verify a brief, compared with verifying an uncited answer to the same question

The test set will be 20 questions: 15 the papers address, and 5 they do not.

Results

No results yet. The engine is built and tested against simulated responses; results will be published here after the first evaluated runs on the live model.

Limitations

Keyword retrieval. BM25 can miss relevant sections that use different words from the question. Semantic (embedding) retrieval is the obvious next step.
A small corpus. It contains four papers, so many questions fall outside it. That is a useful test of refusal, but it limits usefulness without web search.
Citations show where a claim came from, not whether it is right. A correctly cited passage can still be misread, which is why citation accuracy needs human review.
Web results are only as good as the pages found. The engine does not yet rank sources by credibility.
Rate limits. Per-connection limits are best-effort, so a spending limit should also be set with the model provider.

Code and resources

The engine runs on this page through the site's server. A public repository will be linked here once it is published.