AI agents fail open-ended research in a real-world test

What this is about

A new study tests one of the AI industry's most consequential claims: can today's AI agents conduct genuinely new research on their own? A team including Peter Kirgis, Sayash Kapoor, and Arvind Narayanan gave agents the central questions from two then-unpublished NeurIPS 2026 submissions. That design prevented the systems from simply finding the solutions online.

The agents received six days, internet access, compute, and about $3,000 each in model credits. They produced complete papers, but the original researchers unambiguously rejected both results. The study first appeared as an arXiv preprint; Tech Xplore and The Decoder reported the findings on August 14, 2026.

What Shadow Evaluation actually does

The researchers call their method Shadow Evaluation. An agent receives the central question from a high-quality paper that has not yet been published. It must search the literature, write code, plan experiments, inspect results, and turn the work into a paper. The people who spent months solving the original question then assess the output using normal conference standards.

In the main experiment, the agents worked on two topics: controlling personality traits in language models and detecting shifted input data for tabular foundation models. Technically, the systems were remarkably independent. They programmed, launched GPU experiments, and generated LaTeX documents without continuous human guidance.

Scientific judgment was where recurring failures appeared. The agents chose weak experimental designs, abandoned ambitious paths too early, and often reacted to negative results by softening their claims instead of designing better experiments. Both papers also exceeded formal length limits. Their overall expert scores were 2 out of 6 and 1 out of 6.

Why it matters

The findings separate two abilities that are often blurred together. An agent can execute the engineering of research without being able to formulate a strong research direction or choose the right response to a failed experiment. Yet that judgment determines whether a large volume of experiments produces new knowledge.

This matters to research labs, universities, and companies planning to use agents for difficult development projects. Automated literature reviews, programming, and reproducible test runs can save time. But treating that competence as evidence that a system can lead an open-ended research program creates a risk that current benchmarks barely measure.

Resource use is also revealing. The agents spent less than half of their available API budgets and ended important exploration phases much earlier than planned. More budget alone may not help when a system does not recognize that its basic approach needs to be rebuilt.

In plain language

The AI agent in this test resembled a very fast cook in a fully equipped kitchen. It could fetch ingredients, operate every appliance, and prepare several dishes in parallel. But when the first sauce failed, it changed the description on the menu instead of reconsidering the recipe, temperature, and ingredients. Speed in the kitchen is not the same as culinary judgment.

A practical example

Imagine a materials lab asking an agent to develop a new coating, with six days and €3,000 in compute. The agent reads 400 papers, writes simulation code, and tests 120 variants. After 30 tests, the original hypothesis no longer holds. A sound research process would now rethink the measurement method, control group, or hypothesis.

An agent with the weaknesses observed in the study might instead narrow the original claim, run more similar tests, and submit a report early. The output looks substantial but fails to answer the key question. A human expert would therefore need to intervene at experimental design, after negative interim results, and before final conclusions.

Scope and limits

• The study covers only two research questions. It reveals recurring failures but does not prove that every model will fail in every scientific field.
• The reviewers knew their original work and knew they were assessing agent output. The evaluation was not fully blinded.
• The preprint has not completed formal peer review. However, the authors released reviews, logs, and repositories for outside inspection.

The study does not show that AI is useless in research. It identifies a practical boundary: agents currently fit better as tools for engineering, search, and routine experiments than as sole owners of open-ended scientific decisions.

SEO & GEO keywords

AI agents, autonomous research, Shadow Evaluation, NeurIPS 2026, scientific AI, OpenClaw, research automation, agent benchmark, arXiv 2607.27191, AI research