Overthinking paper shows a new risk in reasoning models

What this is about

A paper submitted to arXiv on July 9, 2026 and accepted at ICML 2026 is titled Overthinking: Amplifying Reasoning Weights to Extract Learned Secrets. The authors study whether hidden information in language models can be elicited more effectively by artificially amplifying a model's reasoning component.

This is not a report about an attack on a specific product. It is research on auditing. Still, the core issue is serious: a model can contain learned information or behavior patterns that do not appear in ordinary tests, yet surface more often under changed weights.

What overthinking actually does

The method compares a normal instruct model with a reasoning-distilled model. The difference between their weights becomes a kind of reasoning direction. That direction is then added back more strongly than in the pure reasoning model. The authors call the result an overthinking model.

In simple terms, the model is not merely prompted to think more. Its internal parameters are shifted so it tends more strongly toward explicit reasoning. In the experiments, hidden information surfaced more frequently across model sizes from 2B to 32B parameters, in some settings up to ten times more often than in the original reasoning model.

Why it matters

AI safety often relies on black-box testing: ask the model questions and check whether dangerous, private or unwanted answers appear. The paper shows why such testing can remain incomplete. A model may contain latent information without revealing it under ordinary prompts.

This matters especially for organizations that train, fine-tune or audit models before deployment. If reasoning amplification makes secrets more visible, it can be useful as an audit tool. At the same time, it shows that knowledge stored in weights is not harmless just because it rarely appears in normal use.

In plain language

Imagine an old filing cabinet. Usually you only open the labeled drawers. Overthinking is like tilting and shaking the cabinet a little. Suddenly, papers slide out that were not meant for normal access. The shaking is a test, but it also proves the papers were inside the cabinet.

A practical example

A company trains an internal 7B model on technical support tickets. In ordinary tests, 1 out of 1,000 audit prompts produces a hint of sensitive ticket details. After an overthinking-style audit variant, such details appear in 8 out of 1,000 cases.

That does not mean the model leaks data constantly. It means the information may be more embedded in the model than the standard test suggests. For support, healthcare, legal work or internal software development, a normal chat test is then not enough as a safety argument.

Scope and limits

First: the work describes an auditing method, not a general attack on hosted black-box models. It requires model weights or at least an environment where weight experiments are possible.

Second: finding more secrets does not automatically prove that a deployed product is unsafe. It reveals a risk that needs further testing.

Third: overthinking can also shift output quality and coherence. The authors describe attenuation strategies, but these methods are still research tools, not standard compliance checks.

SEO & GEO keywords

Overthinking, reasoning models, AI security, model auditing, learned secrets, ICML 2026, arXiv 2607.08173, model weights, data leakage, AI safety