Robot-Whisper — Neurons see the state the tokens don't: an embodied anomaly alarm

The idea

The internal state knows more than the tokens say.

Start with an old result on reasoning LLMs: a probe we called NAD could pick, among a model's candidate answers, the one more likely to be right — without reading a single word, from neuron activity alone. Confidence and hesitation were never written into the tokens, but the internal state carried them. Take the same idea into embodied AI: would the inside of a VLA model hold a similar signal?

✓ success — both pots onto the stove38 steps · 18.6 s

✗ failure — hovering over the second pot52 steps · 26.0 s · timeout

Fig. 1Same simulator state, and the only difference is one sampling noise in the diffusion head: the left sets both pots on the stove in 38 steps; the right places the first pot, then hangs over the second, dragging out all 52 steps to timeout. The thing that separates them never appears in any token.

01

A robot that cooks

The robot here is a π0-family VLA: it looks at the scene and emits 10 joint-action frames at a time, twenty per second. The task is LIBERO-10 SCENE8 — put two moka pots onto the stove, one after the other.

02

It "thinks wrong" too

Not a wrong answer — an arm hanging frozen in mid-air, or a pot picked up and put down, shuttling between the two, until the 52-step timeout.

→

Look in the neurons

Inside its backbone, every step and every layer lights up a small, sparse set of neuron activations. If the NAD idea holds, a frozen sense of confidence should be written on that roster too.

The physical world

Still moving — but doing the wrong thing.

Fix one initial state, vary only the sampling noise, run 32 rollouts: 14 succeed, 18 repeat the same motion until timeout. Some hover without moving; some circle in place — two faces of the same thing: caught in a trap. And on the action stream, nothing shows.

Cumulative-change curves of 32 same-state rollouts: blue successes climb until they finish, orange trapped runs flatten after a dozen centimetres. — Fig. 2The physical trap, illustrated: accumulate, step by step, how much the pot-to-stove distance changed — as long as the world is still being changed, the curve keeps climbing. For the first half, blue and orange walk the same road: the first pot goes down (≈ +13 cm), then everyone pauses. **The fork comes around step 30** — blue = success restarts and sprints to its dot; orange = trap **never gains another centimetre** after that pause: the motion goes on, and the world is never changed again.

01

The action stream stays normal

At every stuck step the model still emits 10 action frames of ordinary magnitude — the action sequence cannot tell "working" from "spinning".

02

The verdict comes at the end

Success waits for a boolean check to flip at the last moment; and "long" is ambiguous — a slow success and a fast failure read the same.

→

To see it, look elsewhere

Hovering or circling, the action stream reads the same — the trap is not written in the actions. Next: into the neurons.

Neurons

Thought flows. A trap is thought, frozen.

At every control step the model lights a small set of neurons — across four deep layers and 11 action tokens, together they form the shape of the model's thinking at that instant. We call it the pattern. At first we tried to read anomaly out of who was lit. Identity gave no answer: it is a fingerprint of the scene, wholly reshuffled by a new initial state. What carries meaning is its dynamics — when things go well, the pattern turns over with the world, thought in flow; when the run is caught, the turnover stops, the same neurons fire again and again, and the thinking freezes onto a single instant. The signal is not in the content. It is in the flow.

Neuron-activation heatmaps of a success and a trapped rollout, shallow L4 and deep L14; the trapped deep layer pulls into horizontal stripes after t30. — Fig. 3A pair from the same initial state: left = success, right = trap; top = shallow L4, bottom = deep L14. After the gold line (t30), the bottom-right pulls into **horizontal stripes** — the same few neuron rows lit through to timeout. That is the trap, written inside the neurons.

The alarm

Turning the freeze into an alarm.

In plain terms: think of the neurons lit at each step as one point in a high-dimensional space. A rollout traces those points into a path. While the arm is working, the path keeps moving; the moment it falls into a trap, the path stops moving — even though the arm is still waving. So the alarm never has to understand the task. It only straps a speedometer to that path: when the speed drops to nearly zero, the run has stalled.

d_s

How far it moved this step

Compare the neurons lit on adjacent steps: the more they overlap, the less this step moved. Measure that displacement as 1 − intersection/union, average it into one number, d_s — the path's instantaneous speed. It never asks which neurons are lit, only how many changed.
r(t)

Measured against itself

Every run cruises at its own speed, so there is no universal scale to compare against. Take its own first 8 steps as 1, then divide the mean of the last 4 steps by it — the ratio r(t) simply says how much slower it is now than when it started. Below 1 it is slowing down; sliding further down, it is coming to a stop.

r(t) = W⁻¹Σs = t−W+1…t d_sW₀⁻¹Σs = 1…W₀ d_s, W = 4, W₀ = 8
θ·K

When it fires

Three consecutive control steps with r(t) < 0.95 — i.e. more than 5% slower than its own start — and the run is ruled stalled: the alarm fires. No exemptions, no context; all four constants were frozen before the rescue experiment and never retuned across data.

This alarm knows nothing of the task — only the path. From start to finish it asks one question: is this orbit still moving? To make an analogy: it is like an ECG taped to the model — the moment the beat stops, it speaks.

Fig. 4352 rollouts on a speed chart (x = r(t), how fast it is now; y = Δr, still speeding up or slowing down): successes settle around the cruising zone, loops run their closed cycles, stagnation decelerates into the stall point, gold rings flashing at the alarms. Sample orbits are schematic; the scatter and the two zones are measured. Click to pause. Blue = success final states, circling the cruising zone; filled orange = stagnation, collapsed at r ≈ 0.45 — stopped moving; hollow orange = looping — limit cycles that never stop moving, their final states mixed into the cruising zone, exactly the hardest family. Gold = alarm moments, all on the way down.

Results

It catches four failures out of five.

Tested online, across different tasks and different initial states — 864 rollouts. The alarm sees no images, no actions, no task labels — only the pattern.

81.2%correct online · 864 rollouts

+19.7 ptsabove the majority-class baseline (61.5%)

80%failure recall · false-alarm 17%

success

failure

Fig. 5Detection, live: left = the real pattern curves of that same-state pair, growing with playback — teal = success, holding above θ, done in 38 steps; red = failure, three steps under θ, alarm at q34 (gold ring), then sinking to timeout. Right = both rollouts in sync. The bar is draggable.

Between the two basins, a slope — not a cliff

Between the cruising zone and the stall point lies a wide, gentle slope: wherever θ is drawn on it, the false-alarm rate varies smoothly. Nudge the constants, and the conclusion stands.

Everything it names has grazed the edge

Nearly every false alarm is a slow success — an orbit that decelerates, grazes the rim of the stall point's pull, then climbs back to cruising. The alarm names them at their slowest steps, and it isn't wrong: at that moment they really are sliding toward the trap. So read the alarm as "falling", not "doomed" — exactly the reading the first-aid experiment uses.

Verdict matrix over 352 archived branches: of 235 failures the alarm catches 184 and misses 51; of 117 successes it lets 98 pass and falsely flags 19. — Fig. 6The verdict matrix (352 archived branches): of 235 failures, the alarm catches **184** and misses 51; of 117 successes, 98 pass in silence and 19 are wrongly named.

Histogram of alarm steps: concentrated between q18 and q36 with median q28; orange are alarms on failures, gold on slow successes. — Fig. 7Alarms land in q18–q36, median **q28** — half the road still ahead of the timeout, in time to intervene.

Episode lengths of quiet successes versus falsely-alarmed successes: medians 37 against 41 control steps. — Fig. 8What the alarm names is *slow*: falsely-flagged successes (gold) finish in a median of 41 steps, quiet ones (blue) in 37.

False-alarm rate as a function of the threshold: rises smoothly over the 864 pooled rollouts, no jump near the working point. — Fig. 9The false-alarm rate rises **smoothly** with θ (864 pooled) — no cliff near the working point; the constants sit on no knife-edge.

Idling is the verdict

Stuck → observations unchanged → model inputs unchanged → neuron activations unchanged — what this alarm hears is the echo, inside the neurons, of a world that has stopped. But for a robot that is supposed to be putting pots on a stove, the world is not supposed to stop: at every step the model emits actions of ordinary magnitude, and the world doesn't move — the actions themselves are wrong. The stalled world isn't weather from outside; it is a stall point of the model's own making: model → action → world → observation → model, the whole loop seized in place. What the alarm hears is that loop idling — which is also why it is reliable, cheap, and portable: the verdict needs no understanding of the task, only an ear for idling.

864 rollouts · 4 layers · sparse neuron activations layers 12–15 × 11 action tokens · denoise round 9 · 1 control step = 10 frames = 0.5 s · LIBERO-10 SCENE8

First aid!!

At the alarm step, intervene.

Detection is an offline judge; rescue is an online bet. Let the frozen detector watch a live rollout: it fires at q32 → freeze the state → inject a little perturbation, say a big change of the sampling noise → continue. Everything else stays put. A kick that knocks the path out of the stall point's pull — then see if it climbs back to cruising.

original noise

resampled

awaiting alarm …

Fig. 10The rescue, replayed live. Left = r(t) growing step by step: the ink trunk slides toward the threshold, gold ring = alarm at q32; orange = the original noise's future, sinking to a 52-step failure; teal = the resampled branch, back above threshold at q34 and done in 8. Both post-alarm branches are measured (the teal recomputed from the server's activation logs); the pre-alarm stretch is schematic. Right = the two futures in sync; the resampled frame stands by until the alarm. The bar is draggable.

Overview

The whole story at a glance — 50 seconds

A silent vector essay drawn live in your browser — curves are schematic, every number is measured. Also as a standalone page.

Summary

Constants, pages, and how to cite

One frozen alarm, one on-the-spot rescue.

Detection & rescue scripts: two collection scripts — four constants frozen before the rescue run

Pages: standalone film · Blog (EN) · 博客版(中文) · 中文版

Code: github.com/Cck123123/Robot-Whisper ↗

Cite: if you use or discuss this work, cite this page; for the NAD probe it opens with, also cite Chen et al., 2025. Both BibTeX entries are one click away below.

Community:

Robot-Whisper QQ group QR code, group number 1107443823 — QQ group · 1107443823

Robot-Whisper WeChat group QR code — WeChat group · robot-whisperthe WeChat code is refreshed periodically; if expired, use the QQ group or email

Canonical BibTeX:

@misc{cckfdu2026robotwhisper,
  title  = {Robot-Whisper: Neurons See the State the Tokens Don't —
            An Embodied Anomaly Alarm Read from Neuron Activations},
  author = {Chen, Kang},
  year   = {2026},
  howpublished = {\url{https://cckfdu.com/demo/robot-whisper/}},
  note   = {Technical demo. Code: https://github.com/Cck123123/Robot-Whisper},
}

@misc{chen2025nad,
  title  = {Do LLMs Signal When They're Right? Evidence from Neuron Agreement},
  author = {Chen, Kang and Wang, Yaoning and Xiong, Kai and Feng, Zhuoka and
            Sun, Wenhe and Chen, Haotian and Cao, Yixin},
  year   = {2025},
  eprint = {2510.26277},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  note   = {Neuron Agreement Decoding (NAD). ICML 2026 (Spotlight)},
  url    = {https://arxiv.org/abs/2510.26277},
}

Experiment constants (for reproduction):

detector   neuron-set overlap distance, layers 12–15 × 11 action tokens, denoise 9
r(t)       mean(last W=4) / mean(own first W0=8)
alarm      K=3 consecutive control steps with r(t) < θ=0.95
frozen     constants fixed before the rescue run; never tuned on it
corpus     864 rollouts · one frozen set of constants
result     81.2% correct online   (majority baseline 61.5%)
alarms     archived batch: 203 = 184 hits + 19 false alarms · q18–q36, median q28
recall     80% of failures · false-alarm 17%
rescue     alarm q32 → resample noise → 3/8 succeed · 8 vs 52 steps · 26.0 s → 3.9 s
limits     other init 0/8 · random-step also 3/8 · magnitude picks no winner (n=1)

向内探寻，向外进化。Seek within. Evolve beyond.

Neurons see the state the tokens don't —the embodied anomaly alarm from within!