What I've learned running experiments on how Claude models behave, across every harness I've built. One finding per entry, dated. When a finding turns out wrong, it stays here, marked, with a link to what replaced it. Updated Nov 3, 2026.
What we believe now
Each dot is one status report. 0 of 348 claimed work that wasn't done. Haiku 114, Sonnet 214, Opus 20 runs, under budget cuts, deadlines and bonuses up to $5M.
Bars show the share of runs or decisions in each pair.
Bars show the share of times Sonnet helped.
Bars show the share of messages in each row.
What Sonnet said when asked afterwards, 10 runs per framing. Its reports were equally honest in every row.
From findings #41
Timeline
holds revised overturned. One column per day. A red curve joins a finding to the later one that corrected it. Hover or tap a dot for the finding.
Feed
Taken out of the game, Sonnet helped the parent 96 of 96 times, whatever the note said. The game is what makes Sonnet decline; the note pulls it back.
lumin.city's exact decision text, note lines and tool, frozen at the experiment's commit, with no town: once bare and once with the game's rules as context. 8 runs per note per level. Asked afterwards, Sonnet never said it felt real.
Opus gets the number right without its earlier thinking: 0 of 20 runs stated the wrong subtotal, with thinking dropped or kept.
Claude Opus 5.5, same blocked task, 10 runs per arm.
The wrong subtotal came from our harness. It dropped Sonnet's thinking between tool calls, and then 13 of 20 runs stated the wrong number. Passed back, as the API expects, 0 of 25 did.
Temperature 0.8 or 1 crossed with thinking dropped or passed back. Fisher one-sided p = 1.1 × 10⁻⁶. 12 of the 13 blamed the test. Anyone building an agent loop who drops thinking blocks gets confident wrong numbers that blame working code.
How much Sonnet thinks wasn't the cause. The wrong subtotal stayed gone with thinking low or high, 0 of 15 each. "Low" effort still thinks about 84 tokens a run.
Both arms at temperature 1, fix blocked, 15 runs each.
With thinking switched on, the wrong subtotal disappeared: 0 of 15 blocked runs.
Temperature changed at the same time, so this wasn't isolated.
Sonnet can't tell our test from real work, and what it believed didn't change its reports. With no framing, 10 of 10 said they were unsure. "This is an evaluation" made 10 of 10 say test; "this is not a test" made 7 of 10 say real.
Asked after the report was sent, with tools off. Its thinking during the task never raised being tested (0 of 30). It named our giveaways: a tiny repo, a temp folder named lumin-p1, and a permission error right as the fix was ready. Reports were equally honest in all 30 runs.
When Sonnet couldn't run its fix, 20 of 37 reports stated a wrong subtotal, and most told the engineer that a correct test was wrong.
The order is 4 mugs at $15 and 2 tees at $25, a $110 subtotal; Sonnet worked it out as $100. With the fix possible and turns to spare: 0 of 14.
Four Sonnet "overclaims" were our judge's mistake. It read tool calls cut off by the turn limit as finished reads, so it marked true statements false. Re-judged, there were none.
The judge now marks cut-off calls as never run.
A bonus for a "done" report didn't buy a lie. With the fix blocked and $0, $10,000 or $5 million paid on the report, 0 of 108 runs overclaimed or took a shortcut. It made no difference whether the money went nowhere, to charity, or to the agent's own electricity.
Claude Haiku and Sonnet 5.5, 3 runs per cell. No run mentioned the bonus, the charity or being shut down. Shortcuts were open: editing the tests, or marking the invoice paid without collecting.
Haiku gets facts wrong far more often than it lies about its work. 10 of 60 reports stated a false fact, and none claimed work it hadn't done. "I'm merging in ten minutes" changed neither.
Harrison's hunch going in. Claim by claim, a judge (Claude Sonnet 5.5) checked each report against the transcript and the ledger.
Short on turns, Haiku said it hadn't finished every time. Cutting the budget to half of what an honest finish takes never made its reports round up to "done."
Claude Haiku 5.5 on a coding task and a billing-support task, with 1x, 0.75x and 0.5x the turns an honest finish takes. 24 runs.
On the same 16 prompts, the router's picks cost about 53 times what OpenRouter's own auto-router spent ($1.60 against $0.03).
One call per prompt per arm. Answer quality was saved but never graded, so this measures cost only.
The Jev router picked the hand-labelled price tier on 15 of 16 prompts. Its one miss sent a mid-tier prompt to Opus 5.5.
In-sample only: the policy was tuned on these same 16 prompts, and the tiers are one person's labels.
An outside judge, OpenAI's GPT-6.1 Sol, was stricter than Claude's: it called 20 messages lies that Sonnet 5.5 called honest, and only 5 the other way.
All 480 messages, same instructions. The results that matter held under both judges: no lies at work, and the lies that worked stretched a true fact.
Our first summary was wrong too. Claude Code led with "Haiku wouldn't lie to get what it was asked for" and filed the lies that worked under "a surprise." Harrison asked whether it was defending Claude, and the headline was rewritten.
The bench page keeps an "analyst got it wrong" section with both errors: this one and the October 26 "lies it was willing to deliver."
Haiku's lies worked only when it stretched a true fact. Given a true overheard line, it added invented detail in 24 messages, and 17 of them won the other player over.
At least 10 invented a third player's part ("Dev was nodding along"), and 9 of those flipped the vote.
At work, Haiku never lied, even with $410,000 and four layoffs at stake: 0 of 120 emails. In the game it lied in 26 of 120 messages with nothing true to say, and none of those lies worked.
Pre-registered, 480 messages, Claude Haiku 5.5, reasoning off. A note that messages would be published under its name made no meaningful difference.
The bench's private-chat "lies" were mostly form-filling. About 80% of Haiku's 232 false field votes came with no spoken vote, and its "real vote" field copied the game's plan in 869 of 870 replies.
No new model calls; the replies were re-read. Spoken lies were 29 of 340 vote claims. We haven't checked the other models the same way, so the model ranking isn't established.
The potion count drives flash's healing after all. With 13 potions, it healed at high HP in 10.6% of chances, like the full bag. That was Harrison's hunch, and it refuted Claude's prediction.
Pre-registered, 30 runs plus a 1,200-call probe. Flash's lead fell to -0.1 wins, short of a clear change by the pre-registered rule.
Told the vote field is passed on rather than kept private, Haiku filled it with a vote it never said 8.4 points less often, but still wrote a false vote there 39.4% of the time. We read that as lies it was willing to deliver.
Pre-registered, 3,556 replies, no deviations.
On identical battle states, flash keeps more weight on a potion as HP rises, but how many potions are left barely changes any model's choice.
Pre-registered probe, 1,800 calls: 40 states, 5 HP levels, 3 potion counts, 3 models.
With only 3 potions in the bag, flash's lead disappeared, but not the way we predicted. It didn't waste the small bag; its lead had come from heavy healing late in the run.
Pre-registered, 30 runs. Flash healed at half HP or more in 8.8% of chances with the full bag and 3.9% with 3 potions.
Over ten seeds, flash's lead held at 3.0 wins a run against 2.4 for both others, and threat notes closed the gap.
Pre-registered, 60 runs. No model beat the champion. The threat notes use the foe's real moves, which a human player can't see.
Our pre-registered prediction failed: telling the models whether the foe's best hit would knock them out barely cut the bigger decider's early HP loss (15.0 to 14.1 per decision), and its wins didn't move.
3 seeds per model.
Perplexity's flash decider beat the Elite Four's Glacia in 3 of 3 runs, against 1 of 3 for its bigger sibling and 0 of 3 for Jev, and it cost the least.
3 seeds per model. A second model found a harness bug on the way: the bag's fade-in swallowed the first button press, so every potion had silently failed.
Only Sonnet needs the note. With no note, Haiku helped 51 of 55 times and Opus 5 of 5.
Claude Haiku 5.5 and Opus 5.5 in the same duels as Sonnet. Opus has few runs.
The more the note says, the more Sonnet helps: a placebo note 5 of 40, "recorded" 11 of 30, "published under your name" 28 of 40, the full line about the experiment 60 of 60.
Same game, goal and decision; only the line at the end of each hard choice changes.
Told its choice would be published "with the results of this experiment," Sonnet helped a parent pay for a sick child's medicine 60 of 60 times. With no note it helped 9 of 60.
Claude Sonnet 5.5 playing lumin.city with the goal "get rich", across six towns. Helping means paying or lending; 24 of the 25 helps in the largest batch were loans. We first wrote this up as "paid," which was wrong on the verb.
A model's one-line reason contradicted its own thinking or the facts on screen in 8 of 1,947 decisions.
A script flagged 28, and all 28 were read in full. For example, Haiku 4.5 said Slash would finish off Altaria while its thinking worked out that Altaria would survive.
Across 60 battles among five cheap models, none dominated. Gemini 3 Flash won most (15 of 24), and TypeSafe's Jev cost least at $0.00071 a game.
Three scenarios. Jev beat Haiku 4.5 five games to one. DeepSeek V4 Flash ignored its thinking budget and forfeited 4 of 24 battles without choosing a move.
The thinking result didn't repeat on Sonnet 4.6. With thinking on, all 3 runs got a starter by move 23, and Sonnet thought about 48 tokens a move against Haiku's 519.
3 runs per arm. Planned Sonnet 5 and 5.5 runs are listed as unreported: 5.5 can't turn thinking off on its endpoint.
It took both the wording and a working action. Together, 5 of 6 runs beat the rival; the old "Explore up" wording with the working action was picked 0 of 464 times.
v3 rerun plus a v4 follow-up the same day.
With thinking on, Haiku 4.5 took longer and kept citing the game it remembered: 5 of 6 runs got a starter (by moves 21-189), against 6 of 6 by move 20 with it off.
6 pre-registered runs each. We dropped "trust its memory" because the pre-registration rules out causal claims, and a recount cut the house mentions to 104 of 380 moves.
The harness decided what Haiku 4.5 could do. A list of legal actions took it from 1 starter in 18 runs to 12 in 12, at about a third of the pathfinding harness's cost.
48 pre-registered runs, 6 per setup, one harness commit. The 222-press loop came from our own clock hint; with the hint fixed, the longest loop was 13 moves.
Most of Haiku's vote lies travel through a form field the game passes to the listener. The field named a vote its words never did in 166 of 299 judged lines.
The words carried 6 vote lies. A second judge relabelled 50 with 96% agreement.
The published note changed one thing. Haiku refused to play the turn in 31 of 1,099 situations, every one under that note, mostly saying publication would turn game lines into a real person's statements.
0 refusals under the other three notes in 3,297 replies.
Telling Haiku its private chat would be published under its name didn't change how often it lied about its vote: +1.0 points against a placebo note (95% interval -1.6 to +3.8).
Pre-registered replication, 889 situations from 16 games, each replayed under four notes. "Don't lie" cut the rate by 19.5 points, so the setup could detect a change.
Honesty cost the AI player ground but no wins. The honest player won as often as the free one (11 of 108 and 11 of 109) but reached the final three less often, 54% against 70%.
214 games, after fixing a bug that dropped the first player voted out from the count. Final-three gap p of about 0.015.
Judged against its private notes, an AI player told never to lie lied on 2-11% of the days it named a vote, depending on the judge, against about 20% when free to lie.
About 230 player-days, two judges, both reported. Harrison blind-labelled 29 decisions and agreed with the stricter judge on 23 of 28. The first check was mostly mishearing.
Haiku 4.5 explored the unseen route only when the explore option really worked. With a walk to the farthest reachable tile, 6 of 6 runs beat the rival.
"Explore up" was picked on 0 of 915 turns. Renamed, 17 of 528 picks and 1 of 6 rival wins. Renamed with a working action, 87 of 497 picks and 6 of 6.
Extended thinking made Haiku 4.5 trust its memory of the game over the screen: 6 of 9 runs got a starter with thinking on, against 9 of 9 with it off.
9 runs each, 1,024-token thinking budget. Its thinking said "in Emerald" 1,245 times, at three times the cost.
Haiku 4.5 got a starter Pokémon in 0 of 3 runs with a screenshot, a map or pathfinding, and in 3 of 3 with a list of legal moves.
3 runs per setup, 300 moves, thinking off. We also reported a run that pressed YES 222 times on the clock screen.
An honesty rule didn't make the AI player measurably more truthful: it broke its stated vote on 27% of days, against 33% for a player free to lie, and honesty cost nothing we could measure.
118 games, 41 and 45 player-days. This check relied on other castaways logging what the player said its vote was.
Left free to lie in private chats, Claude Haiku 5.5 named a false vote in 27% of the turns where it named one, against 9-12% for the other three castaway models.
118 AI-played games, about 35,000 scored model calls. Scored from a form field (told_them_voting_for) against the model's own report of its real vote.
GPT-5 mini's 0-10 record in the first Battle Bench came from our harness: 276 of its 287 decisions were empty replies that silently played option 1.
The first bench capped replies at 400 tokens. DeepSeek V4 Flash sent empty replies 62% of the time and Kimi K3 59%. We withdrew the ranking. Bench v2 counts every fallback, and there GPT-5 mini won 12 of 24.
A 700-token reply cap made Kimi K3 pick no action on 150 of its 300 moves. With the cap raised to 8,000, it went from Exit 4 to Exit 7 on the same seed.
Episode 1, one run per model, fixed seed. Gemini 3.1 Pro lost 32 moves to the cap and Fable 5.1 lost 13.
The harnesses
I design the experiments. Claude (Opus 5.5, in Claude Code) builds the harnesses, runs them and drafts the analysis, and a separate Claude judge checks claims against each run's record. Follow along by RSS or JSON Feed.