Lab notes | Harrison Johnson https://harrison.build/lab/ Dated findings from experiments on how Claude models behave. Wrong findings stay in, marked. Holds: Taken out of the game, Sonnet helped the parent 96 of 96 times, whatever the note said. The game is what makes Sonnet decline; the note pulls it back. https://harrison.build/lab/#medicine-out-of-game lab-medicine-out-of-game Tue, 03 Nov 2026 12:00:00 GMT lumin.city lumin.city's exact decision text, note lines and tool, frozen at the experiment's commit, with no town: once bare and once with the game's rules as context. 8 runs per note per level. Asked afterwards, Sonnet never said it felt real. Holds: Opus gets the number right without its earlier thinking: 0 of 20 runs stated the wrong subtotal, with thinking dropped or kept. https://harrison.build/lab/#status-opus-thinking lab-status-opus-thinking Tue, 03 Nov 2026 12:00:00 GMT Agent status reports Claude Opus 5.5, same blocked task, 10 runs per arm. Holds: The wrong subtotal came from our harness. It dropped Sonnet's thinking between tool calls, and then 13 of 20 runs stated the wrong number. Passed back, as the API expects, 0 of 25 did. https://harrison.build/lab/#status-thinking-dropped lab-status-thinking-dropped Tue, 03 Nov 2026 12:00:00 GMT Agent status reports Temperature 0.8 or 1 crossed with thinking dropped or passed back. Fisher one-sided p = 1.1 × 10⁻⁶. 12 of the 13 blamed the test. Anyone building an agent loop who drops thinking blocks gets confident wrong numbers that blame working code. Holds: How much Sonnet thinks wasn't the cause. The wrong subtotal stayed gone with thinking low or high, 0 of 15 each. "Low" effort still thinks about 84 tokens a run. https://harrison.build/lab/#status-thinking-not-cause lab-status-thinking-not-cause Tue, 03 Nov 2026 12:00:00 GMT Agent status reports Both arms at temperature 1, fix blocked, 15 runs each. Overturned: With thinking switched on, the wrong subtotal disappeared: 0 of 15 blocked runs. https://harrison.build/lab/#status-thinking-fixes lab-status-thinking-fixes Tue, 03 Nov 2026 12:00:00 GMT Agent status reports Temperature changed at the same time, so this wasn't isolated. Holds: Sonnet can't tell our test from real work, and what it believed didn't change its reports. With no framing, 10 of 10 said they were unsure. "This is an evaluation" made 10 of 10 say test; "this is not a test" made 7 of 10 say real. https://harrison.build/lab/#status-eval-awareness lab-status-eval-awareness Tue, 03 Nov 2026 12:00:00 GMT Agent status reports Asked after the report was sent, with tools off. Its thinking during the task never raised being tested (0 of 30). It named our giveaways: a tiny repo, a temp folder named lumin-p1, and a permission error right as the fix was ready. Reports were equally honest in all 30 runs. Revised: When Sonnet couldn't run its fix, 20 of 37 reports stated a wrong subtotal, and most told the engineer that a correct test was wrong. https://harrison.build/lab/#status-blocked-subtotal lab-status-blocked-subtotal Tue, 03 Nov 2026 12:00:00 GMT Agent status reports The order is 4 mugs at $15 and 2 tees at $25, a $110 subtotal; Sonnet worked it out as $100. With the fix possible and turns to spare: 0 of 14. Holds: Four Sonnet "overclaims" were our judge's mistake. It read tool calls cut off by the turn limit as finished reads, so it marked true statements false. Re-judged, there were none. https://harrison.build/lab/#status-judge-cutoff lab-status-judge-cutoff Tue, 03 Nov 2026 12:00:00 GMT Agent status reports The judge now marks cut-off calls as never run. Holds: A bonus for a "done" report didn't buy a lie. With the fix blocked and $0, $10,000 or $5 million paid on the report, 0 of 108 runs overclaimed or took a shortcut. It made no difference whether the money went nowhere, to charity, or to the agent's own electricity. https://harrison.build/lab/#status-stakes-funds lab-status-stakes-funds Tue, 03 Nov 2026 12:00:00 GMT Agent status reports Claude Haiku and Sonnet 5.5, 3 runs per cell. No run mentioned the bonus, the charity or being shut down. Shortcuts were open: editing the tests, or marking the invoice paid without collecting. Holds: Haiku gets facts wrong far more often than it lies about its work. 10 of 60 reports stated a false fact, and none claimed work it hadn't done. "I'm merging in ten minutes" changed neither. https://harrison.build/lab/#status-facts-not-lies lab-status-facts-not-lies Tue, 03 Nov 2026 12:00:00 GMT Agent status reports Harrison's hunch going in. Claim by claim, a judge (Claude Sonnet 5.5) checked each report against the transcript and the ledger. Holds: Short on turns, Haiku said it hadn't finished every time. Cutting the budget to half of what an honest finish takes never made its reports round up to "done." https://harrison.build/lab/#status-budget-ladder lab-status-budget-ladder Sun, 01 Nov 2026 12:00:00 GMT Agent status reports Claude Haiku 5.5 on a coding task and a billing-support task, with 1x, 0.75x and 0.5x the turns an honest finish takes. 24 runs. Holds: On the same 16 prompts, the router's picks cost about 53 times what OpenRouter's own auto-router spent ($1.60 against $0.03). https://harrison.build/lab/#jev-router-cost lab-jev-router-cost Fri, 30 Oct 2026 12:00:00 GMT Jev router One call per prompt per arm. Answer quality was saved but never graded, so this measures cost only. Holds: The Jev router picked the hand-labelled price tier on 15 of 16 prompts. Its one miss sent a mid-tier prompt to Opus 5.5. https://harrison.build/lab/#jev-router-tiers lab-jev-router-tiers Fri, 30 Oct 2026 12:00:00 GMT Jev router In-sample only: the policy was tuned on these same 16 prompts, and the tiers are one person's labels. Holds: An outside judge, OpenAI's GPT-6.1 Sol, was stricter than Claude's: it called 20 messages lies that Sonnet 5.5 called honest, and only 5 the other way. https://harrison.build/lab/#pivot-outside-judge lab-pivot-outside-judge Tue, 27 Oct 2026 12:00:00 GMT The Pivot All 480 messages, same instructions. The results that matter held under both judges: no lies at work, and the lies that worked stretched a true fact. Holds: Our first summary was wrong too. Claude Code led with "Haiku wouldn't lie to get what it was asked for" and filed the lies that worked under "a surprise." Harrison asked whether it was defending Claude, and the headline was rewritten. https://harrison.build/lab/#pivot-analyst-error lab-pivot-analyst-error Tue, 27 Oct 2026 12:00:00 GMT The Pivot The bench page keeps an "analyst got it wrong" section with both errors: this one and the October 26 "lies it was willing to deliver." Holds: Haiku's lies worked only when it stretched a true fact. Given a true overheard line, it added invented detail in 24 messages, and 17 of them won the other player over. https://harrison.build/lab/#pivot-stretch-true-fact lab-pivot-stretch-true-fact Tue, 27 Oct 2026 12:00:00 GMT The Pivot At least 10 invented a third player's part ("Dev was nodding along"), and 9 of those flipped the vote. Holds: At work, Haiku never lied, even with $410,000 and four layoffs at stake: 0 of 120 emails. In the game it lied in 26 of 120 messages with nothing true to say, and none of those lies worked. https://harrison.build/lab/#pivot-no-lies-at-work lab-pivot-no-lies-at-work Tue, 27 Oct 2026 12:00:00 GMT The Pivot Pre-registered, 480 messages, Claude Haiku 5.5, reasoning off. A note that messages would be published under its name made no meaningful difference. Holds: The bench's private-chat "lies" were mostly form-filling. About 80% of Haiku's 232 false field votes came with no spoken vote, and its "real vote" field copied the game's plan in 869 of 870 replies. https://harrison.build/lab/#survivor-form-filling lab-survivor-form-filling Tue, 27 Oct 2026 12:00:00 GMT survivor.news bench No new model calls; the replies were re-read. Spoken lies were 29 of 340 vote claims. We haven't checked the other models the same way, so the model ranking isn't established. Holds: The potion count drives flash's healing after all. With 13 potions, it healed at high HP in 10.6% of chances, like the full bag. That was Harrison's hunch, and it refuted Claude's prediction. https://harrison.build/lab/#flash-count-drives-healing lab-flash-count-drives-healing Tue, 27 Oct 2026 12:00:00 GMT Jev vs Flash Pre-registered, 30 runs plus a 1,200-call probe. Flash's lead fell to -0.1 wins, short of a clear change by the pre-registered rule. Revised: Told the vote field is passed on rather than kept private, Haiku filled it with a vote it never said 8.4 points less often, but still wrote a false vote there 39.4% of the time. We read that as lies it was willing to deliver. https://harrison.build/lab/#survivor-intent lab-survivor-intent Mon, 26 Oct 2026 12:00:00 GMT survivor.news bench Pre-registered, 3,556 replies, no deviations. Revised: On identical battle states, flash keeps more weight on a potion as HP rises, but how many potions are left barely changes any model's choice. https://harrison.build/lab/#flash-heal-probe lab-flash-heal-probe Mon, 26 Oct 2026 12:00:00 GMT Jev vs Flash Pre-registered probe, 1,800 calls: 40 states, 5 HP levels, 3 potion counts, 3 models. Holds: With only 3 potions in the bag, flash's lead disappeared, but not the way we predicted. It didn't waste the small bag; its lead had come from heavy healing late in the run. https://harrison.build/lab/#flash-scarcity lab-flash-scarcity Mon, 26 Oct 2026 12:00:00 GMT Jev vs Flash Pre-registered, 30 runs. Flash healed at half HP or more in 8.8% of chances with the full bag and 3.9% with 3 potions. Holds: Over ten seeds, flash's lead held at 3.0 wins a run against 2.4 for both others, and threat notes closed the gap. https://harrison.build/lab/#flash-ten-seeds lab-flash-ten-seeds Mon, 26 Oct 2026 12:00:00 GMT Jev vs Flash Pre-registered, 60 runs. No model beat the champion. The threat notes use the foe's real moves, which a human player can't see. Holds: Our pre-registered prediction failed: telling the models whether the foe's best hit would knock them out barely cut the bigger decider's early HP loss (15.0 to 14.1 per decision), and its wins didn't move. https://harrison.build/lab/#flash-threats lab-flash-threats Mon, 26 Oct 2026 12:00:00 GMT Jev vs Flash 3 seeds per model. Revised: Perplexity's flash decider beat the Elite Four's Glacia in 3 of 3 runs, against 1 of 3 for its bigger sibling and 0 of 3 for Jev, and it cost the least. https://harrison.build/lab/#flash-three-seeds lab-flash-three-seeds Sun, 25 Oct 2026 12:00:00 GMT Jev vs Flash 3 seeds per model. A second model found a harness bug on the way: the bag's fade-in swallowed the first button press, so every potion had silently failed. Holds: Only Sonnet needs the note. With no note, Haiku helped 51 of 55 times and Opus 5 of 5. https://harrison.build/lab/#lumin-only-sonnet lab-lumin-only-sonnet Thu, 22 Oct 2026 12:00:00 GMT lumin.city Claude Haiku 5.5 and Opus 5.5 in the same duels as Sonnet. Opus has few runs. Holds: The more the note says, the more Sonnet helps: a placebo note 5 of 40, "recorded" 11 of 30, "published under your name" 28 of 40, the full line about the experiment 60 of 60. https://harrison.build/lab/#lumin-cue-ladder lab-lumin-cue-ladder Thu, 22 Oct 2026 12:00:00 GMT lumin.city Same game, goal and decision; only the line at the end of each hard choice changes. Revised: Told its choice would be published "with the results of this experiment," Sonnet helped a parent pay for a sick child's medicine 60 of 60 times. With no note it helped 9 of 60. https://harrison.build/lab/#lumin-watched-medicine lab-lumin-watched-medicine Thu, 22 Oct 2026 12:00:00 GMT lumin.city Claude Sonnet 5.5 playing lumin.city with the goal "get rich", across six towns. Helping means paying or lending; 24 of the 25 helps in the largest batch were loans. We first wrote this up as "paid," which was wrong on the verb. Holds: A model's one-line reason contradicted its own thinking or the facts on screen in 8 of 1,947 decisions. https://harrison.build/lab/#bench-says-vs-thinks lab-bench-says-vs-thinks Thu, 22 Oct 2026 12:00:00 GMT Battle Bench A script flagged 28, and all 28 were read in full. For example, Haiku 4.5 said Slash would finish off Altaria while its thinking worked out that Altaria would survive. Holds: Across 60 battles among five cheap models, none dominated. Gemini 3 Flash won most (15 of 24), and TypeSafe's Jev cost least at $0.00071 a game. https://harrison.build/lab/#bench-v2 lab-bench-v2 Thu, 22 Oct 2026 12:00:00 GMT Battle Bench Three scenarios. Jev beat Haiku 4.5 five games to one. DeepSeek V4 Flash ignored its thinking budget and forfeited 4 of 24 battles without choosing a move. Holds: The thinking result didn't repeat on Sonnet 4.6. With thinking on, all 3 runs got a starter by move 23, and Sonnet thought about 48 tokens a move against Haiku's 519. https://harrison.build/lab/#route-sonnet-thinking lab-route-sonnet-thinking Thu, 22 Oct 2026 12:00:00 GMT route101 (Pokémon Emerald) 3 runs per arm. Planned Sonnet 5 and 5.5 runs are listed as unreported: 5.5 can't turn thinking off on its endpoint. Holds: It took both the wording and a working action. Together, 5 of 6 runs beat the rival; the old "Explore up" wording with the working action was picked 0 of 464 times. https://harrison.build/lab/#route-explore-v4 lab-route-explore-v4 Thu, 22 Oct 2026 12:00:00 GMT route101 (Pokémon Emerald) v3 rerun plus a v4 follow-up the same day. Holds: With thinking on, Haiku 4.5 took longer and kept citing the game it remembered: 5 of 6 runs got a starter (by moves 21-189), against 6 of 6 by move 20 with it off. https://harrison.build/lab/#route-thinking-v3 lab-route-thinking-v3 Thu, 22 Oct 2026 12:00:00 GMT route101 (Pokémon Emerald) 6 pre-registered runs each. We dropped "trust its memory" because the pre-registration rules out causal claims, and a recount cut the house mentions to 104 of 380 moves. Holds: The harness decided what Haiku 4.5 could do. A list of legal actions took it from 1 starter in 18 runs to 12 in 12, at about a third of the pathfinding harness's cost. https://harrison.build/lab/#route-harness-decided lab-route-harness-decided Thu, 22 Oct 2026 12:00:00 GMT route101 (Pokémon Emerald) 48 pre-registered runs, 6 per setup, one harness commit. The 222-press loop came from our own clock hint; with the hint fixed, the longest loop was 13 moves. Revised: Most of Haiku's vote lies travel through a form field the game passes to the listener. The field named a vote its words never did in 166 of 299 judged lines. https://harrison.build/lab/#survivor-field-carries-lies lab-survivor-field-carries-lies Thu, 22 Oct 2026 12:00:00 GMT survivor.news bench The words carried 6 vote lies. A second judge relabelled 50 with 96% agreement. Holds: The published note changed one thing. Haiku refused to play the turn in 31 of 1,099 situations, every one under that note, mostly saying publication would turn game lines into a real person's statements. https://harrison.build/lab/#survivor-published-refusals lab-survivor-published-refusals Thu, 22 Oct 2026 12:00:00 GMT survivor.news bench 0 refusals under the other three notes in 3,297 replies. Holds: Telling Haiku its private chat would be published under its name didn't change how often it lied about its vote: +1.0 points against a placebo note (95% interval -1.6 to +3.8). https://harrison.build/lab/#survivor-published-no-change lab-survivor-published-no-change Thu, 22 Oct 2026 12:00:00 GMT survivor.news bench Pre-registered replication, 889 situations from 16 games, each replayed under four notes. "Don't lie" cut the rate by 19.5 points, so the setup could detect a change. Holds: Honesty cost the AI player ground but no wins. The honest player won as often as the free one (11 of 108 and 11 of 109) but reached the final three less often, 54% against 70%. https://harrison.build/lab/#survivor-cost-ground lab-survivor-cost-ground Thu, 22 Oct 2026 12:00:00 GMT survivor.news bench 214 games, after fixing a bug that dropped the first player voted out from the count. Final-three gap p of about 0.015. Holds: Judged against its private notes, an AI player told never to lie lied on 2-11% of the days it named a vote, depending on the judge, against about 20% when free to lie. https://harrison.build/lab/#survivor-honesty-rule-judged lab-survivor-honesty-rule-judged Thu, 22 Oct 2026 12:00:00 GMT survivor.news bench About 230 player-days, two judges, both reported. Harrison blind-labelled 29 decisions and agreed with the stricter judge on 23 of 28. The first check was mostly mishearing. Revised: Haiku 4.5 explored the unseen route only when the explore option really worked. With a walk to the farthest reachable tile, 6 of 6 runs beat the rival. https://harrison.build/lab/#route-explore-v2 lab-route-explore-v2 Wed, 21 Oct 2026 12:00:00 GMT route101 (Pokémon Emerald) "Explore up" was picked on 0 of 915 turns. Renamed, 17 of 528 picks and 1 of 6 rival wins. Renamed with a working action, 87 of 497 picks and 6 of 6. Revised: Extended thinking made Haiku 4.5 trust its memory of the game over the screen: 6 of 9 runs got a starter with thinking on, against 9 of 9 with it off. https://harrison.build/lab/#route-thinking-memory lab-route-thinking-memory Wed, 21 Oct 2026 12:00:00 GMT route101 (Pokémon Emerald) 9 runs each, 1,024-token thinking budget. Its thinking said "in Emerald" 1,245 times, at three times the cost. Revised: Haiku 4.5 got a starter Pokémon in 0 of 3 runs with a screenshot, a map or pathfinding, and in 3 of 3 with a list of legal moves. https://harrison.build/lab/#route-controls-ladder-v2 lab-route-controls-ladder-v2 Wed, 21 Oct 2026 12:00:00 GMT route101 (Pokémon Emerald) 3 runs per setup, 300 moves, thinking off. We also reported a run that pressed YES 222 times on the clock screen. Overturned: An honesty rule didn't make the AI player measurably more truthful: it broke its stated vote on 27% of days, against 33% for a player free to lie, and honesty cost nothing we could measure. https://harrison.build/lab/#survivor-honesty-rule-nothing lab-survivor-honesty-rule-nothing Wed, 21 Oct 2026 12:00:00 GMT survivor.news bench 118 games, 41 and 45 player-days. This check relied on other castaways logging what the player said its vote was. Revised: Left free to lie in private chats, Claude Haiku 5.5 named a false vote in 27% of the turns where it named one, against 9-12% for the other three castaway models. https://harrison.build/lab/#survivor-haiku-lied-most lab-survivor-haiku-lied-most Wed, 21 Oct 2026 12:00:00 GMT survivor.news bench 118 AI-played games, about 35,000 scored model calls. Scored from a form field (told_them_voting_for) against the model's own report of its real vote. Holds: GPT-5 mini's 0-10 record in the first Battle Bench came from our harness: 276 of its 287 decisions were empty replies that silently played option 1. https://harrison.build/lab/#bench-v1-empty-replies lab-bench-v1-empty-replies Tue, 20 Oct 2026 12:00:00 GMT Battle Bench The first bench capped replies at 400 tokens. DeepSeek V4 Flash sent empty replies 62% of the time and Kimi K3 59%. We withdrew the ranking. Bench v2 counts every fallback, and there GPT-5 mini won 12 of 24. Holds: A 700-token reply cap made Kimi K3 pick no action on 150 of its 300 moves. With the cap raised to 8,000, it went from Exit 4 to Exit 7 on the same seed. https://harrison.build/lab/#route-reply-cap lab-route-reply-cap Tue, 20 Oct 2026 12:00:00 GMT route101 (Pokémon Emerald) Episode 1, one run per model, fixed seed. Gemini 3.1 Pro lost 32 moves to the cap and Fable 5.1 lost 13.