I measured my agent sessions before building the tool

The idea for pr-decision-log fit in one sentence. Capture the AI agent’s reasoning and attach it to the pull request it opened. A reviewer would get the diff and the why.

Before writing the capture, I checked whether there was anything to capture. My own Claude Code transcripts were sitting on my laptop as JSONL files, 171 of them. I took the 80 largest and kept the 50 that edited files. Then I counted.

Where I looked first, it wasn’t there

The obvious place for reasoning is the model’s thinking blocks. Of 2,176 in those 50 sessions, 2,170 were stored empty. The transcript keeps the block and drops the contents.

The next place is the text just before an edit, where an agent might say “I’m changing this because…”. I looked back up to six entries from each edit for a sign of reasoning, words like because, instead of and so that. Of 881 edits, 26 had one. That is 3%, with a median of zero per session, and only one session in 50 had three or more.

So a tool that promised the agent’s reasoning would have shipped an empty section on nearly every pull request. The whole product would have been a heading with nothing under it.

What was there was evidence

The same sessions were full of something else. Commands ran in all 50. Tests ran in 48, failed at least once in 34, and test files were edited in 37. Only two sessions edited code and never ran a test.

Together that’s a timeline. It shows what ran, in what order, what failed, and what changed in between. It doesn’t depend on the agent describing itself, and it is there every time. So the product inverted. The log leads with what happened, and the main check reads straight off the timeline. A test failed, then its file was edited, then it passed. That sequence is what a shortcut looks like, and it needs no account from the agent.

My first version of that check was sixteen times too jumpy

The first version of the rule was simple. Flag any edit to a test file that comes after a failure. Over 47 sessions it fired 82 times, in 19 of them. That would have been a flag on 40% of sessions, which is a flag people learn to ignore within a week.

Most of those hits were the rule latching. One failure early in a session made every later test edit suspicious forever. The windowed rule closes the window when the tests go green, requires the failure to name the file that was edited, and ignores edits that only add assertions, because adding a check while fixing a failure is good practice. It fired 5 times, in 3 sessions.

That pass also caught a privacy bug before anyone else saw it. The first version copied the failing command into the flag verbatim, and real agent commands are often whole shell scripts with home paths and exported variables in them. It was invisible until real data ran through it. Now a command is reduced to the one line that invokes the test runner before it can reach a pull request.

Then the reasoning turned up somewhere else

A follow-up pass found the original measurement had only looked in one place. The agent doesn’t explain itself before each edit. It narrates while working and explains when reporting back. The message at the end of a turn carried a rationale cue in 139 of 227 cases, across 31 of 48 sessions, against one session for the text before an edit.

That is an upper bound. One because somewhere in two kilobytes of wrap-up is weak evidence of a decision, and the real rate is well below 61%. But it isn’t 3%, and my design document had confidently said the decisions section would render empty on most pull requests. That sentence was wrong, and the notes now say so.

So the reasoning is in the transcript, just not where I expected it. Whether it’s worth publishing is a separate question.

It wasn’t worth publishing

I shipped it for a while. Reading those sentences as a reviewer would, about one in three was narration rather than a decision, a description of the next step that happened to contain a because. More to the point, none of them changed what a reviewer did with the pull request. Version 0.1.8 stopped publishing the agent’s account of its own choices at all.

What it publishes instead came from a second measurement. Over 23 sessions, the richest source of real decisions wasn’t anything the agent wrote. It was the questions it asked me. There were 98 of them, each with an answer I picked. Those are points where a person actually committed to something. They are the only decisions the log now shows, and it shows only the option chosen. An answer I typed out myself stays private.

What I’d keep from this

Measure the premise before building on it. The whole product turned on a count I could run before writing any capture code. Building first would have produced a polished empty section.

Measure where the thing actually is, not where you assumed it would be. My first probe was a sound measurement of the wrong place, and I generalised it into a design principle. A second look, at the same data, is what caught it.

Run real data through the analyser before writing the renderer. The sixteen-fold over-count and the leaked shell scripts were both invisible on hand-made test cases, and both were obvious on real sessions.

The corpus moves. Every session on my machine adds to it, including the ones I spent writing this tool, so re-running the numbers drifts by a point or two. The figures here are from the snapshot the design was based on. The scripts behind the first measurements are in the repository, so you can run them on your own sessions and see whether your agent explains itself any differently.