pr-decision-log

The problem

I use Claude Code every day, and at work every pull request is reviewed by a person. The same thing keeps happening. The agent does an hour of work, I open a pull request, and the reviewer gets a diff and a two line description. All the things that would make the review quick, like what ran, what failed and what the agent changed after it failed, sit in a log file on my laptop that nobody will ever read.

What I assumed, and what measuring did to it

The original idea was to capture the agent’s reasoning. Before building that, I checked whether there was any reasoning to capture. I took the 80 largest of my own Claude Code sessions and kept the 50 that edited files.

Where I looked first, there wasn’t. Of 2,176 thinking blocks, 2,170 were stored empty. The agent explained why before an edit 26 times in 881 edits. That’s 3%, and most sessions had none at all.

What was there was evidence. Commands ran in all 50 sessions and tests ran in 48. So I flipped the product around. It now leads with what ran and what happened.

A second look found where the agent’s explanations had gone. They were in the wrap up message at the end of each turn, which showed signs of reasoning in 31 of 48 sessions. The text was real, but about one in three of those sentences just described the next step rather than a decision, and reviewers didn’t act on any of them. So in version 0.1.8 I stopped publishing the agent’s account of itself altogether. The decisions the log still shows come from a different place. A separate measurement over 23 sessions found 98 questions that I had answered, each one a moment where a person actually made a choice. Those are the only decisions the log shows.

What that led to

The main check became TEST_EDITED_AFTER_FAILURE. A test failed, its file was edited, and then it passed or never passed again. That pattern is what a shortcut looks like, and it comes from the timeline, not from anything the agent says about itself.

Later checks read the same timeline. One notices code that changed after the last test run, which happened in 3 of 26 of my pull requests. Another notices a changed Dockerfile or migration whose check never ran. A third names commits made outside any recorded session, so the reviewer knows what the log doesn’t cover. Files written from the shell count as edits too. 73% of writes went through the shell, so a sed -i that quietly weakens a test gets flagged like any other edit.

Privacy had to be built in from the start rather than scrubbed at the end. The log only lets through a short list of things. That means the name of each test, build or lint run, whether it passed, file paths inside the repo, the names of failing tests and how many assertions changed. Raw output, full file paths and file contents never reach the page. One test hides a fake AWS key in a command’s output and checks that it can’t end up in the log.

The log is added by editing the pull request after it’s opened, rather than by rewriting the command that creates it. If two tools both rewrote that command, the order they ran in would be random, and anyone using another such tool would lose their log now and then for no visible reason. That’s the kind of bug nobody can report.

What it cost

A smaller promise. “Here’s what the agent did” is less exciting than “here’s what it was thinking,” but only the first one is true.

Fewer warnings than I’d hoped for. I ran the diff rules over all 320 commits in my own repositories, and they raised 45 warnings about removed code. None of them was a shortcut. A diff can see what disappeared, but never why. So now a removal only raises a warning when the timeline also saw that file edited after a failure, and the same commits then produced no warnings at all. The downside is that the only proof it still catches real shortcuts comes from planted test cases and one real agent run.

Trust in my own green checks. One real agent run showed that npm test | tail reports success however the tests went. That had quietly switched off the main check for one of the most common ways agents run tests. The tool now reads the test runner’s own output in that case. Separately, doctor once guessed that everything was set up and showed green while every session was recording nothing. It now reads the setting that decides this directly.

Where it is now

It installs as Claude Code hooks and publishes on its own, with no step to remember. Both ways of publishing, editing the pull request’s description or adding a comment, were tested against the real GitHub API on its own pull requests. It has 607 tests, and CI runs them on Node 22 and 24. It’s published on npm and also installs as a Claude Code plugin. Either way, pdl init --user and pdl doctor finish the setup.