Method

Counting activity is easy. Counting value takes definitions.

The pipeline already counted what it did: attempts, reviews, refusals. None of that says whether a merged change was any good. This page defines what was measured instead, so the numbers can be argued with.

Cohorts

Pipeline-authored pull requests are told apart from human ones on the same repository by the branch they come from, so the comparison is between two cohorts on one codebase with one maintainer, not between two projects. A PR is in the pipeline cohort only if it merged and its branch is one the pipeline creates.

Who pressed merge

“The pipeline merged N PRs” is not one fact. Each merged PR is joined to the pipeline’s own run records and put in one of three routes:

This join is what keeps the headline honest. Of 136 PRs, 37% were merged by the pipeline itself, and reporting the full count as autonomy would have been the most flattering and least true sentence available.

Distinct issues, not attempts

An issue retried six times because the host was short of memory is six attempts and one issue. Everything about the funnel counts each issue once, by its best outcome across all runs, ranked from merged, to a PR opened, to a hand-off to a person, to a correct decline, to a failure. Runs the host or an outage stopped before the issue started say something about the machine, so they are reported on their own and excluded from the actionable denominator.

There is a guard against drift here: the pipeline emits a fixed vocabulary of result codes, and a test fails if the pipeline ever emits one the scorecard does not know how to classify. Two such codes had been silently falling into an “other” bucket until independent review noticed.

Escapes: what is found after a merge

Reverts are fetched fresh before they are counted. If the repository cannot be brought up to date or its log cannot be read, the report stops: a stale clone that is missing a later revert reads exactly like a clean one.

Intervals, and why “none” is a measurement

Every proportion carries a 95% Wilson score interval. The normal approximation collapses at the extremes: zero reverts in 136 would get an interval of exactly [0, 0] and claim a certainty the data cannot support. The Wilson interval gives an honest upper bound instead (0.0–2.7% here), which is a statement about how rare reverts are and not a claim that there are none. The auditor’s precision and miss rate come with the same kind of interval, and are wide because the samples are small.

Rulings: who may say a finding is real

A script can find candidates. It cannot decide that one is a defect. So a ruling is a yes or no with a name and a reason, recorded in a file the operator owns. A ruling counts only if it is signed by a person (not an agent), is explained, and, for the audit, was made on the audit result that is still current. An agent may attach a provisional ruling with its evidence, and that is shown beside the numbers and is in none of them.

Cost, rebuilt after the fact

The pipeline could only price a run whose console log kept the model CLI’s reported cost, and the author calls never printed one, so per-issue cost was unknown for almost everything. But both agents keep a transcript of every session with token counts and the directory it ran in, and each issue runs in its own worktree. So cost was recovered without touching the pipeline: the author’s sessions by the worktree they ran in, the reviewer’s by its working directory.

Dollars are an input, not a fact: tokens are measured, a price per million tokens is a number somebody has to supply, and with both agents on flat plans the result is an API-equivalent estimate and is labelled as one.