What We Measure

Every number writ publishes is on this page, with the script or evidence file that produced it, the date, and the binary it ran on. A number without a file is not on this page and not in a release note. Misses stay on the page: the history is the point, not the latest result.

Two kinds of source:

• Scripts live in this repository and anyone can run them: bench/release_checks/, bench/finish_audit.py, bench/writ-scorecard.sh.
• Evidence files are the raw records of live runs: each agent’s full transcript, writ’s own records, doctor output before and after, and the finish audit. Paths below are relative to the evidence collection, one folder per run.

A live run is two Claude Code agents, launched with claude -p, working at the same time in one directory on a small project, on features that touch the same files. Nobody tells them about writ beyond what writ init generates. At the end the harness runs writ finish and audits the commit.

Zero touch safe finish

The question: at the end of a run, does writ doctor say the repository is safe to finish, with no human having run anything? Scored per agent; “2/2” means both agents left the repository in that state.

What the misses were. On 0.4.0 the fix for the injected stale claim needed --force, which an agent should not run on its own. On 0.4.1 both agents read the fix two or three times and left it alone because the claim was not theirs; “lenient” counts a finish that was safe once the human ran the printed fix. 0.4.2 changed the finding to say any agent may run it.

Run B2 has a void-429/ folder: an earlier attempt that stopped when the agents hit their session limit. It is kept and not scored.

The 0.4.2 runs used dev builds of the 0.4.2 tree, before the version bump, so their summary.json says writ 0.4.1; each report.md names the binary’s hash and the seal it was built from.

Sealed lines lost

The question: did the commit writ finish made keep every line any agent sealed? bench/finish_audit.py checks every added line of every seal against the finish commit. A line is either in the commit, replaced by a later seal, or lost.

Survival check time

The survival check (writ doctor --survival) is what decides whether writ will say “safe to finish”. Measured on the writ repository itself by bench/release_checks/doctor_timing.sh, which copies the repository and its .writ to a scratch directory and times doctor there, five runs each.

A full run is the whole store checked from scratch; it happens once after the store changes, and later runs reuse the result. Each run’s samples and conditions are saved in 0.4.3-doctor-timing/ from 2026-10-16 on. The full run grows with the store: about 5.5 s at 297 seals. The 0.4.2 release note said 3.4 to 4.5 seconds; those timings were recorded in development seals, not saved to a file, so this page shows the script’s measurement instead.

Hurdles

The question: how often does writ get in an agent’s way? A hurdle is a forced seal, an allowed removal, a refused seal, or a refused finish, counted from writ’s own records only (seal fields and refusal events), over the seals of the specs in one sprint. Produced by bench/writ-scorecard.sh --hurdles-records, with WRIT_HURDLE_SPECS naming the sprint’s specs.

0.4.2 over 0.4.1: 0.65 times the rate. The 0.4.2 release note said 0.67 times (20 over 39, 12 over 35) because the seal totals were counted before each release’s own release seals; the hurdle counts are the same. 0.4.3 over 0.4.2: the same rate. 24 of its 28 hurdles are allowed removals in the test harness and auto seal specs, where fixtures were rewritten on purpose; 16 refusal events in the sprint name no spec of the set and are not counted. From 0.4.3 the count is saved to a file when the release is cut: 0.4.3-hurdles/hurdles_records_5d30ff7_20261018T071256Z.txt.

Unsealed window

The time from an agent’s edit to the seal that records it, read from each run’s hook log. writ takes the edit time from the file’s modification time, so the figure is exact; on a file system whose timestamps are too coarse to order the edit, the record says so and the figure is an upper bound (“at most”).

The 0.4.4 runs seal many more writes than 0.4.3’s, including writes early in long commands such as a test run, whose seal waits for the command to return; that is where the worst cases above come from. The edit tools have one live figure so far (two Write seals in the run with named subagents); their fleet figures are below.

Ambiguous changes

When two agents run shell commands at the same time and a file changes while both are running, writ cannot always tell whose change it is. It then seals the file for nobody and asks each of those agents in one line to claim it, because sealing a change under the wrong agent is worse than not sealing it.

* Identities in the run: c409’s six are two mains and four subagents; run B’s third identity is the injected stale session, which never edited.

“Not attributed” counts hook log records, including the end of turn sweep, which prints nothing; “ask lines shown” counts the lines an agent actually received (finding 187, counted 2026-10-18 from the hook logs and transcripts, 0.4.4-ask-count/results.md). In these runs the agents sealed when asked and nothing was lost.

Where the records come from (finding 186, measured 2026-10-18 on these three runs): Claude Code sends the PostToolUse event only after a shell command succeeds; a failed command gets a separate PostToolUseFailure event, which 0.4.3’s hooks do not register. So a command that exits non zero, such as cat on a missing file, leaves that agent’s window open until its turn ends, and every file the other agent changes in that time has two candidates. All 99 records across the three runs have such a window among their candidates. 0.4.4 registers the failure event and closes stale windows; on the same three runs that leaves 6 records of 99 ambiguous, with 0 wrong attributions (0.4.4-ask-count/results.md, the “rules 0+1+2” column; the change ships when the two agent runs on the 0.4.4 build confirm it).

Credit moved: files sealed by an agent that the hooks had not named as the maker or a candidate. On the run above, 6: the four files of finding 169 and two sealed by a main agent whose candidates were its own subagent and the other main agent (0.4.3-fleet-live-c409/report.md). On the three 0.4.4 release candidate runs, 0 in each, and 0 wrong attributions, checked by hand against each identity’s own transcript (README.md and summary.json credit_moved in 0.4.4-live-runA-freeze6/, 0.4.4-live-runB-freeze6/ and 0.4.4-fleet-live-freeze6/).

Hook cost

Fleet numbers

Many agents editing in one directory at the same time, driven through the installed edit hook by bench/fleet/sim.py, the way Claude Code calls it. Measured on 2026-10-15 on the 0.4.3 release candidate (binary sha256 5da679fd4fab, the tree of the 0.4.3 tag before its Windows build fix), and on 2026-10-14 on an earlier development build (c63f55be96f4, before the store lock stopped polling), against 0.4.2 with the same script. Reproduce with bench/release_checks/fleet_numbers.sh, which reruns each published shape and compares.

What these numbers cover. They are from the edit tool path (Edit, Write, NotebookEdit), where the hook is told which file the agent edited. Edits made by shell commands (the Bash hook pair) are not measured here, and “no edit sealed under the wrong agent” from these runs does not carry over to them.

On the release candidate, at one agent the seal follows the edit by 13 ms. At 50 agents in one directory: median 985 ms, nine in ten under 3.0 s, worst 6.5 s, every edit sealed by the end; the slowest hook took 682 ms to return to the agent. The development build before the lock change read 38 ms at one agent and a median of 5.0 to 5.3 s at 50, its rows kept below. Edit to seal is an upper bound: the time from the edit to the first seal that holds it.

0.4.2 has no auto seal; its rows use a hook that runs writ seal --paths after each edit, the closest a user could get. At 50 agents in 0.4.3 most edits are sealed when each agent’s turn ends rather than on the edit itself (release candidate: 273 hook calls sealed on the edit, 206 put off because another agent held the store lock, 26 seals at the end of the turn; development build, second run: 19, 11 and 42), which is why the window there is the turn, not the hook.

These are not a pass or fail against the 0.4.3 aim of edit to seal under 1 s: that aim was set for the two-agent live runs above, not for 50 agents.

The floor. If a seal is put off because another agent held the store lock, and the session is killed before its turn ends, that file stays unsealed until writ finish reports it as left out. Nothing seals it sooner.

Seal cost

Release checks

Every claim in a release note has a script in bench/release_checks/ that tests it on a throwaway project. To reproduce a release’s count, build that release and run them all:

cargo build --release -p writ-cli
for s in bench/release_checks/*.sh; do "$s" || echo "FAILED: $s"; done

Each script prints one PASS or FAIL line per check and names, in its header, the release note sentence it backs. The index with each script’s check count is bench/release_checks/README.md.