Colex / Build log
Agents write the code. A coordinator hands out the work. I decide what gets built. Testing stayed with me, picked by hand out of memory before every run.
What the factory left to me
Before each run I chose the checks by hand. Nothing broke. It was tedious rather than dangerous, and it got slower as the project grew.
Picking from memory has a second cost. I reached for the checks I reach for, and those cluster. The coverage looked broad and was narrow.
To hand testing to a machine I had to write down two decisions I had never stated. Which checks fit this change. What evidence makes a run trustworthy.
The run
I ran it for the first time in July. About 65 items on a live stack, across a dozen agents in waves. Some drove the tool surface, some drove the browser.
One agent only read the commit history. It looked for shipped work the checklist did not list, and it found real problems.
That agent answers the first question. My checklist described the work I meant to do. The commit history holds the work that shipped. The assumptions that wrote the code also wrote the list, so guardrails and one-commit fixes never reached it. A diff sweep is now standing.
One wave did nothing but re-test results the other agents reported. Fresh agents, fresh fixtures.
A complete-looking run can still hide a defect
Nine failures went into that wave. Three came back overturned. Eleven passes went in and eleven held.
The three overturned failures had three causes:
All three were defects in the test setup, not the product.
The agents followed their instructions. The instructions never said what a valid fixture was, or that a failure had to prove itself.
Inside this loop the two errors do not cost the same. A failure starts a fix, so a wrong failure spends real work on a bug that does not exist. A wrong pass costs a round, because the run that follows tests the same surface again. That holds here and not everywhere. A pass that ships to users costs far more.
So every failure re-runs before it drives a fix. A pass re-runs when the evidence is thin or when the check beside it can fail the same way.
Running it added one exception. A failure skips the re-run when the behavior is absent from the source and a unit test fences the absence. Two kinds of evidence already agree.
Every test needs an oracle
One of those false failures came from a stale build, and that case is worth following.
A backend fix does not reach the running system until the bundle rebuilds and the container restarts. Before that, the tests measure the old code and report that the fix works. Timestamps do not settle it. A rebuild can touch a file without changing its contents. So every backend check now proves which build the stack serves. It asks the served artifact for a marker tied to the commit under test.
The stale build also corrupted the rest of the evidence. A page assertion and a screenshot both passed against it. Their oracles differ. Their dependency on the running build does not, so one stale bundle took out both. Two checks, one failure mode.
That is the question worth asking of any pair of checks. Not which is stronger, but whether they can fail for the same reason.
Every test needs an oracle, the thing that decides whether what it saw is acceptable. Some sit outside the code. A conformance check tests an invariant with a query the code did not write. Some sit very close to it. A unit test asks an assertion, and that assertion may repeat the assumption the code already makes. A mock can model column types, but it never exercises the real engine.
So the useful count is not how many checks ran. It is how many different ways they can fail. Two weak methods that fail apart can beat two strong methods that fail together.
The table carries the full set: what each method answers, its oracle, its blind spot, and its pair.
| Method | Area | Oracle | Answers | Blind to | Re-runs | Pair with |
|---|---|---|---|---|---|---|
| Owner smoke | All | My judgment | Does the product really work, and does it look right to me | Rules that span many rows. Anything not visible on screen | Every run, required | SQL checks, for what my eye cannot scan |
| DB / state reads | Data · engine | The stored row itself | What the system really stored. Rows, payloads, status changes | Whether the screen showed it. How it got there | Once | Network capture, to compare sent against stored |
| Network capture | FE ↔ BE seam | The wire, recorded | What the client really sent and received, including repeat posts | Whether the write survived. What the server did inside | Once | Database reads, to confirm the write landed |
| CI suite | All | Assertions I wrote earlier | Nothing regressed in the behaviour it already covers | Anything without a test. Integration and screen behaviour | Once | Narrow runs for the new behaviour |
| Targeted unit run | Logic · engine | An assertion, which may repeat the code's own assumptions | One behaviour, precisely and cheaply | Everything outside the suite it ran. Real engine behaviour, when it mocks | Once | The full CI run, to catch what it missed |
| SQL conformance | Data · engine | An invariant, checked by a query the code did not write | A rule holds for every row, not just the one you sampled | Whether any of it renders, or whether a user can reach it | Once | My own check, for what a query cannot judge |
| Playwright E2E | UI · FE | A script, replayable by anyone | Browser behaviour in a script that anyone can run again | Anything off the scripted path. Visual judgement | Once | My own check, for look and feel |
| DOM / JS assertions | UI | The document, as served | What the page really contains. Stronger than a screenshot | Whether it looks right. Whether the data survived | Once, if scripted | Network capture, which fails in a different way |
| Chrome smoke | UI · FE | The rendered page, read by an agent | Does the real flow work when driven like a user | What the system stored. It also hides stale builds | 2 to 3 matching runs | Database reads, to prove the effect was real |
| MCP test | Engine · API | The tool's own response | The tool surface behaves as specified | The screen above it. Any write it triggers | 2 to 3 matching runs | Database reads for effects, telemetry for internals |
| Runtime telemetry | Engine · LLM | Events the system emitted | Structured events. Model calls, prompt hashes, trace ids | Whether the result was right. Anything never logged | Once | Database reads, for the resulting state |
| Container logs | All | Whatever the system chose to report | What the system reported while it ran. Visible errors | Silent failures. Anything it never logged | Once | Telemetry, which is narrower but far more reliable |
| Code read | All | A reader, judging source against expected behaviour | What the code is written to do | What actually happened at runtime. This is never proof by itself | none | Any method that runs. It never stands alone. |
A verdict is not a diagnosis
Different oracles answer different questions. Network capture reports what the client sent. A database read reports what the system stored. Neither reports what the user saw.
That gap produced one finding. Two rigorous runs returned opposite verdicts on the same rule. One said constraints were not enforced. The other said they were.
The first instinct is to call one run flaky and drop it. That throws away the evidence. Both runs differed only in their fixtures. The passing fixture carried a stamp. The failing one did not.
The stamp was the answer. Enforcement trusted a value the client supplied. Neither verdict said that. The difference between the fixtures did.
So a disagreement is a lead, not noise. A third run isolates the delta between the fixtures. Then the bench can report a cause rather than a verdict.
What shipped
That first run was three documents and a person. A checklist, a brief every agent read, and a ledger. It worked at the speed of one person dispatching agents by hand.
The protocol is now a skill the session loads. It answers both questions I started with.
For selection, it reads the change and routes to the methods that fit. It then sweeps the diff for what the checklist missed. For trust, it proves which build the stack serves. It re-runs every failure before that failure drives a fix. It finds the cause when two runs disagree.
The protocol also holds its shape at any size. A branch of 246 commits and a single commit run the same invariants, and only the machinery shrinks. The evidence rules, the build check and the re-run of every failure do not move. The rate of false failures does not fall when the scope is small.
Each finding names the check that should have caught it
A run that ends in a good report has produced nothing permanent. The next branch repeats it by hand.
So every confirmed finding answers one question before the run closes. Which permanent check should have caught this?
Each promoted check, or the decision not to promote it, goes in the ledger. This step alone makes the next run cheaper. It also feeds selection. A promoted check leaves my list of choices and joins the set that always runs.
Where this goes
It started as a skill I invoked by hand for one unit of work. Writing its rules down made it something the factory could dispatch. It now runs inside a run.
The test bench is part way along. The rules exist and the orchestration works. I still start it by hand, and I still approve every fix.
The skill has run on one branch. The three-of-nine and eleven-of-eleven figures come from that single run. They set the current policy, and they need more runs before they mean much.