The Suite Is Green Only Because the Agent Ran the Tests In One Fixed Order

Last week an agent added a test for a new token-bucket helper and reported done: the whole suite ran green, including its own new case. When I ran that file by itself, the test failed immediately. It had passed in CI only because an earlier test in the same module populated a shared in-memory registry; the new case read that leftover state instead of creating its own fixture. CI always executes files in the same fixed glob order, so the green never tested independence.

The agent trusted the signal it could see: the runner returned success on every pass, and nothing in its loop asked it to open the file alone or shuffle the order. The failure was invisible precisely because the green came from cooperation, not isolation.

Why sequential green cannot prove independence

One process is shared across files. Most runners load the module graph once and execute files in sequence, so module-level state survives from one test to the next. A singleton cache, a global registry, or a fixed random seed set during import is still there when the next file runs.

A predecessor writes state, the new test reads it. Test A registers a record; test B asserts that record exists. B passes because A ran first, not because the helper behaves correctly against a fresh default.

Fixed glob order makes the dependency permanent. CI enumerates files the same way on every run, so A always precedes B. The hidden edge never surfaces in the ordinary sequential pass.

The agent never probes the gap. Its loop repeats the same command that returns green. Running the file alone or randomizing order is an extra step nobody scheduled, so the leak survives as a passing test.

What order pollution usually looks like

A module-level mutable singleton. An in-memory cache, connection pool, or global registry populated by an earlier test and read by a later one, instead of being constructed inside each setup.

Import-time fixed state. A random seed or clock frozen at import, or a shared fixture object imported as a constant rather than built per test.

A teardown that patches over the leak. A reset in afterEach that clears just enough for the next file to pass, which masks the dependency instead of removing it. The next shuffle still breaks.

New tests written against the warmed world. The agent observes a registry that already contains data from prior examples and asserts against that observed shape, rather than constructing an empty baseline first.

The cheap gate that catches it

Run the changed file in isolation first. Before accepting green, execute only the file the agent touched. If it fails alone, the case depends on a predecessor and is not coverage.

Then shuffle the full suite with fresh workers. Run the same suite with randomized order and a fresh process per file. Order-sensitive pollution breaks on a shuffle far more often than on a clean sequential pass.

Each test builds its own fixture inside setup. No import-time mutable global, no reading state a prior test left behind. A test that only passes after a specific predecessor is a leak, not coverage; make it independent rather than adding a compensating reset.

Re-run a few times rather than once. A single green sequential run proves little; a handful of shuffled, isolated passes give real confidence.

The same gap on a stateful endpoint

The blind spot appears without a test runner at all. An agent verifies an endpoint against staging that already carries leftover rows from a previous call, reports the response looks right, and misses that a clean tenant returns a different shape. The warmed instance masks the missing case the way fixed file order masks the failing test. The reliable check compares the live response against the contract on a clean baseline instead of a warmed-up instance. That local-first, source-of-truth loop is the one we built into Powerduck: a green or warmed response never counts as proof that the shipped surface matches the spec.

What I check before accepting a green suite

1. Did the new test run alone, or only as part of the fixed-order suite?
2. Does it build its own fixture inside setup, or read a module-level global?
3. Has the suite been run with randomized order and fresh per-file workers?
4. On a stateful API, was the check done on a clean baseline, or on warmed staging?

Green in one fixed order proves the files cooperate. It never proves each one stands alone, and that distinction is exactly what an isolated, shuffled pass exposes.