Fixture

Writing

We mutation-tested 10 popular open-source projects across five languages

Frank O'Hara ·

Every project in this study has a test suite most teams would envy. Thousands of tests, high coverage, CI green on every merge (mostly). We planted realistic defects in ten widely used open-source codebases, in Python, JavaScript, TypeScript, Go and Rust, and measured how many of those defects the tests caught.

TL;DR:

Why coverage lies to you

A line is "covered" when a test executes it. Nothing in that definition requires the test to check what the line did. A test that calls a function and asserts on an unrelated return value covers every line of that function while verifying none of them.

Mutation testing closes that gap by measuring the thing you actually want to know: does the suite fail when the code is wrong? A tool introduces a small, realistic defect (an inverted guard, an off-by-one boundary, a dropped negation), runs the tests, and records whether anything noticed. A defect that survives is a place where your tests run the code but do not check it.

your code, with one small planted defect run your tests tests fail → killed the suite caught it tests pass → survived a gap coverage can't see
The whole idea in one loop. A "survivor" is code your tests run but do not check.

This has been an academic technique for decades. Two things make it urgent now. First, AI assistants write a growing share of tests, and they are good at producing tests that execute code and assert something plausible, which is exactly the failure mode coverage cannot see. Second, the same assistants make it cheap to act on the results: a surviving mutant plus the tests that should have caught it is a precise, mechanical prompt.

What we ran

Flawd is our mutation-testing tool. We ran it at whole-codebase scope against ten projects, chosen for popularity and for covering the five languages it supports. The scores below are the verified numbers: every survivor behind them was individually confirmed against the project's full harness suite, for reasons the next section makes painfully clear.

Project Language Verified score Mutants
Celery Python 100% 300 (budgeted)
Vector Rust 100% 100 (budgeted)
Parse Server JavaScript 95.5% 200 (budgeted)
Traefik Go 91.2% full pool
Superset Python 90.7% 2,000 (full depth)
Ghost JavaScript 89.0% 400 (budgeted)
Loki Go 87.7% 1,859 (full depth)
Quickwit Rust 84.6% 1,799 (full depth)
Prisma TypeScript 83.7% 1,038 (full depth)
TypeORM TypeScript 72.6% 146 (full pool)

Score is killed divided by killed plus survived; timeouts and infrastructure errors are excluded from the denominator and reported per project in the raw data. Budgeted runs sample the highest-ranked mutants first, so treat cross-project comparisons as directional, and read the two 100% rows with their stories (below) rather than at face value: Celery's number is real but was only reachable after we fixed our own harness, and a sixth of Vector-class Rust kills come from the compiler rather than the tests.

A score in the high eighties is not a grade to be ashamed of. These are excellent suites, some carrying well over a decade of accumulated engineering. The point is not that flagship projects are sloppy. It is that even suites this good leave a measurable slice of realistic defects undetected, and that slice is invisible until you go looking for it.

The part where we distrust our own numbers

Between the first survey runs and this writeup we spent two weeks trying to break our own results, and repeatedly succeeded. Six distinct problems turned up, each capable of moving a published number, and several were bugs in our own tool that we fixed across three releases before trusting anything. If you only read one section, read this one, because every mutation-testing writeup you have ever seen is subject to some of these and most never mention them.

Targeted execution invents survivors. For speed, Flawd runs each mutant against only the tests that cover the mutated line. If that selection misses a relevant test, a mutant gets marked surviving when the full suite would have killed it. We re-ran every reported survivor against its full suite, project by project, and the false-survivor rate was all over the map:

Celery 100% (44 of 44) Superset 38% (113 of 295) Loki 26% (79 of 306) Prisma 3% Parse Server 0% (0 of 9)
Share of initially reported survivors that the full suite actually kills. Nothing about a project predicts its bucket, which is why every survivor count in this post is full-suite confirmed and why Flawd now ships --confirm-survivors.

An early Celery run of ours had reported 160 survivors through exactly this mechanism, with no confirmation pass. That number was garbage, and we would have published it.

Batched execution invents kills. Compiled languages get "schemata" batching: many mutants compiled into one binary, toggled at runtime. Anything that leaks between batch runs can credit the suite with a kill it did not earn, so we re-ran batched kills one at a time in fresh isolation. Go came out spotless: 1,638 kills re-verified across Traefik and Loki, zero false. Quickwit was the shocker: 214 of 1,178 batched kills, 18.2%, were false. The mechanism turned out to be mundane and fixable, a working-directory bug that made four filesystem-dependent tests fail identically under every batch, and it is fixed at the source now. But without kill verification we would have published a Quickwit score inflated by a config error, and the residual, genuine false-kill rate after the fix (about 0.1%) is the number we actually stand behind.

Misaligned coverage invents unkillable survivors. Vector's coverage recipe measured tests its test command never ran (a --skip filter applied to one and not the other), so ten files were marked covered by tests that do not execute. Mutants in those files would have survived anything. No tool warning catches this, because from the tool's perspective the coverage data is simply wrong. We caught it auditing fixtures by hand, fixed it, and re-ran.

Compile errors are not kills, and they favor Rust. When a mutant fails to compile, the suite did not detect anything; the type system rejected an invalid program. In Quickwit's run, 146 of 1,476 kills were build failures. Rust's type system rejects far more mutants than Python's runtime does, so scoring build failures as kills inflates exactly the cross-language comparison this study invites. Read the Rust rows with that in mind.

Generated code pads survivor counts. Nine of Traefik's surviving mutants sat in zz_generated.deepcopy.go, a file the project forbids editing by hand, and eight of Loki's sat in a generated yacc parser our exclusion patterns missed. Survivors in generated code are real gaps in a technical sense and useless in a practical one. We report actionable survivors separately from raw counts.

Your harness's exclusions become the project's false gaps. This one cost us a finding we had already written up. Our Celery verification run produced three survivors in detect_settings, all full-suite validated, all reproducible, a textbook untested branch. Except: our Docker harness had silently excluded one test file it could not collect (a missing pydantic install, it turned out), and that one file kills two of the three. The third fell when we rebuilt the container correctly and reran everything. All three of our "textbook findings" were artifacts of how we ran the suite, not gaps in it. Any mutation-testing setup that skips tests it finds inconvenient (and every containerized setup skips something) will report the gaps those tests would have covered as if they were the project's. We audited every harness in this study for this, it paid out again on Traefik, and the appendix lists what each harness excludes so you can weigh the numbers accordingly.

Some of these push scores up and others push them down. They do not cancel out, and averaging them away is not a methodology. The honest version of this study took two weeks longer than the credulous version would have, made most scores go up, not down, and left us with a tool that now ships the verification passes we wished we'd had on day one.

What actually survives, in real code

A score tells you less than one concrete miss. Here are verified examples, each confirmed against the project's full suite and reproduced by hand.

Traefik: inverted guards in the routing config

NewConfig in pkg/config/runtime copies each section of a dynamic configuration (HTTP services, middlewares, their TCP and UDP counterparts) into runtime maps. Three of its length guards are invertible with every unit test green:

if len(conf.TCP.Services) > 0 {   // flip to <= 0: a populated section
    // ...populate runtime map      // now produces a nil map, silently
}

The mechanism is instructive. NewConfig's test-file callers all use it to build a fixture and then assert on GetRoutersByEntryPoints. Every line of the function is covered and nothing that tests the function reads the maps it fills, so its coverage is perfect while its direct detection rate is zero. We wrote a table-driven test that asserts on every section of the result, verified it fails against each mutant, and opened it upstream (issue, PR).

Celery: caught, but by accident

Celery's suite is the strongest we tested: across four runs and roughly a thousand executed mutants on the corrected harness, zero survivors. The most interesting kill deserves a closer look, because the suite catches it the way you catch a glass someone else knocked off the table.

detect_settings in celery/app/utils.py decides whether a configuration uses old- or new-style setting names. Mixing the styles is an error, and the message tells you which side to rename based on which style holds the majority. Negate the if is_in_new: guard and the only observable change is in the tie: a config with equally many old and new names gets told to rename toward the deprecated format. Every mixing test asserts that mixing raises; none asserts which way the message points. Run the affected test file alone under this mutant and it passes clean. The mutant still dies in our runs, because 905 tests into the full sequential suite an unrelated filesystem-backend test trips over leaked state that the flipped branch perturbs. Remove that accident of ordering and the defect ships. We wrote a thirteen-line test that pins the tie direction deterministically and offered it upstream.

Superset: the newest code has the softest assertions

Superset's verified score is 90.7%, and the location of its 186 confirmed survivors is the finding: 34 of them, the largest cluster by far, sit in superset/mcp_service/, the AI-integration service added to the codebase most recently. Guard inversions and boundary shifts in its chart rendering, schema discovery and token utilities pass the entire 9,200-test unit suite. The decade-old query-model core, by contrast, is dense with tests that bite. There is also a small cluster of five in security/manager.py, which is not many, but they are in security dispatch code, so they are arguably the five most valuable missing tests in this study.

New code having thinner tests is not surprising. New code written in the AI era having tests that execute everything and check little is exactly the trend this whole post is about, showing up inside the measurement itself.

Ghost: the member counter that can't count to ten thousand

Ghost's unit suite caught 356 of 400 planted defects, and 26 of the 44 misses are equivalent mutations, so its meaningful catch rate is about 95 percent. The most tellable miss is memberCountRounding, the function that turns a raw member count into marketing copy like "10,000+" or "550k+". Its test exercises 55, 580, 5,555 and so on: comfortable mid-range values, never a threshold. Change the 10000 constant to 9999 and the function returns undefined for a site with exactly ten thousand members, with every test green. The tests pinned the rounding and left every boundary loose, which is precisely the kind of test an autocomplete writes from the function's happy path. A 20-line boundary test kills all three killable mutants; we verified it against each and offered it upstream.

An honest wrinkle: two of the five boundary mutants in that function turned out to be unkillable, because at the single re-routed value the adjacent branches happen to compute the identical string. No test can distinguish them. Our tool tagged 26 equivalents in this run and missed these two; we filed that gap against ourselves.

Our own tracker: 86 survivors in the codebase we dogfood daily

We run Flawd on Lash, our task tracker, precisely so we cannot exempt ourselves. Score 77.5 percent, 86 confirmed survivors, which is worse than nearly every project in the table above. Three clusters became a single upstream-style test PR that killed eight of them, each new test verified to fail against its mutant before the fix landed.

What the maintainers did with it

We opened a thread at every project before publishing: working test PRs where the survivors were most actionable, findings posts or emails everywhere else, each written to the project's contribution rules and carrying a one-line disclosure that the work came out of a tool we build.

The honest status at publication: early. Traefik's maintainers acknowledged the issue, and the test PR is open awaiting review. Ghost's boundary-test PR went up last, after the project's forum-first etiquette left our post sitting in moderation. Celery and Vector came back clean, so their threads are findings notes rather than fixes (celery, vector); Celery's tie-direction test sits on a pushed branch, offered if the thread wants it. The TypeORM, Prisma, and Parse Server threads, and the notes to Superset, Quickwit, and Loki, are posted and quiet so far. We will update this section as the threads move.

Running this on your own repo

curl -fsSL https://fixture.dev/download/flawd/darwin/aarch64 -o flawd
chmod +x flawd
cd your-repo
./flawd init          # detects language, test runner, coverage
./flawd run

(Linux: swap in linux/x86_64. All platforms are on the product page.)

Run Flawd on your own tests to see where you stand. Everything runs locally; no code leaves your machine. flawd init works out your test command and coverage setup for common stacks. On a mid-sized repo expect minutes, not hours, for a budgeted run. Two flags we consider non-negotiable after writing this post: --confirm-survivors before you act on any survivor list, and --verify-schemata-kills before you quote a compiled-language score. Both exist because of what this study taught us. The sample report is a full unedited run if you want to see the output shape first.

There is a 14-day evaluation at fixture.dev/eval; the sample report needs no signup at all.

Where this goes

We came out of this more optimistic than we went in. The suites were better than the cynical take predicts, the misses were specific and cheap to fix once located, and maintainers now hold a dozen ready-made tests that each pin a real, demonstrated gap. The economics have quietly flipped: finding a gap costs a CPU-hour, writing the test that closes it costs an AI-assisted afternoon, and verifying the fix is mechanical. The only expensive part left is knowing where to look, which is exactly the part a mutation score automates.

If assistants are going to write an ever-larger share of the world's tests, something has to check the checkers, continuously and without ceremony. We think that check belongs in the same place coverage lives today: in CI, on every merge, unremarkable. This study is what it looks like to hold that tool to the standard it will need. We broke it six ways getting here, fixed all six, and published the breakage alongside the scores, because a verification tool you cannot audit is just another green checkmark.

Methodology, in full

The raw run reports for every project are available on request; the per-project fixture configurations are in the fixtures repo above.

Appendix: what each harness actually runs

No mutation-testing harness runs "the test suite." Each of ours runs a reduction of it, and after the Celery lesson above we think publishing the reduction is as important as publishing the score. Survivor-level audits against these scopes are complete for all ten projects.