Rails mutation-score rebaseline
12 October 2026 · Post-removal rebaseline: 9 of 10 selected Rails/RSpec targets complete. Chatwoot is pending.
| Project | Mutation score | Measured scope |
|---|---|---|
| paper_trail | 81.3% | record_trail / widget specs |
| alchemy_cms | 95.9% | Attachment / attachment specs |
| fat_free_crm | 66.7% | Account / accounts controller specs |
| doorkeeper | 89.5% | Access-token and application mixins / model specs |
| administrate | 92.1% | Field::Base / base field specs |
| solidus | 80.0% | StockItem / stock-item specs |
| publify | 73.9% | Four sidebars and Gravatar / six specs |
| mastodon | 74.3% | Account / account specs |
| lobsters | 61.2% | Story / story specs |
| chatwoot | Pending | Conversation / conversation specs; not included in the nine completed results |
These are mutation scores for selected source files and their configured specs, not whole-app scores or a ranking of the projects. Higher scores mean the selected tests detected more of the tested changes. No-coverage work is excluded, so a high score does not establish broad coverage.
The post-removal engine is pinned to 1154806b6c66ef03826f8c137e9d9ad4b5616ae0, after the coarse-survivor gate was removed. These are measurements of that snapshot, not of every later release. Smaller eligible mutations now run even when a whole-body mutation survives. All nine rows used RSpec, seed 1, --jobs 2, harness v8.4 and POST_REMOVAL=1. Each app ran alone on its host. No multi-seed stability claim is made. Download counts, Ruby versions and exact app/engine pins (CSV).
Score = (killed + timeout) / (killed + survived + timeout). Assertion kills, crash kills and timeouts all contribute; errors, resource-limit stops and skips do not. A crash kill means changed code broke, not that a test asserted its return value. These completed runs had no tool errors or resource-limit stops. Mastodon has one per-mutant timeout, counted in its score.
Each result is a single complete run on a selected workload. The unchanged configured suites passed before mutation, and kills were checked against unchanged code. That does not prove flaky tests are absent. No speed or memory ranking follows from these scores, and different mutation sets make cross-version score comparisons limited.
Reading the scores
- Assertion kill
- A mutant made a test assertion fail and the unchanged code passed the check.
- Crash kill
- A mutant caused an exception or another recorded crash. It counts as detected, but is different evidence from an assertion.
- Survivor
- The selected tests passed. This may be a missing check, an equivalent change, flakiness or a tool limit; human review is needed.
- No coverage
- No selected test reached the edited code. It is skipped, not credited as a kill or counted as a survivor.
- Timeout
- The mutant exceeded its measured deadline. Current scoring counts it as detected; it is not an assertion kill.
- Incomplete
- Errors, resource-limit stops or no tested mutants mean the run cannot be called complete, even if a partial score exists.
Kill evidence by project
Counts, not a separate accuracy ranking. Timeouts are shown separately from assertion and crash kills.
| Project | Tested | Assertion kills | Crash kills | Timeouts | Survivors |
|---|---|---|---|---|---|
| paper_trail | 209 | 60 | 110 | 0 | 39 |
| alchemy_cms | 97 | 51 | 42 | 0 | 4 |
| fat_free_crm | 39 | 11 | 15 | 0 | 13 |
| doorkeeper | 390 | 185 | 164 | 0 | 41 |
| administrate | 101 | 53 | 40 | 0 | 8 |
| solidus | 75 | 39 | 21 | 0 | 15 |
| publify | 46 | 18 | 16 | 0 | 12 |
| mastodon | 183 | 75 | 60 | 1 | 47 |
| lobsters | 896 | 308 | 240 | 0 | 348 |
Scope and caveats
- Skipped subjects remain outside these scores. For example, Lobsters has 33 and Mastodon has 37 uncovered methods in the selected targets.
- Lobsters: 548 kills, 348 survivors, 119 no-coverage mutants; 896 tested. Assertion share 34.4%, crash share 26.8%. Score without whole-body mutants: 58.5%. All 843 old digests remain present with counted verdicts unchanged; 196 formerly gate-suppressed mutations are now 41 kills, 123 survivors and 32 no-coverage outcomes. The new snapshot adds 172 digests.
- Paper_trail and Alchemy dependencies were freshly resolved because original locks were unavailable. Alchemy dependencies include moving branches; these reconstructions are not exact dependency replicas of the earlier run.
- Fat_free_crm used Ruby 4.0.2; the older environment record is ambiguous about Ruby, so its pre/post pair is not an exact runtime comparison.
- Solidus, Publify and Mastodon are reconstructed pre/post pairs with dependencies frozen between the two engines, not proven identical to the expired run environment. Mastodon uses a Ruby pin patch from 4.0.7 to 4.0.2 and minimal libvips 8.14.5 without optional media loaders. Media behavior is outside its selected account-spec workload.
- The earlier engine is f992720 for the reconstructed pairs. Some older preserved results predate that pin. Scores changed with both the removed gate and additional candidates; a lower score alone is not a regression of test verdicts.
Historical measurements
The earlier saboteur-versus-mutineer comparison and paired hashie fork-sharing experiment remain available in the archive. They are not a current competitor ranking. The competitor comparison rerun is separate and still pending; its pinned historical workload has not been changed.