Archived comparison: saboteur vs mutineer

Current Rails score rebaseline · Historical measurements, not a current product ranking.

Note: measured at 2b6ec02, before the coarse-survivor gate was removed and before the rebaseline. Counts and times are historical; a rerun is pending.

2026-10-10 · saboteur 0.1.0 (commit 2b6ec02 engine) vs mutineer 1.6.0 · by Vamsi Pavan Mahesh Gunturu

saboteur and mutineer are two open-source tools that measure how good a Ruby project's tests are. We ran both on the same four RSpec projects and checked their answers against ground truth. saboteur came out ahead on Rails support and speed, and its kill-checking sample found no fake kills (a sample check, not a ranking); mutineer came out ahead on memory, choice of change types and CI features.

Verdict

saboteur wins 3 of 7 ranked metrics, mutineer wins 2, and 2 are ties. The two trust rows are sample checks and are not ranked. The most notable finding is: on lobsters, 8 of mutineer's kills that we hand-checked were not caught by any test. That is a check of 8 identical mutants, not a measure of either tool's overall accuracy.

Green: better result in the row. Red: worse. Grey: tie. Minitest support is out of scope for saboteur by design and is not scored.

Glossary

Mutation testing

Checking how good tests are by making small deliberate changes to the code and seeing whether the tests notice.

Mutant

One such change, for example a < b changed to a <= b, or one line removed.

Killed (caught)

At least one test failed with the mutant in place and the unchanged check passed. An assertion kill records a failed assertion; a crash kill records an exception or other crash. These are different evidence, even when both count as detected.

Survived (missed)

The selected tests passed with the mutant in place. A survivor can mean a missing check, an equivalent change, flakiness or a tool limit; human review is needed.

Fake kill

A mutant reported as killed when the same tests also fail without the mutant. The failure came from something else, such as leftover test data. A crash can also be a legitimate kill when the mutant causes it. A fake kill hides a real gap.

No coverage

No test runs the changed line, so the mutant is not tested at all.

Timeout

The tests ran past the time limit, often because of an endless loop. The current saboteur counts a timeout as detected in the score.

Error

The tool could not reach a clean verdict. saboteur also uses this when a killing test fails even without the mutant.

Mutation score

Current saboteur: (killed + timeout) / (killed + survived + timeout). Assertion and crash kills both count; the score alone does not distinguish them. The scores on this page come from the measured version and may use a different definition. Comparable only between runs with the same set of mutants.

Operator

A kind of change a tool makes, for example removing a statement or flipping a comparison.

Defaults, matched

Defaults: each tool's own change types. Matched: the closest common set, so both tools test similar changes.

Worker

One parallel process running mutants.

Actionable, equivalent, noise

Labels for missed mutants. Actionable: a test should catch it. Equivalent: the change cannot alter behaviour. Noise: it changes behaviour, but not in a way worth testing.

Setup

Both tools ran on the same machine, the same Ruby, the same source files and the same test files, with caches cleared before every run. mutineer was configured as its README recommends.

Machine: Apple M3 Max, 14 cores, 36 GB, macOS 14.3. Ruby 3.4.3, and 4.0.0 for lobsters. Each project's tests passed 3 runs in a row before use; for human-essentials, 1 unreliable test file and 6 files that fail when run on their own were left out for both tools.

Result 1: fake kills

mutineer reported fake kills on lobsters; saboteur had none in its sample.

mutineer scored 100% on the three lobsters models and named no killing test for any of its 91 kills. saboteur reported the same 8 mutants as missed, which is correct. saboteur also refused 3 of its own kills, because the killing test failed even without the mutant (leftover test data). The cause inside mutineer was not found.

Result 2: Rails apps and Postgres

saboteur ran both Rails apps, in parallel where asked; mutineer ran lobsters on one worker and could not run human-essentials.

Result 3: speed and memory

saboteur ran each mutant 5 to 11 times faster on the libraries and 1.2 times faster on the Rails app; mutineer used less memory.

How many times faster saboteur runs each mutant than mutineer. Wall time divided by mutants run, cold caches.

These rows are historical (measured versions, summed process RSS, which counts shared pages more than once), not a current ranking. The saboteur memory figures in this table were measured before the fork-memory work. Result 6 below has the paired before and after on hashie.

One mutineer run is unexplained: hashie with 1 worker took 2,640 s, against 34.9 s with 6 workers. It needs a repeat and is left out of the verdict. CI cost is small for both: at GitHub's $0.006 per minute for a standard Linux runner (source), each run here costs under $0.03 (estimate).

Result 4: usefulness of missed mutants

Both tools found about the same number of useful missed mutants; mutineer's misses were more often noise.

10 missed mutants per tool from hashie were mixed with tool names hidden. Three reviewers (two Claude, one Grok) labelled all 20 and agreed on 14 to 18 of 20 per pair; the split items were settled by reading the code before unblinding.

Result 5: real past bugs

Neither tool detected any of the 3 hashie bugs: all 3 fixes added missing code that no change type can produce.

Result 6: memory after the fork-sharing change (hashie)

Peak PSS dropped about 17-18% with 2 or more workers; verdicts did not change.

Paired A/B runs against the pre-PR commit (32c0f21), 5 pairs per job level with AB/BA alternation and cold caches, on a 2 vCPU / 2 GB shared sandbox. Peak memory is per-process-tree PSS from /proc/smaps_rollup sampled at 0.3s. Verdict parity: 0 verdict differences in 23,700 mutant outcomes (1,580 mutants x 15 pairs); kill reasons (assertion vs crash class) vary run to run on both variants, as before.

Peak RSS is unchanged (median ~660 MB at jobs=6 both variants): the PR does not touch less memory, it keeps more of it shared between workers after fork, which is what PSS measures. On this workload the PSS drop is about 17-18% at 2 and 6 workers. Swap was not measured.

Wall time on uncontended runs, per-cell minimums, PR first: 48s vs 49s at jobs=1, 32s vs 31s at jobs=2, 30s vs 32s at jobs=6. These are noisy per-cell minima from historical runs and do not show a direction either way. The shared sandbox had multi-minute CPU-starvation windows during this session (walls up to 594s); affected runs are flagged via sampler cadence and excluded from wall analysis. Absolute times are specific to this 2-core sandbox.

Environment and method

• Candidate: saboteur at d2b7abb336eae53ec8cd5a71c111f9daef66218a (merge of PR #4, "Pack the heap at fork boundaries"). Baseline: 32c0f21bdcbc059ac5938ae7630eb117442cd5fc, the first code-change ancestor; 32c0f21..d2b7abb^ is doc-only (AGENTS.md, docs/status.md).
• Target: intridea/hashie at 9c81e8340d6b69940e14d8411f34a3efd7a6aa24 (261 subjects, 719 tests, 1,580 mutants).
• Hardware: shared CI sandbox, 2 vCPU (x86_64), 2 GB RAM, Ubuntu 22.04.5, Ruby 3.4.3 (+PRISM). Absolute wall times are sandbox-specific; only same-box paired ratios are meaningful.
• 30 runs: jobs in {1,2,6} x {baseline, candidate} x 5 pairs, AB/BA alternation (odd pairs baseline-first, even pairs candidate-first), serial under flock, cold cache (rm -rf .saboteur) per run.
• Memory sampled every 0.3s across the whole process tree from /proc: RSS from status, PSS from smaps_rollup (falls back to RSS when unreadable). Peak over the run is reported. Wall is end-to-end run_one duration including the coverage-map build.

Host contention

The sandbox shares CPU with other tenants. Starvation windows show in the sampler cadence (healthy 0.3s per sample; starved over 1s per sample). Runs flagged contended (wall inflated, memory peaks unaffected): hashie-j2-baseline-1 (83s, borderline cadence 0.94s), hashie-j6-baseline-1 (273s), hashie-j1-candidate-2 (279s), hashie-j2-candidate-2 (261s), hashie-j6-candidate-2 (251s), hashie-j1-baseline-3 (292s), hashie-j1-candidate-3 (282s), hashie-j2-candidate-3 (275s), hashie-j6-candidate-3 (267s), hashie-j2-candidate-4 (580s), hashie-j6-candidate-4 (571s), hashie-j1-baseline-5 (594s), hashie-j1-candidate-5 (288s), hashie-j2-candidate-5 (268s), hashie-j6-candidate-5 (264s). Contended windows hit baseline and candidate slots both (order-bound, not variant-bound). Wall analysis uses clean runs and per-cell minimums; PSS and RSS peaks are valid in all windows.

Features compared

The pinned report at bench/vs-mutineer/REPORT.md records 20 operators for mutineer 1.6.0. A later README table says 22 without a matching measured-version record. This archive preserves the measured report's 20; neither count is asserted as a current competitor feature count.

Capabilities at the measured versions (saboteur as measured, mutineer 1.6.0), historical, not current. Some rows have changed since; for example saboteur now has --threshold, an HTML report and a GitHub Action.

Where mutineer won

At the measured versions. Not a statement about current releases.

1. Lower peak memory in every pair measured.
2. 20 change types with a switch to choose them; saboteur has one fixed set.
3. CI features: baseline, score threshold, HTML report, GitHub Action with PR annotations, inline disable comments.
4. A report of tests that never catch anything, and of redundant tests.
5. No runtime dependencies.
6. It refuses to score when a test file fails on its own. That blocked it on human-essentials, but it is a defensible safety rule.

Limitations

• Results 1-5: one run per configuration, so no ranges and no repeatability numbers. Result 6 has 5 pairs per job level.
• Rails apps were sampled: 3 lobsters models and 1 human-essentials model.
• mutineer's fake kills were checked on 8 identical mutants, not a random sample, because its report names no killing test.
• Only 3 real bugs, none detectable by either tool; 10 missed mutants per tool for usefulness.
• Results 1-5 ran on macOS only; mutineer's Postgres failures may not occur on Linux.
• Not measured: time to first missed mutant, diff-mode timing, database connections left after runs.
• The author of this benchmark is also the author of saboteur.

Reproduce it

Everything is in bench/vs-mutineer: the target commits and setup (PHASE0.md, PHASE1.md), the scripts (run_all, run_one), the raw JSON results and the full report (REPORT.md). Issues found in mutineer are written up in upstream-issues.md.