saboteur - mutation testing for Ruby

How it works

One small change at a time, checked by your own tests.

Change

saboteur finds your methods and makes one small change per mutant: flip a <, swap && for ||, swap min for max, drop a #{...} from a string, delete a line, empty the whole method.

Run only the tests that matter

A per-test coverage map tells saboteur which tests ran the changed line. Only those run, likeliest killers first. The first failure stops the run.

Check the kill

Every kill is run again on the unchanged code. If that test fails there too, it is not a kill. A kill that also fails on the original is not credited.

Read the survivors

You get a score and a diff for each survivor. A survivor can mean a missing test. It can also be an equivalent change no test could notice, a flaky test, or a limit of the tool, so each one needs a human look.

a survivor, as reported

# Billing::Invoice#total  - survived
-   return 0 if quantity >= 10
+   return 0 if quantity > 10
# No test uses quantity = 10. Add one.

Why saboteur

Built for real Rails suites, not only small gems.

Picks tests per mutant

Only the tests that executed the changed statement run. A change no test reaches is reported as "no coverage", not as a survivor.

Kills are re-checked

A kill must fail on the mutant and pass on the original. In one benchmark, 0 of 20 randomly sampled kills on lobsters were fake.

Coarse first

Each method body is replaced first: with nil (true and false for predicate methods ending in ?) unless the body is empty or only logs, and with raise unless a catch-all rescue would absorb it. A log-only body can still get raise; a body with a catch-all rescue can still get the nil or true/false replacements. Then the smaller eligible changes run for each covered method, within the per-method cap and any --since scope. A whole-body survivor does not hide a smaller one.

Diff mode

--since origin/main changes only the methods you edited on this branch. On one real change in saboteur's own history it took a file from 139 mutants to 7, and median wall time fell 22.9% (measured before the gate removal; one file, not a general speed claim).

Rails and parallel

Run with --jobs N workers. Each worker gets its own database copy. Tested on a Rails 8.1 app on Postgres with 6 workers, with the same verdicts as 1 worker.

Safe to run

In the default in-process mode, each mutant runs in a watched child process: everything it starts is tracked and killed, with limits on time, memory and process count. With --test-command, your command's process group is killed at a deadline instead; the memory and process-count limits do not apply there.

Open report

JSON in the Stryker mutation-testing-report format, so existing viewers can open it.

Repeatable

Every test run starts from the same random seed. Pass --seed to repeat a run.

Extendable

Add your own mutations by file and byte range, add hooks around each mutant, or write an adapter for another test framework.

Benchmarks

Rails mutation-score rebaseline

12 October 2026 · Post-removal rebaseline: 9 of 10 selected Rails/RSpec targets complete. Chatwoot is pending.

Project	Mutation score	Measured scope
paper_trail	81.3%	record_trail / widget specs
alchemy_cms	95.9%	Attachment / attachment specs
fat_free_crm	66.7%	Account / accounts controller specs
doorkeeper	89.5%	Access-token and application mixins / model specs
administrate	92.1%	Field::Base / base field specs
solidus	80.0%	StockItem / stock-item specs
publify	73.9%	Four sidebars and Gravatar / six specs
mastodon	74.3%	Account / account specs
lobsters	61.2%	Story / story specs
chatwoot	Pending	Conversation / conversation specs; not included in the nine completed results

Selected files and their configured RSpec suites, not whole-app scores. Post-removal engine 1154806; single complete runs with seed 1 and --jobs 2, not a performance comparison. Chatwoot is pending.

The score includes assertion kills, crash kills and timeouts. Errors, resource-limit stops and no-coverage work are excluded. A high score does not prove broad coverage or the absence of flaky tests.

Full method, scope and caveats

Compare

Measured head-to-head

Pinned historical runs against mutineer 1.6.0, kept in the archive. Measured versions, not a claim about today's tools; the fresh comparison is still pending. For the wider picture, see the open-source landscape comparison.

No fake kills

On lobsters, the 8 mutants both tools built identically were all reported as kills by mutineer - all 8 were fake. saboteur reported the same 8 as missed, which was correct, and 0 of its 20 randomly sampled kills were fake. It also refused 3 of its own kills, because those tests failed even without the mutant.

5-11x faster per mutant

Cold-cache wall time per mutant on the pinned libraries, and 1.2x on the Rails app; mutineer used less peak memory. On 6 workers, saboteur finished the Rails run in 125 s instead of 442 s with all 231 verdicts identical.

Runs the Postgres suite

saboteur ran both pinned Rails apps. On human-essentials (2,415 tests, Postgres) it finished item.rb: 98 mutants, 85 killed, 11 survived, in 520 s. mutineer did not run it after 5 tries, about 1 h 50 min.

Get started

Run it on your suite

Ruby 3.3 or newer, Linux or macOS, RSpec.

two commands

bundle add saboteur --group test
bundle exec saboteur run

Open the quickstart

Rails apps

The generator writes this for you (bundle exec rails generate saboteur:install). By hand, it is:

config/saboteur.rb

Saboteur.configure do |c|
  c.include = ["app/**/*.rb"]
  c.require = ["./config/environment"] # boots the app
  c.rails = true                       # per-worker databases after fork
  c.jobs = 4                           # optional; default is 1
end

Full Rails setup in the quickstart