Run Only Relevant Tests

When working on large software projects, "running the test suite" often makes me sigh. Waiting 20 to 30 minutes (or longer) for all tests to complete interrupts my workflow and kills my productivity. But here's what really frustrates me: as Aaron Patterson points out, "99% of the tests aren't even running the code that I changed!" Why waste time on tests unrelated to my modifications?

This is where Regression Test Selection (RTS) comes into play, a technique designed to predict which tests are likely to fail after code changes. The goal is to identify a minimal subset of your test suite that still ensures the integrity of your changes. If you can run just those relevant tests, you can significantly accelerate your development cycle.

Enter rspec-big-infer – a set of helper tools I created to solve this exact problem in big projects. This gem aims to help you work efficiently with your RSpec test suite by inferring which tests to run based on recent codebase changes.

How RSpec::Big::Infer Works Its Magic

The core idea behind rspec-big-infer is to map changes in your code to the tests that cover those specific areas, allowing you to run only the necessary tests. Here's a breakdown of its usage:

Installation

To get started, you simply add the gem to your application's Gemfile by executing bundle add rspec-big-infer.

Generating a Test Map

The first step involves creating a map of your tests. This is done by running your RSpec suite in a dry-run mode with a special formatter:

bundle exec rspec --dry-run --format RSpec::Big::Infer::Formatter --out tmp/rspec_infer.json

This generates a JSON file that maps your test files to the source files they test. Here's what the output looks like:

{
  "version": "3.13.5",
  "seed": 34755,
  "examples": [
    {
      "test_file_path": "spec/controllers/attachments_controller_spec.rb",
      "described_class_defined_source_location": "app/controllers/attachments_controller.rb"
    }
  ]
}

Crucially, rspec-big-infer works by leveraging described_class and const_source_location to understand the relationships between your code and your tests. The Module#const_source_location method, added in Ruby 2.7, returns the Ruby source filename and line number where a constant is first defined. It functions similarly to Method#source_location but for constants. This allows rspec-big-infer to quickly calculate the test map on the fly, as it gathers metadata without actually executing the test code itself. This is a key advantage, leading to a much faster setup phase for Regression Test Selection compared to methods that require full test execution.

Identifying Code Changes

rspec-big-infer then uses git diff --name-only --diff-filter=D origin/develop to identify files that have changed in your codebase. This is crucial for determining which parts of the application have been affected.

Inferring and Running Relevant Tests

With the test map and the list of changed files, rspec-big-infer can then infer which tests are relevant to your recent modifications. You can optionally preview which tests will be selected, then run only those inferred tests. See the gem's documentation for the current command syntax and usage examples.

For a more comprehensive automation, rspec-big-infer can be handled by bin/run-infer-rspec in your project. This script can handle checking for a clean working directory, fetching the latest changes, calculating file relationships, parsing the generated JSON, and finally running the suggested tests.

The Benefits for Large Projects

The advantages of implementing a tool like rspec-big-infer are particularly significant for large projects with inherently long test suites:

• Drastically Reduced Test Times: By running only the tests relevant to recent changes, you significantly cut down the time spent waiting for test completion. This is perhaps the most immediate and impactful benefit.
• Enhanced Developer Productivity: Less waiting means developers can stay focused and productive, without context switching due to long test runs.
• Efficient Resource Utilisation: You're not wasting computing resources on running tests that are highly unlikely to fail given the specific code changes.
• Ideal for Legacy Codebases: The value of such a tool is especially high in legacy codebases where tests might be slower or the codebase is complex and vast.

Background and Alternatives

It's worth noting that the concept of Regression Test Selection has been explored by others, such as Aaron Patterson, who advocated for this approach in 2015, even encouraging others to build and maintain such a tool as a gem. Crystalball, another Ruby library for Regression Test Selection, also aimed to select a minimal subset of an RSpec test suite. However, the toptal/crystalball repository was archived by its owner on 24 January 2025 and is now read-only.

A key difference between crystalball and rspec-big-infer lies in their map generation approach:

• Crystalball (and Aaron Patterson's original implementation) required actually running your tests to collect per-test code coverage information. This involved recording coverage snapshots before and after each test execution to determine what code a specific test ran. Consequently, generating the execution map with Crystalball typically required the full test suite to be executed at least once, which could be time-consuming for large projects.
• In contrast, rspec-big-infer generates its test map using RSpec's --dry-run mode. This mode allows it to traverse the test suite and gather metadata about tests (like described_class and const_source_location) without executing the actual test code. This approach allows the test map to be quickly calculated on the fly and takes considerably less time compared to a full test run with coverage collection.

In conclusion, for any development team struggling with slow test suites in a large RSpec project, rspec-big-infer offers a practical and effective solution. By intelligently selecting only the necessary tests and providing a fast test map generation process, it helps streamline your workflow, save precious development time, and keep your team moving forward with confidence.