How to Reduce Test Interference in Minitest
Global state can easily lead to interference between test cases and cause random failures. In this article, we’ll discuss a technique for alleviating this problem when reducing the global state is infeasible.
Background
I was working on the test suite for active_record_doctor
when I run into an interesting problem. Many active_record_doctor
tasks are applied at the model-level which means they iterate over all Active Record models and process them one-by-one.
I had to replace the dummy test app (that the Rails generator had created) with test cases based on dynamically-defined Active Record models. This improved test cohesion and readability but unexpectedly resulted in unpredictable test failures.
After spending way more time than I’d like on diagnosing the root cause, it turned out that Rails is leaking these dynamically-defined models. They were not garbage collected even after removing all references from my code. This caused the tests to break because active_record_doctor
would iterate over these redundant classes.
The most likely culprit was ActiveSupport::DescendantsTracker
. I didn’t want to spend more time on a solution especially that I wanted to support Rubies from 1.9.3 to 2.5.1 and Rails 4.2. to 5.2. Fixing the problem in one setup would make it reoccur in a different one.
I was left with the simplest solution: run one test per process. This would guarantee no leaks between test cases. I could have changed the way the test suite was run by listing all test files and passing them to rails test
via xargs
but I wanted to avoid leaking such implementation issues to the outside.
This forced me to create a custom Minitest runner that I packed up and published as minitest-fork_executor
. In this article, we’ll go through the reasoning behind the gem’s design.
Understanding Minitest
Minitest implements high-level logic in the Minitest
module in minitest/minitest.rb
. The entry point to the test suite is Minitest.run
. The comment above the method shows a call stack driving the test suite execution which will be helpful in a moment. It also does some housekeeping, part of which is starting and shutting down the parallel executor. parallel_executor
is defined via an accessor on Minitest
and is expected to be configured before starting the test suite.
The executor should implement #shutdown
and may implement #start
. Except for starting and shutting down, Minitest doesn’t reference the executor explicitly due to the assumption that calling #start
modifies Minitest
in-place. This means we should use #start
to modify Minitest
and make it fit our needs.
Knowing we need to implement an executor with a #start
method, let’s focus on what we need to change. The comment above Minitest.run
shows the call stack behind a test run:
##
# This is the top-level run method. Everything starts from here. It
# tells each Runnable sub-class to run, and each of those are
# responsible for doing whatever they do.
#
# The overall structure of a run looks like this:
#
# Minitest.autorun
# Minitest.run(args)
# Minitest.__run(reporter, options)
# Runnable.runnables.each
# runnable.run(reporter, options)
# self.runnable_methods.each
# self.run_one_method(self, runnable_method, reporter)
# Minitest.run_one_method(klass, runnable_method)
# klass.new(runnable_method).run
We can see it iterates over runnables and runs test methods one at a time. A runnable is usually a subclass of Minitest::TestCase
but can also be a benchmark, etc. Running a single test method is implemented in Minitest.run_one_method
. This is the method we need to modify in order to make forking the default behavior. The last step is figuring out what code to write to make forking work seamlessly with Minitest.
To sum up, we need to:
- Implement an executor.
- Make the executor override
Minitest.run_one_method
- Run a test method in a forked process.
Let’s tackle these problems one by one.
Step 1: Implementing an Executor
The executor is a simple two-method class that looks like this:
module Minitest
class ForkExecutor
def start
# We'll implement this in the next section.
end
def shutdown
# Nothing to do here but required by Minitest.
end
end
end
#shutdown
is required as Minitest
always calls it. #start
is the method where we’ll modify Minitest
. Before providing method bodies, let’s quickly cover how the executor is supposed to be used.
In a Rails app, we should configure the executor in test/test_helper.rb
in the following way:
require 'minitest/fork_executor'
Minitest.parallel_executor = Minitest::ForkExecutor.new
That’s it!
Step 2: Overriding .run_one_method
Minitest.run_one_method
is defined on the Minitest
module. It means it’s an instance method on the singleton class of Minitest
. It’d be ideal to be able to still call the original .run_one_method
to make our executor robust in face of changes to Minitest
. The initial implementation used module prepending to achieve that but I had to use a different technique in order to support Ruby 1.9.3.
Overriding .run_one_method
is a three-step process:
- Get a method object corresponding to
.run_one_method
. We need to reuse the original implementation from our implementation. You can think of as more elaboratesuper
. - Remove
.run_one_method
in order to avoid warning when redefining it. - Define a new
.run_one_method
onMinitest
.
Step 2 has one caveat. There’s define_singleton_method
but no remove_singleton_method
. This means we need to call remove_method
on the singleton class of Minitest
. Other than that, a straightforward translation of these steps into code looks like this:
def start
original_run_one_method = Minitest.method(:run_one_method)
class << Minitest
remove_method(:run_one_method)
end
Minitest.define_singleton_method(:run_one_method) do |klass, method_name|
# This is where we'll put our custom implementation in the next section.
end
end
We could store original_run_one_method
in an attribute and use it to reinstate the original method in #shutdown
but we’ll skip that for now. Time to focus on building the forking mechanism.
Step 3: Fork and Run
Forking is a UNIX concept that means creating another copy of the calling process. In Ruby, we can use #fork
which returns true
to the parent process and false
to the child process. In our case, the child process will run original_run_one_method
and the parent will simply receive the result from the child and return to Minitest
.
Forking is easy but we also need a way of sending the result object returned by original_run_one_method
back to the parent process. We can easily send Ruby objects over IO
by using the Marshal
module. In order to do that, we need to somehow create these IO
objects. This is where UNIX pipes enter the scene.
Without diving into low-level details, a pipe can be created by calling IO.pipe
. It returns a pair of IO
objects – one for reading, one for writing. They’re connected to each other in the sense that whatever is written to the write object can be read from the read object. On top of that, fork
will cause the child process to inherit the pipe so that both the child and the parent will be able to read from and write to the same pipe.
We’ve got all the ingredients for the algorithm:
- Open a pipe.
- Fork the process.
- Make the child run
original_run_one_method
and marshal the result back via the pipe. - Make the parent process unmarshal the result object via the pipe.
The code to do that is below. Please notice we also have extra steps in form of enabling the binary mode on the pipe and closing unused IOs.
Minitest.define_singleton_method(:run_one_method) do |klass, method_name|
read_io, write_io = IO.pipe
read_io.binmode
write_io.binmode
if fork
# Parent: load the result sent from the child
write_io.close
result = Marshal.load(read_io)
read_io.close
Process.wait
else
# Child: just run normally, dump the result, and exit the process to avoid double-reporting.
result = original_run_one_method.call(klass, method_name)
read_io.close
Marshal.dump(result, write_io)
write_io.close
exit
end
result
end
Mission Accomplished
I recommend you look at the complete gem. If you ever run into a test isolation problem at the process level then the gem can be a quick solution. This may be especially useful in projects with heavy meta-programming or lots of global state.
Enjoyed the article? Follow me on Twitter!
I regularly post about Ruby, Ruby on Rails, PostgreSQL, and Hotwire.
Copyright © 2019-2023 Greg Navis. All rights reserved.