I’ll admit it – I’m a test nerd. As much as love the creative process of writing an app, I’m equally fascinated by the problem of verifying all that code.
Although my real passion is in writing dynamic testing systems, I’m also a big believer in static tests – AKA unit, functional, and integration tests. Luckily, Ruby and Rails make it super easy to write tests.
Writing good tests is obviously important, but there’s also a hidden art to running your tests. You’d think this would be pretty straightforward and, when you just have a few tests, it is. But as you write more and more tests, it takes longer and longer run them all. And the longer it takes to run your tests, the less useful they are to you.
There are two ways to speed up your tests. Obviously, one way is to increase the speed of running each test – for example, by writing more efficient tests (for instance, by using mocks) or by getting a faster machine.
The other way is to run fewer tests. Although this seems dangerous in theory, in practice it’s really useful. When writing some code, you generally know which tests are most likely to break, so it makes sense to run a subset of your tests. Similarly, if you’ve just caused some tests to fail, you’ll likely want to continue to run just those tests until they pass. Just make sure to run all your tests before moving on to your next task – and definitely run every test under the sun before you even think about checking code into source control.
With that in mind, let’s survey some of the ways you can run your tests:
Rake
The simplest way to run your tests in Rails is with Rake. Simply run one of the following commands in your project directory
rake – Run all your tests. Equivalent to running rake test
rake test:units – Run just your unit tests
rake test:functionals – Run just your functional tests.
Well, you get the idea. Run rake --tasks | grep test to see all the test tasks at your disposal. As you’ll see, Rake is awesome at running all your tests or important sets of tests.
The problem with Rake is that you don’t have much control over which tests you are running. As I mentioned earlier, you might just want to run one testcase or even just one single test. Which brings us to …
Ruby
You can have finer grained control over your tests by invoking the Ruby interpreter directly. Let’s say you only want to run all the tests in one file. You can simply do:
ruby test/unit/foo_test.rb
Much better. On my machine, running all my unit tests via Rake took 10.7 seconds, whereas running just one specific test file took just 2.3 seconds.
But the fun doesn’t stop there. Oh no. Let’s try running just one specific test.
ruby test/unit/foo_test.rb --name=test_bar__some_test
Down to 1.9 seconds. What’s even cooler is that ‘–name’ can take a regular expression so you can run a group of tests (or even save time by not typing out the entire test name):
ruby test/unit/foo_test.rb --name=/test_bar/
OK, let’s review the results of my very non-scientific performance experiment:
* All unit tests via rake (110 tests) – 10.7s
* All tests in a single unit test file (7 tests) – 2.3s
* One test within a single unit test file – 1.9s.
What’s going on here? Even though I am running 1/110th of the tests, the tests only run about 5.6 times faster.
The problem with both Rake and Ruby is that it takes a long time to load the Ruby interpreter every time you run the tests. This is a small fraction of the overall time if you have lots of tests, but if you are just running a few tests, it really sucks. It would be nice if we could keep the Ruby interpreter in memory and just run the tests we need, so we turn to …
autotest
autotest is part of the awesome zentest suite. It’s easy to install:
gem install zentest
Now simply navigate to your project directory and run ./autotest
autotest will start by running all your tests and reporting any failures. Now for the sweet part – autotest will monitor and analyze both your application and test code and automatically run only those tests that validate code that has changed since you last saved. That’s just so freaking cool.
Not only does autotest speed up your tests by only running the ones that you need, it also runs the tests faster, since it doesn’t need to reload the Ruby interpreter over and over. Running the same single unit test in autotest took me about .1 seconds. Hot. In summary: if you’re not using autotest, you should.
I’ve only found two problems with autotest. First, sometimes the analysis algorithm gets confused and runs too few or too many tests (but this happens very, very rarely). More importantly, if you set a breakpoint in your code, autotest will just hit the breakpoint and wait – it won’t drop into an interactive Ruby prompt. To solve these (minor) problems, I wrote…
fast_test.rb
Using Rake tasks, I wrote my own test helper (with an admittedly terrible name). It’s the mutant offspring of the above methods (and it’s included at the bottom of this post, since it’s kind of long).
– Like autotest, it loads the Ruby interpreter once, so tests run faster
– Like the Ruby interpreter, you can provide a regular expression to run specific tests
– Like Rake, you can easily run just your unit, functional, or integration tests.
– Like Ruby or Rake, when you hit a breakpoint, you are dropped into an interactive environment
To use the script, place it somewhere in your application folder (I put it in ./script). Then, in your project directory, open (or create) .irbrc and add the following line
require 'script/fast_test.rb'
Save and close .irbrc and open up a rails console:
ruby script/console
Now you can do the following
test_all – Run all your tests
test_quick – Run just the unit and functional tests (not integration or other tests)
test_units – Run just your unit tests
test_functionals – Run just your functional tests
Again, you get the idea. Also, any of the above methods take an optional test name that is treated as a regular expression. So, to run a single unit test, you could do
test_units "test_name"
which, on my machine, took only .05 seconds. Now that’s how I roll.
Disclaimer: You’re free to use fast_test.rb, but I hereby declare that it’s current implementation may be terrible. Using rake tasks just seemed like the easiest way to accomplish what I needed and it’s worked pretty well for me for the past month or so. If you have any suggestions (on how to improve it or to suggest other, better written alternatives), I’d love to hear it.
Bringing it all together
So which is the best way to run your tests? It completely depends on your needs. I personally have one terminal window running autotest with another one running a rails console with fast_test.rb loaded. I generally let autotest do most of the work, but if I need to use the debugger or run a specific test of tests over and over, I use the console window. One warning: if autotest is running tests, don’t run tests in the console simultaneously – they’ll step all over each others’ toes and cause weird failures.
Like I said at the beginning, I’m a huge test nerd. So if you know of any other ways to improve your testing environment, let me know.
Oh, and before I go, here’s the code for fast_test.rb
### fast_test.rb ####################################
require 'rake/testtask'
def run_tests(sub_dir=nil,test_name=nil, long=false)
Rake::Task.clear
Rake::TestTask.new(:test) do |t|
t.libs << "test"
if(sub_dir.nil?)
t.pattern = 'test/**/*_test.rb'
else
t.pattern = "test/#{sub_dir}/*_test.rb"
puts t.pattern
end
t.verbose = true
t.options = "--verbose --name=/#{test_name}/" unless test_name.nil?
end
Rake::Task[:test].execute
"Done"
end
def test_units(pattern=nil)
run_tests("unit",pattern)
end
def test_functionals(pattern=nil)
run_tests("functional",pattern)
end
def test_integrations(pattern=nil)
run_tests("integration",pattern)
end
def test_quick(pattern=nil)
test_units(pattern)
test_functionals(pattern)
end
def test_all(pattern=nil)
run_tests(nil,pattern)
end
Pretheory: The Scariest Fun 