Adventures in Testing, Part I : Running Your Tests

by

I’ll admit it – I’m a test nerd. As much as love the creative process of writing an app, I’m equally fascinated by the problem of verifying all that code.

Although my real passion is in writing dynamic testing systems, I’m also a big believer in static tests – AKA unit, functional, and integration tests. Luckily, Ruby and Rails make it super easy to write tests.

Writing good tests is obviously important, but there’s also a hidden art to running your tests. You’d think this would be pretty straightforward and, when you just have a few tests, it is. But as you write more and more tests, it takes longer and longer run them all. And the longer it takes to run your tests, the less useful they are to you.

There are two ways to speed up your tests. Obviously, one way is to increase the speed of running each test – for example, by writing more efficient tests (for instance, by using mocks) or by getting a faster machine.

The other way is to run fewer tests. Although this seems dangerous in theory, in practice it’s really useful. When writing some code, you generally know which tests are most likely to break, so it makes sense to run a subset of your tests. Similarly, if you’ve just caused some tests to fail, you’ll likely want to continue to run just those tests until they pass. Just make sure to run all your tests before moving on to your next task – and definitely run every test under the sun before you even think about checking code into source control.

With that in mind, let’s survey some of the ways you can run your tests:

Rake

The simplest way to run your tests in Rails is with Rake. Simply run one of the following commands in your project directory

rake – Run all your tests. Equivalent to running rake test
rake test:units – Run just your unit tests
rake test:functionals – Run just your functional tests.

Well, you get the idea. Run rake --tasks | grep test to see all the test tasks at your disposal. As you’ll see, Rake is awesome at running all your tests or important sets of tests.

The problem with Rake is that you don’t have much control over which tests you are running. As I mentioned earlier, you might just want to run one testcase or even just one single test. Which brings us to …

Ruby

You can have finer grained control over your tests by invoking the Ruby interpreter directly. Let’s say you only want to run all the tests in one file. You can simply do:

ruby test/unit/foo_test.rb

Much better. On my machine, running all my unit tests via Rake took 10.7 seconds, whereas running just one specific test file took just 2.3 seconds.

But the fun doesn’t stop there. Oh no. Let’s try running just one specific test.

ruby test/unit/foo_test.rb --name=test_bar__some_test

Down to 1.9 seconds. What’s even cooler is that ‘–name’ can take a regular expression so you can run a group of tests (or even save time by not typing out the entire test name):

ruby test/unit/foo_test.rb --name=/test_bar/

OK, let’s review the results of my very non-scientific performance experiment:

* All unit tests via rake (110 tests) – 10.7s
* All tests in a single unit test file (7 tests) – 2.3s
* One test within a single unit test file – 1.9s.

What’s going on here? Even though I am running 1/110th of the tests, the tests only run about 5.6 times faster.

The problem with both Rake and Ruby is that it takes a long time to load the Ruby interpreter every time you run the tests. This is a small fraction of the overall time if you have lots of tests, but if you are just running a few tests, it really sucks. It would be nice if we could keep the Ruby interpreter in memory and just run the tests we need, so we turn to …

autotest

autotest is part of the awesome zentest suite. It’s easy to install:

gem install zentest

Now simply navigate to your project directory and run ./autotest

autotest will start by running all your tests and reporting any failures. Now for the sweet part – autotest will monitor and analyze both your application and test code and automatically run only those tests that validate code that has changed since you last saved. That’s just so freaking cool.

Not only does autotest speed up your tests by only running the ones that you need, it also runs the tests faster, since it doesn’t need to reload the Ruby interpreter over and over. Running the same single unit test in autotest took me about .1 seconds. Hot. In summary: if you’re not using autotest, you should.

I’ve only found two problems with autotest. First, sometimes the analysis algorithm gets confused and runs too few or too many tests (but this happens very, very rarely). More importantly, if you set a breakpoint in your code, autotest will just hit the breakpoint and wait – it won’t drop into an interactive Ruby prompt. To solve these (minor) problems, I wrote…

fast_test.rb

Using Rake tasks, I wrote my own test helper (with an admittedly terrible name). It’s the mutant offspring of the above methods (and it’s included at the bottom of this post, since it’s kind of long).

– Like autotest, it loads the Ruby interpreter once, so tests run faster
– Like the Ruby interpreter, you can provide a regular expression to run specific tests
– Like Rake, you can easily run just your unit, functional, or integration tests.
– Like Ruby or Rake, when you hit a breakpoint, you are dropped into an interactive environment

To use the script, place it somewhere in your application folder (I put it in ./script). Then, in your project directory, open (or create) .irbrc and add the following line

require 'script/fast_test.rb'

Save and close .irbrc and open up a rails console:

ruby script/console

Now you can do the following

test_all – Run all your tests
test_quick – Run just the unit and functional tests (not integration or other tests)
test_units – Run just your unit tests
test_functionals – Run just your functional tests

Again, you get the idea. Also, any of the above methods take an optional test name that is treated as a regular expression. So, to run a single unit test, you could do

test_units "test_name"

which, on my machine, took only .05 seconds. Now that’s how I roll.

Disclaimer: You’re free to use fast_test.rb, but I hereby declare that it’s current implementation may be terrible. Using rake tasks just seemed like the easiest way to accomplish what I needed and it’s worked pretty well for me for the past month or so. If you have any suggestions (on how to improve it or to suggest other, better written alternatives), I’d love to hear it.

Bringing it all together

So which is the best way to run your tests? It completely depends on your needs. I personally have one terminal window running autotest with another one running a rails console with fast_test.rb loaded. I generally let autotest do most of the work, but if I need to use the debugger or run a specific test of tests over and over, I use the console window. One warning: if autotest is running tests, don’t run tests in the console simultaneously – they’ll step all over each others’ toes and cause weird failures.

Like I said at the beginning, I’m a huge test nerd. So if you know of any other ways to improve your testing environment, let me know.

Oh, and before I go, here’s the code for fast_test.rb


### fast_test.rb ####################################
require 'rake/testtask'
def run_tests(sub_dir=nil,test_name=nil, long=false)
Rake::Task.clear
Rake::TestTask.new(:test) do |t|
t.libs << "test"
if(sub_dir.nil?)
t.pattern = 'test/**/*_test.rb'
else
t.pattern = "test/#{sub_dir}/*_test.rb"
puts t.pattern
end
t.verbose = true
t.options = "--verbose --name=/#{test_name}/" unless test_name.nil?
end
Rake::Task[:test].execute
"Done"
end
def test_units(pattern=nil)
run_tests("unit",pattern)
end
def test_functionals(pattern=nil)
run_tests("functional",pattern)
end
def test_integrations(pattern=nil)
run_tests("integration",pattern)
end
def test_quick(pattern=nil)
test_units(pattern)
test_functionals(pattern)
end
def test_all(pattern=nil)
run_tests(nil,pattern)
end
Advertisements

One Response to “Adventures in Testing, Part I : Running Your Tests”

  1. Dom^2 Says:

    Just a couple observations
    In your first observations regarding rake and ruby testing your rake test averaged 9.04ish tests per second with the ruby package test averaged 3 tests per second and the single ruby test .5 test per second. (Numbers are approximate).
    While reading this i can’t help but think a textbook from you two is not too far fetched of an idea.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: