Archive for the ‘Testing’ Category

Testing Wish List

October 6, 2007

We’ve recently been working on improving our test coverage on Seekler. As a result, I’ve been thinking about some testing tools that I’d really love to have but haven’t seen yet for Ruby.
I’d been writing these ideas down in the hopes that I might implement one over a free weekend. But since it doesn’t look like I’ll have too many free weekends in the near future, I figured I’d post them here. If anyone has seen tools like these – or wants to build them – that would be sweet.
Test Refactoring Guard
All code gets messy as it grows and changes. That includes test code (and yes, you should be keeping your test code clean and DRY, just like your production code).
It’s easy enough to clean up your production code – just simplify and refactor. Your test suite will let you do so without fear (or at least with less fear) of introducing bugs.
But how do you refactor your tests? If you refactor your production code, you’ll know you messed up if a test fails. But if you make a mistake when refactoring your tests, they may not fail. Rather, they may stop checking some important condition in your production code. Your tests will still pass but your test suite is weaker.
The Google Testing blog suggests the following strategy: 1. Intentionally break your production code so that tests fail. 2. Refactor your tests 3. Make sure the tests continue to fail (in the exact same way) 4. Revert your production code to a working state.
Huh? Let’s say I have a function f and three tests for it. I intentionally break f so that my three tests fail. Then I refactor a helper method that is used in all three tests. But what if this same helper method is used in other tests methods that don’t test f? Couldn’t my refactoring break those tests? This strategy assumes that I can correctly predict exactly which tests will be impacted by my refactoring. If I could do that, I wouldn’t need any special strategies or tools to help me.
I’d prefer to have ‘safe mode’ for testing that I could enter when refactoring my tests. The simplest version could just remember simple stats about my tests and make sure they stay constant. For instance, it could remember the number of tests, the names of those tests, the number of assertions, and the types of assertions (for each test). If any of these changed, I’d get a warning.
Or the tool could use code coverage to check my work. It could do a baseline run mapping each test to the code covered by that test. If the coverage changed, I’d get a scary warning letting me know that I’d done something wrong.
Bad Checkin Identifier
We all know the feeling – you notice that something in your app is broken and you’re positive it used to work. Somehow this bug has slipped by your tests and now you need to fix it.
One of the simplest and fastest ways to fix a bug is to find the source control checkin that caused the bug. But manually going through all the checkins is a tedious and time-consuming process.
Assuming you’ve now written a test (or set of tests) that catches the bug, it’d be easy to for a tool to automatically find which checkin caused the bug. You’d feed it a span of revisions and for each one, it’d pull down the code, run your new test, and determine if it passed or failed. Once you find that revision x passed the test and revision x+1 didn’t, you’ve found the culprit.
Of course, there will be some very old versions where the test will fail because the feature being tested wasn’t implemented at all. But there still should be some set of revisions in which the test will pass. The tool would just need to be clever enough to ignore the old failures and find the transition from passing to failure.
Distributed Unit Testing
Well-written unit tests should be independent from each other and they should run quickly – so you can run them a lot. Independent computations that need to run faster? This is a problem just screaming for parallel processing – e.g. distributing tests across computers on a network.
Unfortunately, I haven’t found a widely-used, easy Ruby implementation yet. The closest thing I’ve found is this, which I’m planning on trying out. If that doesn’t work, I might try implementing something on top of starfish, which looks pretty cool.

Like I said, I don’t know of any Ruby implementations for these ideas, but if you do, please let me know. Are there any other Ruby testing tools you’ve been wishing for? Let me know – I’ll add them to my list in case I ever find a free weekend.

Adventures in Testing, Part I : Running Your Tests

June 11, 2007

I’ll admit it – I’m a test nerd. As much as love the creative process of writing an app, I’m equally fascinated by the problem of verifying all that code.

Although my real passion is in writing dynamic testing systems, I’m also a big believer in static tests – AKA unit, functional, and integration tests. Luckily, Ruby and Rails make it super easy to write tests.

Writing good tests is obviously important, but there’s also a hidden art to running your tests. You’d think this would be pretty straightforward and, when you just have a few tests, it is. But as you write more and more tests, it takes longer and longer run them all. And the longer it takes to run your tests, the less useful they are to you.

There are two ways to speed up your tests. Obviously, one way is to increase the speed of running each test – for example, by writing more efficient tests (for instance, by using mocks) or by getting a faster machine.

The other way is to run fewer tests. Although this seems dangerous in theory, in practice it’s really useful. When writing some code, you generally know which tests are most likely to break, so it makes sense to run a subset of your tests. Similarly, if you’ve just caused some tests to fail, you’ll likely want to continue to run just those tests until they pass. Just make sure to run all your tests before moving on to your next task – and definitely run every test under the sun before you even think about checking code into source control.

With that in mind, let’s survey some of the ways you can run your tests:


The simplest way to run your tests in Rails is with Rake. Simply run one of the following commands in your project directory

rake – Run all your tests. Equivalent to running rake test
rake test:units – Run just your unit tests
rake test:functionals – Run just your functional tests.

Well, you get the idea. Run rake --tasks | grep test to see all the test tasks at your disposal. As you’ll see, Rake is awesome at running all your tests or important sets of tests.

The problem with Rake is that you don’t have much control over which tests you are running. As I mentioned earlier, you might just want to run one testcase or even just one single test. Which brings us to …


You can have finer grained control over your tests by invoking the Ruby interpreter directly. Let’s say you only want to run all the tests in one file. You can simply do:

ruby test/unit/foo_test.rb

Much better. On my machine, running all my unit tests via Rake took 10.7 seconds, whereas running just one specific test file took just 2.3 seconds.

But the fun doesn’t stop there. Oh no. Let’s try running just one specific test.

ruby test/unit/foo_test.rb --name=test_bar__some_test

Down to 1.9 seconds. What’s even cooler is that ‘–name’ can take a regular expression so you can run a group of tests (or even save time by not typing out the entire test name):

ruby test/unit/foo_test.rb --name=/test_bar/

OK, let’s review the results of my very non-scientific performance experiment:

* All unit tests via rake (110 tests) – 10.7s
* All tests in a single unit test file (7 tests) – 2.3s
* One test within a single unit test file – 1.9s.

What’s going on here? Even though I am running 1/110th of the tests, the tests only run about 5.6 times faster.

The problem with both Rake and Ruby is that it takes a long time to load the Ruby interpreter every time you run the tests. This is a small fraction of the overall time if you have lots of tests, but if you are just running a few tests, it really sucks. It would be nice if we could keep the Ruby interpreter in memory and just run the tests we need, so we turn to …


autotest is part of the awesome zentest suite. It’s easy to install:

gem install zentest

Now simply navigate to your project directory and run ./autotest

autotest will start by running all your tests and reporting any failures. Now for the sweet part – autotest will monitor and analyze both your application and test code and automatically run only those tests that validate code that has changed since you last saved. That’s just so freaking cool.

Not only does autotest speed up your tests by only running the ones that you need, it also runs the tests faster, since it doesn’t need to reload the Ruby interpreter over and over. Running the same single unit test in autotest took me about .1 seconds. Hot. In summary: if you’re not using autotest, you should.

I’ve only found two problems with autotest. First, sometimes the analysis algorithm gets confused and runs too few or too many tests (but this happens very, very rarely). More importantly, if you set a breakpoint in your code, autotest will just hit the breakpoint and wait – it won’t drop into an interactive Ruby prompt. To solve these (minor) problems, I wrote…


Using Rake tasks, I wrote my own test helper (with an admittedly terrible name). It’s the mutant offspring of the above methods (and it’s included at the bottom of this post, since it’s kind of long).

– Like autotest, it loads the Ruby interpreter once, so tests run faster
– Like the Ruby interpreter, you can provide a regular expression to run specific tests
– Like Rake, you can easily run just your unit, functional, or integration tests.
– Like Ruby or Rake, when you hit a breakpoint, you are dropped into an interactive environment

To use the script, place it somewhere in your application folder (I put it in ./script). Then, in your project directory, open (or create) .irbrc and add the following line

require 'script/fast_test.rb'

Save and close .irbrc and open up a rails console:

ruby script/console

Now you can do the following

test_all – Run all your tests
test_quick – Run just the unit and functional tests (not integration or other tests)
test_units – Run just your unit tests
test_functionals – Run just your functional tests

Again, you get the idea. Also, any of the above methods take an optional test name that is treated as a regular expression. So, to run a single unit test, you could do

test_units "test_name"

which, on my machine, took only .05 seconds. Now that’s how I roll.

Disclaimer: You’re free to use fast_test.rb, but I hereby declare that it’s current implementation may be terrible. Using rake tasks just seemed like the easiest way to accomplish what I needed and it’s worked pretty well for me for the past month or so. If you have any suggestions (on how to improve it or to suggest other, better written alternatives), I’d love to hear it.

Bringing it all together

So which is the best way to run your tests? It completely depends on your needs. I personally have one terminal window running autotest with another one running a rails console with fast_test.rb loaded. I generally let autotest do most of the work, but if I need to use the debugger or run a specific test of tests over and over, I use the console window. One warning: if autotest is running tests, don’t run tests in the console simultaneously – they’ll step all over each others’ toes and cause weird failures.

Like I said at the beginning, I’m a huge test nerd. So if you know of any other ways to improve your testing environment, let me know.

Oh, and before I go, here’s the code for fast_test.rb

### fast_test.rb ####################################
require 'rake/testtask'
def run_tests(sub_dir=nil,test_name=nil, long=false)
Rake::Task.clear do |t|
t.libs << "test"
t.pattern = 'test/**/*_test.rb'
t.pattern = "test/#{sub_dir}/*_test.rb"
puts t.pattern
t.verbose = true
t.options = "--verbose --name=/#{test_name}/" unless test_name.nil?
def test_units(pattern=nil)
def test_functionals(pattern=nil)
def test_integrations(pattern=nil)
def test_quick(pattern=nil)
def test_all(pattern=nil)