Testing Wish List


We’ve recently been working on improving our test coverage on Seekler. As a result, I’ve been thinking about some testing tools that I’d really love to have but haven’t seen yet for Ruby.
I’d been writing these ideas down in the hopes that I might implement one over a free weekend. But since it doesn’t look like I’ll have too many free weekends in the near future, I figured I’d post them here. If anyone has seen tools like these – or wants to build them – that would be sweet.
Test Refactoring Guard
All code gets messy as it grows and changes. That includes test code (and yes, you should be keeping your test code clean and DRY, just like your production code).
It’s easy enough to clean up your production code – just simplify and refactor. Your test suite will let you do so without fear (or at least with less fear) of introducing bugs.
But how do you refactor your tests? If you refactor your production code, you’ll know you messed up if a test fails. But if you make a mistake when refactoring your tests, they may not fail. Rather, they may stop checking some important condition in your production code. Your tests will still pass but your test suite is weaker.
The Google Testing blog suggests the following strategy: 1. Intentionally break your production code so that tests fail. 2. Refactor your tests 3. Make sure the tests continue to fail (in the exact same way) 4. Revert your production code to a working state.
Huh? Let’s say I have a function f and three tests for it. I intentionally break f so that my three tests fail. Then I refactor a helper method that is used in all three tests. But what if this same helper method is used in other tests methods that don’t test f? Couldn’t my refactoring break those tests? This strategy assumes that I can correctly predict exactly which tests will be impacted by my refactoring. If I could do that, I wouldn’t need any special strategies or tools to help me.
I’d prefer to have ‘safe mode’ for testing that I could enter when refactoring my tests. The simplest version could just remember simple stats about my tests and make sure they stay constant. For instance, it could remember the number of tests, the names of those tests, the number of assertions, and the types of assertions (for each test). If any of these changed, I’d get a warning.
Or the tool could use code coverage to check my work. It could do a baseline run mapping each test to the code covered by that test. If the coverage changed, I’d get a scary warning letting me know that I’d done something wrong.
Bad Checkin Identifier
We all know the feeling – you notice that something in your app is broken and you’re positive it used to work. Somehow this bug has slipped by your tests and now you need to fix it.
One of the simplest and fastest ways to fix a bug is to find the source control checkin that caused the bug. But manually going through all the checkins is a tedious and time-consuming process.
Assuming you’ve now written a test (or set of tests) that catches the bug, it’d be easy to for a tool to automatically find which checkin caused the bug. You’d feed it a span of revisions and for each one, it’d pull down the code, run your new test, and determine if it passed or failed. Once you find that revision x passed the test and revision x+1 didn’t, you’ve found the culprit.
Of course, there will be some very old versions where the test will fail because the feature being tested wasn’t implemented at all. But there still should be some set of revisions in which the test will pass. The tool would just need to be clever enough to ignore the old failures and find the transition from passing to failure.
Distributed Unit Testing
Well-written unit tests should be independent from each other and they should run quickly – so you can run them a lot. Independent computations that need to run faster? This is a problem just screaming for parallel processing – e.g. distributing tests across computers on a network.
Unfortunately, I haven’t found a widely-used, easy Ruby implementation yet. The closest thing I’ve found is this, which I’m planning on trying out. If that doesn’t work, I might try implementing something on top of starfish, which looks pretty cool.

Like I said, I don’t know of any Ruby implementations for these ideas, but if you do, please let me know. Are there any other Ruby testing tools you’ve been wishing for? Let me know – I’ll add them to my list in case I ever find a free weekend.


4 Responses to “Testing Wish List”

  1. Daniel Franke Says:

    Darcs has a bad checkin identification feature. You supply a test script as an argument, and it’ll pass it each revision in reverse chronological order and tell you the first one that passes the test. I’ve never used it myself, but I’m pretty sure that one day it’s going to save me a hell of a lot of time.

  2. Ben Says:

    Cool, thanks for the tip. We’ve been considering moving from Subversion to a distributed source control system for awhile now. This is definitely another good argument for Darcs.

  3. Wouter Lievens Says:

    Neat ideas. As for the revision-seeking application, it can be done with a binary search so even in case it has to search over 1000 revisions, it will have found the bug in (I think) at most 11 test runs. Or is that just plain wrong?
    Also: read up on mutation testing. It’s a neat unconventional technique, it may not apply everywhere but the idea is sound.

  4. Ben Says:

    Wouter: Yep, you’re right, binary search would be the way to go. That’s how I’ve been going through revisions when I’ve done this manually.
    Mutation testing is very, very cool. The only tool I’ve used is Heckle
    Have you found any other mutation testing tools you like?
    Another very cool tool is RushCheck (http://rushcheck.rubyforge.org/). It’s a random testing tool for Ruby that is based on QuickCheck from Haskell. I’ve played around with it a little bit and it’s awesome. I hope to write a post about my experiences with it soon.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: