We’ve recently been working on improving our test coverage on Seekler. As a result, I’ve been thinking about some testing tools that I’d really love to have but haven’t seen yet for Ruby.
I’d been writing these ideas down in the hopes that I might implement one over a free weekend. But since it doesn’t look like I’ll have too many free weekends in the near future, I figured I’d post them here. If anyone has seen tools like these – or wants to build them – that would be sweet.
Test Refactoring Guard
All code gets messy as it grows and changes. That includes test code (and yes, you should be keeping your test code clean and DRY, just like your production code).
It’s easy enough to clean up your production code – just simplify and refactor. Your test suite will let you do so without fear (or at least with less fear) of introducing bugs.
But how do you refactor your tests? If you refactor your production code, you’ll know you messed up if a test fails. But if you make a mistake when refactoring your tests, they may not fail. Rather, they may stop checking some important condition in your production code. Your tests will still pass but your test suite is weaker.
The Google Testing blog suggests the following strategy: 1. Intentionally break your production code so that tests fail. 2. Refactor your tests 3. Make sure the tests continue to fail (in the exact same way) 4. Revert your production code to a working state.
Huh? Let’s say I have a function
f and three tests for it. I intentionally break
f so that my three tests fail. Then I refactor a helper method that is used in all three tests. But what if this same helper method is used in other tests methods that don’t test
f? Couldn’t my refactoring break those tests? This strategy assumes that I can correctly predict exactly which tests will be impacted by my refactoring. If I could do that, I wouldn’t need any special strategies or tools to help me.
I’d prefer to have ‘safe mode’ for testing that I could enter when refactoring my tests. The simplest version could just remember simple stats about my tests and make sure they stay constant. For instance, it could remember the number of tests, the names of those tests, the number of assertions, and the types of assertions (for each test). If any of these changed, I’d get a warning.
Or the tool could use code coverage to check my work. It could do a baseline run mapping each test to the code covered by that test. If the coverage changed, I’d get a scary warning letting me know that I’d done something wrong.
Bad Checkin Identifier
We all know the feeling – you notice that something in your app is broken and you’re positive it used to work. Somehow this bug has slipped by your tests and now you need to fix it.
One of the simplest and fastest ways to fix a bug is to find the source control checkin that caused the bug. But manually going through all the checkins is a tedious and time-consuming process.
Assuming you’ve now written a test (or set of tests) that catches the bug, it’d be easy to for a tool to automatically find which checkin caused the bug. You’d feed it a span of revisions and for each one, it’d pull down the code, run your new test, and determine if it passed or failed. Once you find that revision x passed the test and revision x+1 didn’t, you’ve found the culprit.
Of course, there will be some very old versions where the test will fail because the feature being tested wasn’t implemented at all. But there still should be some set of revisions in which the test will pass. The tool would just need to be clever enough to ignore the old failures and find the transition from passing to failure.
Distributed Unit Testing
Well-written unit tests should be independent from each other and they should run quickly – so you can run them a lot. Independent computations that need to run faster? This is a problem just screaming for parallel processing – e.g. distributing tests across computers on a network.
Unfortunately, I haven’t found a widely-used, easy Ruby implementation yet. The closest thing I’ve found is this, which I’m planning on trying out. If that doesn’t work, I might try implementing something on top of starfish, which looks pretty cool.
Like I said, I don’t know of any Ruby implementations for these ideas, but if you do, please let me know. Are there any other Ruby testing tools you’ve been wishing for? Let me know – I’ll add them to my list in case I ever find a free weekend.