One of the key things we’re trying to get with Agile is faster and more effective feedback so that we can make better decisions. This extends into to the technical practices just as much as it does in our meetings and processes.
When I arrived at one client, I discovered that they already had 50,000 automated tests in place, which sounds impressive when you first hear the number, but that’s misleading.
Those 50,000 tests were written in such a way that the entire suite took three full days to run. As you might imagine, being that slow, they didn’t get run very often. Developers would run the subset of tests that they thought covered the part of the code they were working on, but months might pass before someone would run the entire suite. Note that they were deploying something to production at least weekly.
This is an example of a feedback cycle that was ineffective. Although they did have what appeared to be a comprehensive test suite, it was so slow that it wasn’t useful.
The quality practices that we introduce, like TDD, refactoring, and continuous integration, rely on the ability to be able to run tests quickly and get feedback immediately. We should be able to run thousands of tests in a second to see if we’ve made a mistake or to determine what the current behaviour is.
As for the client with the three day test suite, by the time I left, almost all of those tests had been thrown away and replaced. What had made them so slow in the first place was that every one of them had been written to test through the UI, whether or not that made sense. We replaced as many as we could with tiny micro-tests that were just code calling code, and not entering through the UI.
When we were done, the majority of the new tests could be run in seconds by the developers on their local machine, with a small set of slower tests that would be run by the build server. Even those slower ones would be run multiple times per day, to provide feedback in a timely fashion.
Coming with that improved feedback, was a corresponding improvement in quality.
The speed of the feedback loop is critical. If we can’t get rapid and useful feedback then our ability to make effective decisions is impaired. I’ve focused on automated tests here but it’s applicable to every feedback loop we have.
See also: Instructions from Jon Reid on how to make your Xcode tests run faster.