Creating automated tests can be very difficult, especially when the code has gotten long in the tooth and was not created with automated tests to begin with. Many product development teams don’t invest in automated tests. They think they cannot afford them. They think their product is different and can’t be manually tested. This thinking is flawed.
Back in the products younger days, manual test was not too time consuming. But slowly that changed. The system grows, the manual test effort grows. Eventually, it seems that no amount of manual test effort finds all the problems.
In this article I show a simple model that illustrates why manual test is unsustainable and that a sustainable software product development effort must include considerable test automation.
For starters, let’s look at a simple model that illustrates a development process that relies mostly manual tests. Let’s make an assumption, the assumption depicted in this graph:
Explaining the graph, if we spent 10 units of development effort to create a set of new features, let’s say that it only takes 5 units of effort to test the new functionality. Said another way, test effort is proportional to development effort by some constant factor.
Wait a second; we know it’s not that simple. The coefficient is certainly wrong, and to think that all development effort is equal is wrong, but its a simple model the can help us explore the folly of manual testing. In addition, the simple model is implicit in the workings of most companies.
Companies tend to have fixed resources, I mean people, building and testing their products. There’s the developers and the testers. Or maybe there are developers that switch hats and spend some of their time testing. Either way, there is a de facto develop:test coefficient in practice at your workplace.
What’s the problem?! Everyone knows that software, once you make it work, stays working without any further effort. [Huh, what did he say?] Please don’t quote me out of context, I’m just being sarcastic. Let’s pull a couple of bits of wisdom from The Systems Bible to make you more afraid of changing code that is not covered by automated tests.
The Systems Bible says “If a system is working, leave it alone.” “Systmes don’t like being fiddled with.” Well put John Gall. I guess all us programmers are out of work. Our business is to change and evolve software. We need to find a way to accept the warnings from The Systems Bible, and continue to advance our systems.
RB Grady says that 25% of the bugs in your bug list are the consequence of someone adding a new feature or fixing a bug, while not realizing that they broke some other seemingly random thing. So, we can’t just assume that prior iterations’ features are going to work; we have to retest them. Let’s refine the model.
Every iteration, not only does the new functionality have to be tested, but the prior iterations’ working features have to be retested too. No problem, just add some more resources; I mean plug compatible testing units; I mean people with the needed skills and knowledge.
The reality of the situation is that we don’t have unlimited budgets and if we did, you could not hire people, with the needed skills and knowledge, at a fast enough rate to keep the product completely tested. And that means that we have to face the reality of the ever growing untested code gap.
The untested code gap kills productivity and predictability. We don’t choose where the bugs go making them easy to find; they choose their hiding places, and have no mercy on us. They flare up and put teams into fire fighting mode.
Are we are beat? Do we have to put up with buggy products and long test and fix cycles? No, we do not! We must accept that a product test strategy based on manual test is unsustainable. Unsustainable systems eventually collapse. We can’t afford not to automate much of your software test.
©2013 James Grenning's Blog. All Rights Reserved.
.
