Posts Tagged ‘qa’

A short story about the value of automated testing.

Saturday, July 9th, 2011

For the last few weeks, my team has been working on some pretty fundamental changes for Nokia maps on the web.

But that’s the short term. For the last six months my team has been working hard on getting a suite of automated tests up and running, and I’m proud to say we now have close to 1000 Jasmine unit tests and 200 cucumber acceptance scenarios running on every commit.

So I was delighted, when I heard that this week, one of the new changes had broken 52 of our acceptance tests.

Why should I be delighted?

Doesn’t that mean the team now has to go away and fix 52 acceptance tests? Well, yes, and that was certainly the way the team saw it, until I said:

“Now imagine we didn’t have those tests. What would have happened?”

And then we all started to realise, the power of automated testing.

So if we didn’t have automated testing, what would have actually happened?

Well, someone would have gone into the code. Found the piece of code that needed changing. And changed it. Probably tested it in the one place they knew about. Committed it. And moved on. Completely oblivious to the fact that they’d broken the application in 51 other different places.

Then the build would have gone to QA, the QA’s would have found the 52 different breakages, raised 52 new bugs, and sent them back to the developers.

Then the developers would have gone into the code, tried to fix those 52 bugs, and probably caused another 50 regressions. Which would then go to the QA team, be raised and go back to the developers for fixing. Eventually this hugely inefficient cycle would have continued until the build gets to an extremely brittle stability, a bit like building a house of cards, and it would have gone live.

However, with the tests, those bugs were caught. Immediately. And fixed. Immediately. Those 52 broken acceptance tests, saved us potentially months and months of rework.

So the moral of the story is, when you reduce the feedback cycle of a bug, with the saftey net of automated tests, it allows you to develop in confidence, giving you warning to what you’ve broken and giving you the opportunity to fix it then and there (however having the discipline to do this is another story). In Lean, this is referred to as ‘building integrity and quality into the product from the beginning of the cycle’, rather than trying to add it in at the end with a torturous QA cycle.