Automated Testing -- It's the thought that counts

After encountering a painfully embarrassing bug in a production release of our product, I decided to make it personal goal to increase our automated testing. Like many start-ups our focus was on building features and value into our product. While we certainly value testing and automated tests specifically, we always found an excuse to under-invest in this area. The bottom line is that automated testing is hard and requires a tremendous investment....

But wait, is it really a "tremendous" investment or is it simply the thought of it that's "tremendous?" There are always a million excuses why automated tests should be delayed. Including:

• There really isn't a framework that applies to our unique app.
• It will take more time to write the test than the code itself.
• Isn't that what QA is for?
• I can either write a test or deliver it on time.

Honestly the excuse doesn't matter. There always seems to be one. You can solve all the excuses and magically new ones appear.

My solution was the personally attack the problem. Lead by example. Absorb the pain and measure the results.

The Results

In one 10 day Sprint, I managed to develop a new testing framework highlighted by our own Domain Specific Language (DSL) and write automated tests covering basic product functions like creating teams, users, etc. Over the next few blog posts I'll elaborate on the results including the gory technical details.

In this post, I want to focus on the "pain" and "tremendous" sacrifice that I must have endured to get there. So I did cheat and reuse an existing library we had we some infrastructure for sending data, running queries, and running reports. But given that here are the stats for the effort:

Component	Time in Active Hours
Existing Library	137
New DSL and framework	23
TOTAL	160
TEAM WEEKLY AVG	220

Our total investment in this area was less than one week of total active time for our team. This total investment is all time on these artifacts. Hm. Not a "tremendous" investment. Now this does not include any tests that were written using the framework. Here are the figures for the tests:

Number of Tests	18
Average Time to Write a Test	< 1 hr
Total Time on Tests	~ 20 hrs

1 measly hour per test. That's it. That also doesn't seem so painful.

Finally I looked at all of our time spent on specific modules of our product over the last 75 days ( beginning in January ), and our total investment in all testing modules was only 10% of the overall time.

Tremendous is 10%

So according to real quantitative data, the tremendous pain of automated testing 10%. It's less than an hour per day. The next time a developer ( or manager ) is reluctant to invest in automated testing due to the pain inform them that only the thought is painful -- the reality is only 10%. Of course, I'd encourage to measure it like we did to prove it, and if the investment grows above 10% you can always reprioritize.