Friday, December 21, 2007

Frequent Releases Improve Code Quality Faster

There’s a saying that “you can’t test quality into a product.” That may be true, but it is very difficult to assess the level of quality of a product without testing it. I find that one of the problems with quality is the way that it is talked about. There are all different categories of testing such as regression, black-box, white-box, exploratory, automated, manual, etc. Each of these categories serves a useful purpose, but what is the overall purpose of testing?

Some people argue that testing is how you verify that the product works. Other people argue that the purpose of testing is to uncover what does not work. My view is that the purpose of testing is to show that the software works. By “works” I include that it correctly handles bad input, bad data, and diabolical usage. If you try to use the product in a way that was not intended and it crashes or gives no result at all, then you have found a way in which it does not work.

Quality Assurance and testing are not the same thing. You can implement all of the QA you want: good process, code reviews, coding standards, coding for testability, and whatever else you want to assure quality short of actual testing, but without testing, you have no idea how the product will actually perform. Let’s say you produce a product with absolutely no test plan or you have a great test plan but you don't execute it. I suppose that it is possible that the quality that the customer experiences will be exactly the same as if you had a test plan and executed it. The difference is that your knowledge of the quality of the product will come from your customer after you have already shipped your product. The purpose of testing is to derive knowledge about the quality of the product prior to shipping it so that you can make good decisions about what to do now that you have that knowledge.

So, testing itself does not produce higher quality. It only produces knowledge of current quality. There are only two things that produce higher quality: changes to the process that you use to create the product itself (not including testing), and the responses taken based on the knowledge gained from testing. Testing can influence quality, but it does not directly affect it.

Product quality is an ongoing effort. No matter how much testing you do, users will find ways to use your product that find holes in your test plan. For any change, whether it is a bug fix or a new feature, you don't really know the impact until customers start using it. That means that the maturation of a change doesn't really start until the change ships. If you have a long release cycle, you will have changes that are made early that are basically gathering dust until you ship. You can't really say that they are tested until you test the whole product as it will ship, so again, true maturation doesn't start until you ship.

The moral of this story? Release as often as you reasonably can; which may be once a week, once a month, or once a year, but always strive to release often.

Next: The Role of QA in an Agile Project


Isaac Rodriguez said...

I agree with you in some of the points you expose, but I disagree in the overall idea. You make very good points when you say, "Testing itself does not produce quality. There are only two things that produce quality: the process that you use to create the product itself (not including testing), and the responses taken based on the knowledge gained from testing." Therefore, you want to gather information from testing as soon as possible, and the quickest way is to test.

Of course, you cannot possibly think of every scenario your customers are going to run into (or I should say, it is not likely you will be able to define every possible scenario). Under this assumption, releasing often will help you to obtain feedback from your actual customers very soon, which will allow you to deliver fixes to them very quickly. This will increase your software quality, but not from the fact that you release often, but the fact that your customers are testing the scenarios you have not thought about.

You said, "Let’s say you produce a product with absolutely no test plan or you have a great test plan but you don't execute it. There is a chance that the quality that the customer experiences will be exactly the same as if you had a test plan and executed it." Although this is statistically possible, you will have to agree that the actual chances of something like that happening are very slim. If you don't have a test plan, or you have one and don't execute it, it is more likely your final product release will have lower quality than if you executed the plan (no matter how bad the plan was) because you won't have any feedback until your customers actually uses the product. I am sure with your experience you are not trying to convince us that we should release our products without any actual testing. You probably know as well as I do, that it is not a good idea to release untested software, especially if you want to keep your customers around.

The moral of this story and where I disagree with you is that releasing often does not increase quality. Good processes and testing often increase quality. Good processes will allow you to make less mistakes. Testing often will help you to find the little mistakes you make, because we all make mistakes, quicker. With that information, you can act and fix the issues found. Releasing often will help you to gain feedback from your cutomers and act upon what they report, but releasing often will not help you if your often released software is not used; therefore, releasing often does not improve quality; testing often does.

Damon Poole said...

Isaac, thsnks for your thoughtful post. Let's say that you could release every month but instead you release every year. Let's also say that you go above and beyond on your testing efforts. You've got very high test coverage numbers and you are using decision based coverage instead of line coverage. It has been a long time since anybody found a bug, and you've fixed every bug you know about. Lastly, let's say that you developed in 30-day iterations and at the end of every iteration all functionality introduced during that iteration had all of its tests written during that iteration instead of at the end of the one year cycle.

Now at the end of this year you release your product. I guarantee that your customers will find problems. Let's say for the sake of simplicity that they find exactly 12 and each one is linked to functionality introduced in one of the 12 iterations. You waited 11 months to find the issue introduced in the first iteration that your customer would have found right away.

I'm not saying that you should rely on your customers to be part of your testing department. I'm only saying that despite your best efforts, it is inevitable that there will be issues that you only find after you release, so keep on testing, don't stop that. But release as often as you can.

Also in this (contrived) example, your customers were still exposed to the same number of bugs, just not all at the same time.

You are right that I am not suggesting that you forgo testing. My point about possibly having high quality without executing a test plan was that while possible, it definitely is highly unlikely. The point of doing the testing is to gain knowledge about the quality prior to release so that you can then act on that knowledge.

After reading your comment and writing this response I would say that a better moral of the story is "In addition to keeping your testing standards high, it is better to find problems that you are likely to only find by releasing to customers as soon as you possibly can. Therefore, release as often as you can." A bit more wordy, but perhaps a better representation of my point.

Soon Hui said...

Good post, frequent releases are generally good for developers.

But customers may not want it because every download comes at a price and risks.

todsshoesstores said...

requent grew for acquiring a strike with grownup males and Onitsuka Tiger Online Store females instantaneously receiving a last final accomplish result of really effective advertising and advertising campaigns undertaken by grownup males and females referred to as Weyden and Kennedy.Hogan Men shoes Of course, this assortment of shoes was named swiftly good ideal after the well-known basketball participant Michael Hogan Men shoes Jordan and was bound to turn in to some strike inside the really bare lowest among the basketball fans, initially. The Jordan sequence of shoes has also been produced for women, which look about in actuality captivating designs. Franklin & Marshall The shoes are obtainable in an exceptionally amount of colors; blue, white, red, black, pink, green, etc. in actuality sporty,Abercrombie Fitch store they get ladies dressed up for that superb game. In add-on they look about in female sizes to fixture the two the tiny footed along utilizing the huge footed.