Comments on Agile Development Thoughts: Frequent Releases Improve Code Quality Faster

Good post, frequent releases are generally good fo...

2007-12-23T09:22:00.000-05:00

Good post, frequent releases are generally good for developers.

But customers may not want it because every download comes at a price and risks.

Isaac, thsnks for your thoughtful post. Let's say ...

2007-12-22T01:48:00.000-05:00

Isaac, thsnks for your thoughtful post. Let's say that you could release every month but instead you release every year. Let's also say that you go above and beyond on your testing efforts. You've got very high test coverage numbers and you are using decision based coverage instead of line coverage. It has been a long time since anybody found a bug, and you've fixed every bug you know about. Lastly, let's say that you developed in 30-day iterations and at the end of every iteration all functionality introduced during that iteration had all of its tests written during that iteration instead of at the end of the one year cycle.

Now at the end of this year you release your product. I guarantee that your customers will find problems. Let's say for the sake of simplicity that they find exactly 12 and each one is linked to functionality introduced in one of the 12 iterations. You waited 11 months to find the issue introduced in the first iteration that your customer would have found right away.

I'm not saying that you should rely on your customers to be part of your testing department. I'm only saying that despite your best efforts, it is inevitable that there will be issues that you only find after you release, so keep on testing, don't stop that. But release as often as you can.

Also in this (contrived) example, your customers were still exposed to the same number of bugs, just not all at the same time.

You are right that I am not suggesting that you forgo testing. My point about possibly having high quality without executing a test plan was that while possible, it definitely is highly unlikely. The point of doing the testing is to gain knowledge about the quality prior to release so that you can then act on that knowledge.

After reading your comment and writing this response I would say that a better moral of the story is "In addition to keeping your testing standards high, it is better to find problems that you are likely to only find by releasing to customers as soon as you possibly can. Therefore, release as often as you can." A bit more wordy, but perhaps a better representation of my point.

I agree with you in some of the points you expose,...

2007-12-21T23:35:00.000-05:00

I agree with you in some of the points you expose, but I disagree in the overall idea. You make very good points when you say, "Testing itself does not produce quality. There are only two things that produce quality: the process that you use to create the product itself (not including testing), and the responses taken based on the knowledge gained from testing." Therefore, you want to gather information from testing as soon as possible, and the quickest way is to test.

Of course, you cannot possibly think of every scenario your customers are going to run into (or I should say, it is not likely you will be able to define every possible scenario). Under this assumption, releasing often will help you to obtain feedback from your actual customers very soon, which will allow you to deliver fixes to them very quickly. This will increase your software quality, but not from the fact that you release often, but the fact that your customers are testing the scenarios you have not thought about.

You said, "Let’s say you produce a product with absolutely no test plan or you have a great test plan but you don't execute it. There is a chance that the quality that the customer experiences will be exactly the same as if you had a test plan and executed it." Although this is statistically possible, you will have to agree that the actual chances of something like that happening are very slim. If you don't have a test plan, or you have one and don't execute it, it is more likely your final product release will have lower quality than if you executed the plan (no matter how bad the plan was) because you won't have any feedback until your customers actually uses the product. I am sure with your experience you are not trying to convince us that we should release our products without any actual testing. You probably know as well as I do, that it is not a good idea to release untested software, especially if you want to keep your customers around.

The moral of this story and where I disagree with you is that releasing often does not increase quality. Good processes and testing often increase quality. Good processes will allow you to make less mistakes. Testing often will help you to find the little mistakes you make, because we all make mistakes, quicker. With that information, you can act and fix the issues found. Releasing often will help you to gain feedback from your cutomers and act upon what they report, but releasing often will not help you if your often released software is not used; therefore, releasing often does not improve quality; testing often does.