First, many thanks to Isaac Rodriguez whose comments on my previous post elicited a comment from me which I have re-written as a post here. I've also updated my previous post a bit to better state the original point.
Let's say that you develop your software in 30-day iterations. At the end of every iteration all functionality introduced during that iteration had all of its tests written during that iteration instead of at the end of the one year cycle. All of those tests pass. You take testing very seriously; you've got very high test coverage numbers, you are using decision based coverage instead of line coverage, and you fix every bug you find.
Let's say that you could release every month but instead you choose to release every year. Now at the end of this year you release your product. I guarantee that your customers will find problems. Let's say for the sake of simplicity that they find exactly 12 and each one is linked to functionality introduced in one of the 12 iterations. This means that for the bug in functionality in iteration 1, you waited 11 months to find the issue that your customer would have found right away.
I'm not saying that you should forgo testing or rely on your customers to be part of your QA department. I'm only saying that despite your best efforts, it is inevitable that there will be issues that you only find after you release, so keep on testing, don't stop that. But release as often as you can.
Also in this (contrived) example, your customers would still be exposed to the same number of bugs, just not all at the same time.
Moral to this story: "In addition to keeping your testing standards high, it is better to find problems that you are likely to only find by releasing to customers as soon as you possibly can. Therefore, release as often as you can."
Next: Customers Don't Want Frequent Releases
9 comments:
Thanks for this post/reply. Do not get me wrong, I do agree that is a good idea to release often. There are many advantages to release your product often, but the truth is it is not always possible to have weekly/monthly releases.
I work for a company that releases commercial software in a year release cycle. We release about a couple of service packs betweeen releases where we can only provide fixes. New functionality is not allowed in service pack releases because it affects the bottom-line; therefore, we cannot release more often because all the time we spend fixing defects (which me and many believe is critical we do) we are not adding features to the product.
This is by no means an ideal process to work with; therefore, we have to get creative, and we do.
I agree you can improve quality by releasing often because you will receive feedback sooner, but I also understand that it is not always possible to have small release cycles.
I'm only saying that despite your best efforts, it is inevitable that there will be issues that you only find after you release, so keep on testing, don't stop that.
Absolutely! Thank you for stating what we all know, but fail to admit publicly.
I had a mentor who said "Take the 10 engineers in this room right now. Have them all write "Hello World" right now. At least 8 of the 10 will have problems."
Just pointing out that our profession is not the same as licking stamps. Interfaces, tools, networks, datatypes and developers conspire to introduce defects. It is a fact. And no amount of stomping around saying "that's a defeatist attitude!!!" is going to change that fact.
My current frustration stems from the actions of my boss. He's also the owner of the (small and getting smaller) company. He's not technical. Yet he insists on doing ZERO testing (outside of what I do as a developer -- because we're all so good at objectively testing our own code, right? :-) ) and then shipping the product -- LITERALLY - the day I stop developing. And then this ______ will stomp around like Donkey Kong the moment a bug is found by a customer in the field.
Yeah I know, I know, I need a new job. But I wanted to chime in regarding my experience with bugs -- they're in there, it's just a question of when -- and under what circumstances -- you find them.
On one hand it is certainly good to fix problems as soon as you can but...
As a customer, I would WAY prefer the annual release. If we have 12 release with 12 bugs, do you patch each release to fix the bug? If so, there are really 24 releases, and that would annoy me massively. If you would patch in the next iteration, then there would be no point during the year when the system is working properly. Also a bummer...
Melbournian,
You bring up a good point! Releasing frequently is a tool. It won't make sense in all situations.
For any practice, there will be corner cases and how you handle them will depend on your exact circumstances. As with traditional development, people have to make judgment calls. Those twelve bugs might all be trivial and while reported and good to know about, nobody cares that they have to wait until the next release to get it fixed. On the other hand, one of them might be quite serious and require an immediate patch.
Here's a question for you. What is the connection between the frequency of release and how often you upgrade? You seem to imply that it is 1-1. Why? Isn't it more likely that you will upgrade when you have a good reason to do so and that might still only be once a year?
See also my latest post, Tuning the Frequency of Your Releases which is my most comprehensive post on release frequency to date.
"Here's a question for you. What is the connection between the frequency of release and how often you upgrade? You seem to imply that it is 1-1. Why? Isn't it more likely that you will upgrade when you have a good reason to do so and that might still only be once a year?"
It depends on the situation...
I would either grab the patch releases OR any compelling features, so not 1-1.
Lets assume I am on the 2007.3 release... If there is a bug, I may have to grab 2007.4 to get that fixed. But if there is a bug in that, I may have to grab 2007.5 to get THAT fixed, and onto the merry train I jump. By 2007.6 I hate your product!
Alternatively, the Stable/Unstable branch model (you mentioned in the other article) is fine... In practice, given a choice, I will always stay on the stable branch. Unstable is a sort of defacto beta-testing platform. If you take this approach, there is a danger that a potential customer could end up on the unstable branch and ALSO end up hating your product (since they have come to rely on features that are no longer in the stable branch, they can't move back). Often, unstable becomes the realm of the much more tech-savvy users (which is fine).
I guess my only warning is that you need to try and think like one of your customers, and remember that they are the people you are trying to keep happy.
Melbournian,
I made a mistake in calling it "conservative" and "latest and greatest." I was forgetting that the connotation of "latest and greatest" is that it is untested and that is not what I meant at all. All of the releases would be stable, otherwise, you wouldn't have released them. The frequent releases are not like traditional patches, they are like traditional major releases, except their contents is smaller. They are all well tested, stable, and of high quality.
If an organization can't do this, then it should not change its frequency of releases.
I will update the other article to reflect this.
Regarding your comment about grabbing 2007.4, 2007.5 etc, you run that same risk with a traditional major release, and in fact the likelihood is much higher for this with a traditional process.
Your point is well taken regarding customers which is exactly why I advocate Agile development. And by that I do not mean frequent releases with poor quality but rather frequent releases with higher quality than customers are used to from the traditional process and major releases.
Thanks for keeping me honest, I look forward to your future comments.
Hi Damon,
"Regarding your comment about grabbing 2007.4, 2007.5 etc, you run that same risk with a traditional major release, and in fact the likelihood is much higher for this with a traditional process."
and
"I was forgetting that the connotation of "latest and greatest" is that it is untested"
Its not so much a case that latest and greatest is untested...
The chance of a bug being introduced is much higher when you add a feature then when you patch a bug. If you ship regular releases which all have new features, you are much more likely to ship new bugs than if you are just shipping patch releases that are restricted to fixing bugs. It is not a question of less testing - It is just that what you are trying to do is inherently less stable. As a customer, I usually don't want new features (unless there is a compelling reason).
I would advocate having a stable branch for most users, and a bleeding edge release for the enthusiasts (another name for a beta program)- (note - when I say "unstable", I mean "It changes a lot" rather than "It crashes a lot". Release often by all means, but I think it is a good idea to not try and get your everyday users onto that train.
Hi Melbournian,
While the main point of presenting the practice of frequent releases is to provide another tool for the toolbox (though I don't mean to say that it should be used infrequently :-) ), and as such one must use discretion in when and how to apply it, you raise a good point.
In a major release, there will be much more new functionality than if you are doing frequent releases. Perhaps by as much as 12 to 1. That functionality was probably produced under pressure and unevenly tested compared to the frequent releases.
With frequent releases, there will be much less new functionality and it will have received much more consistent testing (or else you are not really ready for frequent releases).
Since it is new functionality, you can be careful to mark it as such and users who want that release for some other reason can avoid the new functionality altogether.
In any case, I think our thinking on "dual trains" and "stable branch" is fairly close and I certainly agree that caution is advised when going further.
Rightly said Damon!!In addition to keeping your testing standards high, it is better to find problems that you are likely to only find by releasing to customers as soon as you possibly can. Therefore, release as often as you can.Do havev a look at our piece of work on this. https://www.scrumstudy.com/blog/all-about-agile-testing/
Post a Comment