Multi-Stage Continuous Integration Part I

I’m a big fan of continuous integration. I’ve used it and recommended it for many years. But it has a dark side as well. When it is combined with the practice of mainline development, especially for a large development effort, it can turn into “Continuous Noise.” In this case, you get notified every 10 minutes or so (depending on how much building and testing is going on) that the build and/or test suite is still failing.


In the diagram, the stability of the mainline is graphed over time. In this example, the graph starts when the product is released and the mainline is 100% stable. It builds without errors and all tests pass. As new development starts, developers make changes that break the build or cause tests to fail. Because they are all working against the mainline, it is likely that just as one developer fixes the problems they caused, another developer is checking in more changes that will create one or more new problems.

As the stability of mainline goes up and down, everybody that is working against mainline is affected. If the mainline doesn't build or there are tests that don't pass, then everybody has to wait until the problem is fixed.

It could be argued that the problem in this case is that the developer that broke the build should have updated their workspace, done a full build, and then run the full test suite after that. Basically, a full integration build and test. In practice this doesn’t work very well.

For any meaningful project, a full integration build and test will take time. Even if you somehow get that time down to 10 minutes, a lot can happen in 10 minutes, such as other people checking in their changes and invalidating your results (whether you realize it or not). It is more likely that this full cycle will take on the order of an hour or more. While you can certainly do something else during that hour, it is inconvenient to wait and necessitates task switching which is a well-known productivity killer.

Even though developer integration build and test has its problems, many shops have implemented it. While this practice is not ideal, it is an idea that is headed in the right direction.

Next: Multi Stage CI Part II .

There is no Bug. It is not the Bug that Bends, it is Only Yourself

In the movie “The Matrix,” when Neo goes to visit the Oracle, he has a conversation with one of the potentials who is bending a spoon with his mind. The boy explains to Neo that “there is no spoon” and that “It is not the spoon that bends, it is only yourself.” While this is rather mystical, it occurred to me that this is a good way to think about bugs!

Often, getting a bug to bend to your will (go away), is about as easy as bending a spoon with your mind. For instance, consider the problem of coloring maps such that no two contiguous regions are the same color. Let’s say that in your experience, you’ve only ever needed 3 colors. You need to color maps using software, so you write it with the assumption that you only ever need 3 colors. You may encounter some bugs which end up putting the same color on two contiguous regions in a map that can be colored with just 3 colors. Eventually you will find and fix those bugs. But if you run your program on a map that requires 4 colors, no amount of bug fixing your program that uses 3 colors will ever work. You must realize that your assumption that 3 colors will always work is wrong.

Another problem with bugs is that sometimes fixing the bug doesn’t actually fix the root problem. You think you know what the bug is, so you make a change so that the software now behaves as desired under the given circumstances. However, there are still 5 other ways that the problem can manifest itself and you didn’t fix any of those. Plus, you’ve introduced a new problem. You don’t know about these 6 problems yet, but they are there. You’ve only fixed the symptom, not the cause. You have not bent the spoon.

Even when you find and fix the root cause of that bug, you will write more software using your existing process and it will have new bugs to replace the old bugs. The progress was an illusion. The spoon is still unbent. If anything, it is even more unyielding than before.

It is tempting to think of a bug as something that creeps into the software after it is written, something that is separate that can be isolated and removed. But really, bugs are introduced during the process of developing software. They are not separate and isolated, they are actually flaws in the process itself. The problems lie within. Only when we acknowledge this ("there is no bug") and accept it can we hope to affect real change. Agile development is an excellent platform for seamlessly incorporating the required introspection and continual improvement.

I’m not saying that real software with zero bugs is an attainable goal, only that you consider a change of perspective in order to bend the spoon a bit. The ultimate goal is to make behavioral and process changes such that no bug is found in the software that you deliver.

Next: Apply Elegant Architecture to Your Dev Team

What is Your Impression of Agile?

I'm interested in hearing people's first and/or current impressions of Agile development. Here's mine.

My first real encounter with Agile development was at the Software Development Best Practices conference in September of 2005. Up until then I had managed to be all but ignorant of it. I was at the speaker’s reception, and unbeknownst to me I had been dropped into a den of Agilistas. They all looked like perfectly normal people to me. Conversely, they all seemed to assume that I was “one of them.” But then they started talking to me about 3x5 cards, pair programming, collocation and the like.

I wondered why these crazy people thought it would be a good idea to send poorly tested software to customers on a weekly basis. And what were they thinking doing their planning by shuffling around 3x5 cards with notes scratched on them? These folks were talking about writing software while sweating hip to hip with a fellow programmer all day as part of a two-person programming team in a small cramped space in close proximity with lots of other sweaty two person teams. And when I objected I was treated as a heretic that “just doesn’t get it.” To top it all off, this stuff is all written up in a Manifesto!!

I wanted to run as far as I could as fast as I could. In talking to other people about their first contact with Agile, I have found that this initial negative reaction is quite common. My reaction was so strong I decided to put together a whitepaper pointing out all of the problems with Agile and introducing an alternative to Agile which could be loosely described as “process and tools done right.” I felt that Agile, and especially Extreme Programming, was an overreaction to unwieldy project management, process, and tools. The more research I did for the paper, the more I found to dislike.

Along the way, I finally realized why Agile is a good thing. But, it took me close to six months of active research motivated by my dislike of Agile to see the light.

There really is something significantly new, different, and worthwhile lurking in there and it doesn't require small teams, pair programming, working in an open area, 3x5 cards, or fanaticism. This blog is an attempt to examine the benefits and inner workings of Agile from a technical perspective. That is, what exactly are the benefits, how exactly are they achieved, and why is it different than traditional development.

Describing Agile is a bit difficult, let's take a look at it at a high level to start with.

Next: Agile in a Nutshell

Agile Development is People Oriented

[This post is a remix of an earlier post and also a repost from my WordPress blog]

There’s a lot of literature available on the usability of both software and everyday objects. But have you ever considered the usability of the process of software development itself? Do you feel like you know exactly what to do next at every step of the process? Or do you feel pulled in a million different directions at once and lose track of what you’ve already done and what you need to do next?

In my experience, traditional software development involving long iterations suffers from poor usability and hinders the ability of otherwise intelligent people to get their work done. On the other hand, there is something special and people-oriented about Agile: short iterations leverage human nature and thus allow us to fall into a more natural rhythm in which we make fewer mistakes and work more productively.

While it is true that people can succeed on projects that have very long timeframes and involve a high degree of stress at the end, people are much more productive and much less prone to error when they work at a constant and sustainable pace. This also means that when there are unforeseen circumstances, people are more likely to be able to respond well.

Consider the following statements. I’ll bet that you’ll agree with most of them.

• When working on software for personal use, people make changes and then use the new version right away.

• One of the things that developers enjoy about software development is the fact that they can make a change and see the result right away.

• When working on a hotfix, people are more likely to cross all of the “t”s and dot all of the “i”s and take extra care to make sure that everything is done just right.

• It isn’t easy to create something that the customer thinks is exactly right the first time.

• People prefer to delay making important decisions as long as possible.

• Towards the end of a release when a new bug is found or a great idea for an enhancement comes in, it is tempting to just put it into the current release rather than the next release because the next release won’t happen for quite a while.

Each of these is a part of human nature. All of the positive points are reinforced with short iterations, and all of the negative ones are reduced with short iterations. Conversely, the exact opposite is true of long iterations. Long iterations go against the grain of human nature and thus create usability problems.

Next: The Agile Waterfall

Desiging Software is the same as Predicting the Future

Software Design Basics
Designing software requires a myriad of skills and knowledge to do well including: domain knowledge, market knowledge, technical aptitude, up-to-date knowledge of the technology relevant to the domain, and the ability to spot patterns and trends. If you have all of these, you have the basics required for the knowledge-based decisions that you must make as you design your software. But designing software also involves dealing with the unknown.

Predicting the Future
It is impossible to know exactly how many customers or users you will have, exactly what features will bring in the most revenue, which implementation technologies will work the best for you, the effective useful life of various implementation technologies, or which of many possible implementations will best satisfy your users. For these factors, you must make guesses. In other words, you must predict the future!

Predicting the future is hard. The more information you gather and the more variables you consider, the better your prediction will be, but the more time your prediction will take. Also, the more possibilities you take into account when you design your software, the bigger and more complex your design will become and the longer it will take you to produce the design. The bigger and more complex your design becomes, the more likely it is that you will have difficult design problems to solve. Solving difficult design problems can take a long time. Once you've finished this big and complex design, it will take a proportionately long time to implement that design.

When you put all of this together it is obvious that the more accurately you try to predict the future, the more time and effort it will take. The question is, how do you decide when your design is good enough? If you take it too far, you'll either never finish your design, or you'll never ship your software. But if you don't spend enough time on design, you may end up with unhappy users or software that needs to be rewritten.

Consider the problem of predicting the weather. Modern forecasting software is pretty good, but the further out you look, the longer it takes for the forecast to run and the more likely it is for the forecast to finish after the day you are forecasting has already come and gone. The other factor to consider when designing software is that technologies and user needs change almost as frequently as the weather.

Iterative Design and Iterative Implementation
What can we do to solve this dilemma? There are two choices: spend lots and lots of time on up-front design and the software that results from that large design or do iterative design on a smaller set of features. When you do big up-front design for a large set of features, it is inevitable that you will have to rewrite large parts of your software in the future to take into account changes in technology or user needs. Of course, you don't have to accommodate change, but then it is also less likely that you will maintain or increase your revenue stream. Since it is inevitable that you will have to make changes to keep your software relevant, why not just accept this fact and start building the skills and implementing the practices to get good at iterative design and iterative implementation? The alternative is that when you are forced to change your software, it wasn't written with change in mind and your team isn't used to making changes on a regular basis.

It isn't that hard to make this decision. After all, everybody knows that predicting the future is hard. What's more important, getting good at predicting the future, or getting good at adapting to change?

Next: The Simplest Thing That Could Possibly Work

It is Better To Find Customer Reported Problems as Soon as Possible

First, many thanks to Isaac Rodriguez whose comments on my previous post elicited a comment from me which I have re-written as a post here. I've also updated my previous post a bit to better state the original point.

Let's say that you develop your software in 30-day iterations. At the end of every iteration all functionality introduced during that iteration had all of its tests written during that iteration instead of at the end of the one year cycle. All of those tests pass. You take testing very seriously; you've got very high test coverage numbers, you are using decision based coverage instead of line coverage, and you fix every bug you find.

Let's say that you could release every month but instead you choose to release every year. Now at the end of this year you release your product. I guarantee that your customers will find problems. Let's say for the sake of simplicity that they find exactly 12 and each one is linked to functionality introduced in one of the 12 iterations. This means that for the bug in functionality in iteration 1, you waited 11 months to find the issue that your customer would have found right away.

I'm not saying that you should forgo testing or rely on your customers to be part of your QA department. I'm only saying that despite your best efforts, it is inevitable that there will be issues that you only find after you release, so keep on testing, don't stop that. But release as often as you can.

Also in this (contrived) example, your customers would still be exposed to the same number of bugs, just not all at the same time.

Moral to this story: "In addition to keeping your testing standards high, it is better to find problems that you are likely to only find by releasing to customers as soon as you possibly can. Therefore, release as often as you can."


Next: Customers Don't Want Frequent Releases

Frequent Releases Improve Code Quality Faster

There’s a saying that “you can’t test quality into a product.” That may be true, but it is very difficult to assess the level of quality of a product without testing it. I find that one of the problems with quality is the way that it is talked about. There are all different categories of testing such as regression, black-box, white-box, exploratory, automated, manual, etc. Each of these categories serves a useful purpose, but what is the overall purpose of testing?

Some people argue that testing is how you verify that the product works. Other people argue that the purpose of testing is to uncover what does not work. My view is that the purpose of testing is to show that the software works. By “works” I include that it correctly handles bad input, bad data, and diabolical usage. If you try to use the product in a way that was not intended and it crashes or gives no result at all, then you have found a way in which it does not work.

Quality Assurance and testing are not the same thing. You can implement all of the QA you want: good process, code reviews, coding standards, coding for testability, and whatever else you want to assure quality short of actual testing, but without testing, you have no idea how the product will actually perform. Let’s say you produce a product with absolutely no test plan or you have a great test plan but you don't execute it. I suppose that it is possible that the quality that the customer experiences will be exactly the same as if you had a test plan and executed it. The difference is that your knowledge of the quality of the product will come from your customer after you have already shipped your product. The purpose of testing is to derive knowledge about the quality of the product prior to shipping it so that you can make good decisions about what to do now that you have that knowledge.

So, testing itself does not produce higher quality. It only produces knowledge of current quality. There are only two things that produce higher quality: changes to the process that you use to create the product itself (not including testing), and the responses taken based on the knowledge gained from testing. Testing can influence quality, but it does not directly affect it.

Product quality is an ongoing effort. No matter how much testing you do, users will find ways to use your product that find holes in your test plan. For any change, whether it is a bug fix or a new feature, you don't really know the impact until customers start using it. That means that the maturation of a change doesn't really start until the change ships. If you have a long release cycle, you will have changes that are made early that are basically gathering dust until you ship. You can't really say that they are tested until you test the whole product as it will ship, so again, true maturation doesn't start until you ship.

The moral of this story? Release as often as you reasonably can; which may be once a week, once a month, or once a year, but always strive to release often.

Next: The Role of QA in an Agile Project

Book Recommendations for Agile Development

Continuous Integration, Paul M. Duvall
There's much more to Continuous Integration (CI) than just kicking off lots of builds and this book proves it. I've always been a big fan of CI, but even so I was skeptical that there could be enough material to fill a whole book (283 pages). There's really much more in this book than just CI. The author uses the context of CI to cover lots of software development best practices. Topics include: Continuous Build, Continuous Test, Continuous Inspection, Continuous Database Integration and many more. Release Engineers in particular will want to recommend this book as it eloquently and effectively covers many topics which are RE pet peeves such as "it works on my machine" and creating a consistent directory structure.

Patterns of Agile Practice Adoption, Amr Elssamadisy
This is a no-nonsense book for those that are looking to adopt some Agile practices, but aren't sure where to start. The book will help you to identify which problems you have that Agile practices can help solve and then gives straightforward advice on how to adopt those practices. It is available from InfoQ for free here.

Extreme Programming Explained - 2nd Edition, Kent Beck
Whether you agree with the idea of 3x5 cards or not, Kent's book is a short and easy read that is well worth the effort. A seminal book on Agile development.

Lean Six Sigma, Michael L. George
I read this book because I was interested in learning more about Six Sigma. However, I selected this book because the idea of "Lean" was appealing. The way that the problem and solutions for finished product sitting on a factory floor is explained immediately suggested parallels to finished features sitting in the source repository waiting for release.

Lean Software Development, Mary Poppendieck and Tom Poppendieck
These folks have done an excellent job of taking the lessons of Lean manufacturing, pioneered by the Toyota Production System, and translating them to software development. I read "Lean Six Sigma" first and on my next trip to the book store "Lean Software Development" became an obvious choice. Reading this book was the tipping point for me.

Agile Project Management with Scrum, Ken Schwaber
If you are already familiar with Scrum, this is a treasure trove of examples of how it has been applied in the real world.

Software Teamwork: Taking Ownership for Success, Jim Brosseau
Independent of your chosen methodology, the people and the personal interactions are a big factor in the success of any software development project. On an Agile project, the tight feedback loop not only surfaces technical problems faster, it will also make people issues much more apparent. This book offers a great framework for looking at the interactions of individuals, groups, teams, and stakeholders as well as many practical approaches for identifying and addressing specific issues. This is a terrific book to have at your side as you take the Agile plunge. [Link to this book on amazon.com]

Unconsciously Agile

The funny thing is that most shops are unconsciously flirting with Agile Development. They do a release on a regular basis which is very similar to what Agile teams do without even realizing it and then feel ashamed that they had to do it. Afterwards, they hope they won’t have to do another one. Those feelings of guilt and shame get associated with the need to do the release and it doesn’t occur to anybody that they are missing what is right in front of them. I know that I’ve been looking the evidence right in the face for years and I never saw it.

I am speaking of maintenance releases, patches, and hotfixes. Forget for the moment that these are created out of necessity due to problems with “the big release” and that they are often done under intense pressure. Forget for the moment that sometimes you have to redo a hotfix with another hotfix. The need for the second hotfix is often found by the intense scrutiny that is in place during the aftermath of the first hotfix and due to the fact that the hotfix probably didn’t get the full spin cycle of the normal QA process.

Think for a moment about how you later felt about that maintenance release. Didn’t you think something like “too bad we had to do it, but we really pulled together and I’m really proud of that release?” Wouldn’t you say that the type of release that spends the most time in production is actually a maintenance release or patch release?

A lot of the time, those maintenance releases aren’t just maintenance releases. Usually, there are a smattering of small enhancements thrown in for good measure for a customer or two.

So, let’s review. Your maintenance release has the highest quality, was produced in a short time, and it has a fair amount of new features in it. Guess what? This is very close to what Agile teams do on a regular basis. Surprise! But then of course, you revert back to your old habits.

I believe that the reason that most shops drift towards and then away from Agile development is that it is the natural rhythm of software development and it is hard to resist it. The challenge is to sustain it. In order to sustain it, you need two things: understanding of Agile and conscious intent. Without conscious intent and the synchronizing signal of short iterations it is hard to notice the difference between being on the path and off the path unless you stay on the path for a while. The other problem is that while everybody in an organization might from time to time all be synchronized into an Agile rhythm by pure chance, the synchronizing signal of the project plan, habits, and entrenched process interferes with the natural rhythm. Thus, the natural rhythm doesn’t get a chance to take hold.

Let me leave you with this thought: if your best releases are your maint/patch releases, and they are similar to what Agile teams do on a regular basis, perhaps you should take a closer look at Agile.

[Note: many thanks for Soon Hui's comments on an earlier version of this post.]

Next: The Agile Waterfall

Top Ten Reasons that Agile Produces Higher Quality

The best way that I currently know to convey why I think that Agile produces higher quality is via a 1 hour PowerPoint presentation. I thought it would be worthwhile to attempt to put it into a blog post. But, I couldn't figure out how to condense it down to a reasonable length. There's lots of diagrams and animation, etc. But while I was working on it I created a list of points to make. Looking at the list I thought "Hey! Here's yet another top ten list!"

Please note that in this list I am referring to truly Agile projects. See “No Really, What is Agile?" for details.

10. More appropriate distribution of test coverage. Typically, QA gets a big dump of functionality near the end of a release and has to figure out how to best use the time available. That often means that the "most important" new features get very thorough testing and the rest get a "spot check." In an Agile project, test plans and automated tests are created throughout the project and each work item is given the amount of QA resources that is appropriate for it.

9. Because testing is done in the same timeframe as the coding, problems are found earlier.

8. Writing tests early catches requirement and design problems earlier.

7. Because problems are found and fixed faster, there is less chance of the quality of a project being poor for long stretches of time. When there are lots of tests that don’t pass, it is difficult to get accurate feedback on new code. In contrast, code written on a stable base is more likely to be stable itself because there will be accurate and timely feedback on the results of the changes.

6. It is hard to succeed in an Agile environment without automated testing. Automated testing helps to increase the consistency of testing.

5. One effect of short iterations is the evening out of resource demands. That means that testing is done consistently and on a regular basis and there is no need to take shortcuts in contrast to typical development which compresses most of the testing to the end of the process which then requires taking shortcuts due to schedule pressure.

4. More frequent customer input on direction. Part of quality is usability and match of features to needs.

3. More frequent customer input on results. Customers are the ultimate arbiter of quality and their level of expectation is often different than you expect.

2. The Development and QA organizations must be integrated for Agile success. Integrated development and QA is far better than the typical “separation of church and state”.

1. Especially when doing one piece flow, there is significantly more opportunity to detect process problems, diagnose them, try corrective action, and gauge the results of the corrective action.

Many of the above deserve further elaboration, but I wanted to get this out there for discussion.

Agile Product Management

One reason that companies fail is that their software isn’t really differentiated from their competition. Just adding all of the bells and whistles that your customers ask for is not a way to differentiate. Your competition is being asked for exactly the same things.

While the appeal of rapidly responding to customer requests is very tempting, there are a few cautions in order. Customers don’t always know exactly what they want and don’t always express their high-level requirements. Often, they will ask for something because they have established a manual process to work around the fact that you are missing a whole feature set and there is one feature that would really help their manual process which wouldn’t even be needed if you had the feature set that they really need. By responding rapidly to customer requests as though they were well thought out requirements, you run the risk of creating Frankenstein’s monster: lots of bits and pieces that don’t really work well together and don’t form a very intelligent solution.

Also consider that unless you have a very big team or a very small stream of requests from users, it is unlikely that your development organization has the bandwidth to keep up with all of the requests from users. And of course, the more you provide, the wider your market appeal will be and the more customers you’ll have and the more requests you’ll get.

Each of the requests that you implement has both a cost and a benefit associated with it. If there is not enough benefit to justify the cost, it is debatable if you should even consider fulfilling the request. Remember though that the benefit may come in the form of customer satisfaction which can lead to more customers. In any case, you should do at least a rudimentary cost/benefit analysis to at least rank a particular request among all of the other requests you could do. Those with the best ROI should bubble to the top.

Even doing an ROI analysis is not really enough. What you really want to do is to add new capabilities to your product or new products to your product line with a good ROI while maintaining the integrity of those products. That is, you want to maintain the quality, usability, and performance of your products.

Next: The Role of Defect Management in Agile Development

More Agile Kool-Aid Please!

I got an interesting response to my post on "No Really, What is Agile?" today. For the complete response, see the comments on that post. It is fairly vitriolic and anonymous, but I have no problem with that because the benefit is that we get to see what is really on the poster's mind. The poster has clearly confused me as a spokesperson for "the Agile movement" which I certainly don't claim to represent. I would hope that anybody that has read a couple of my posts would realize that I'm not in the "Agile can do no wrong" camp. There is also clearly a high level of frustration apparent in the post. Hopefully, the venting was cathartic. :-)

I have read enough to know that the Agile Kool-Aid drinkers think they're immune to such concerns. Sadly, that is not the case. Instead, the Agile types keep moving the goalposts: "Oh, that's not Agile!" or "They're doing Agile wrong!"

Guess what? Eventually you'll move the goalposts so far that you'll hit the stands.

There are plenty of issues with Agile adoption. Some of them can certainly be pinned on the overzealousness of some Agile advocates, but some are due to the adopters themselves. As the Dilbert cartoon points out, adopting something without understanding it and without creating a proper ecosystem for it is destined to fail.

I am reminded of a day long ago when I decided to ignore a "trail closed" sign because 33" of powder had fallen during the night onto the trail which was rocks and grass the day before. It looked great, but there was no base, so it was just the same as riding on rocks and grass.

...where is your movement going to be when they totally bastardize it and projects fail? Well, Kent Beck and his gang started this movement with a huge failure of a project at a big company, but you always conveniently ignore that history.

Well first, it isn't my "movement" any more than my use of AJAX technologies means that I am part of the "AJAX movement". For me, Agile is neither a movement nor a philosophy, but rather a tool like any other. Agile is either a good idea or it isn't. I happen to think it is a great idea, but I agree that we have a long way to go before it is mainstream. As for ignoring history, I'm not sure what that means. Does that mean that when software development projects fail we should stop doing software development? When we fall down when learning to ski should we give up? Does it also mean that traditional development is the best way forward? If it was, there would be no such thing as Agile development because everybody would be loathe to change. Agile development has come to be precisely because of the frustration that people have with traditional development.

Just try going to Slashdot and posting your nonsense. They can be quite vicious.

I suppose my point is: don't be afraid to be self-critical about Agile. Don't make excuses.

Dear anonymous, one of the best ways to move things in a positive direction is to pull things in a positive direction, leading by example. I invite you to read more of my blog and respond to specific points instead of responding generally to Agile just because I have the word Agile in my blog.

No Really, What is Agile?

[Revisions: 6/11/08 general updates, 1/9/08 responding to comments]

Giving a concise definition of Agile is far from easy, probably because Agile is actually an umbrella for a wide variety of methodologies. Some will say that "it isn't about methodologies, it is a philosophy; it is the 4 values defined by the Agile Manifesto." Philosophies and Manifestos are a fine place to start, but at the end of the day you have to get down to brass tacks in order to actually bring about change.

It may be pointless to define Agile development. It may be better to just talk about the concepts and the practices and say "these things are worth learning about regardless of what you call it or how you define the overarching concept." I still think it is worth a shot so as to establish a starting point for discussion. So, let's discuss the definition of Agile a bit and then move on to the nitty-gritty.

Because there are so many practices associated with Agile development, a simpler way to define Agile is to do it in terms of the benefits. It is not a perfect way to define something, but it is currently the best way that I know. There would be no point in doing Agile development unless it was better, so I define the benefits in relation to traditional development. I consider waterfall to be part of traditional development, but I use the phrase "traditional development" instead of waterfall because there are many shops which would say "we're not doing Agile, but we're not waterfall either."

I define Agile development as that which, in comparison to traditional development provides the benefits of more flexibility, higher ROI, faster realization of ROI, higher quality, higher visibility, and sustainable pace. Let’s take a brief look at these benefits.

More Flexibility/Options
With traditional development the software may look like a construction site until just before the release: some parts are done, others are not, but the building is not currently usable at all. Changing plans during the development cycle means ripping out work in progress and possibly work that is done but depends on other work that is not done.

With Agile development, you develop your software in short iterations where “short” means a month or less. At the end of each iteration you have software which is shippable. That means that at the end of each iteration you have the option to easily change plans before the beginning of the next iteration in order to take into account new business requirements or information gained during the previous iteration.

Higher ROI
Because you have more flexibility you can change your plans to take into account the current market conditions instead of what seemed like a good idea when you started working on the release. So, if traditional development would have released after 6 months and had features A, B, and C which had an ROI of 9 during planning at release has an ROI of 7, Agile might produce A, B, and D in the same timeframe which has an ROI of 11. The idea is that while C looked like a good idea at the beginning it turned out later to be worth less than expected and D came along which had much more ROI so we did that instead.

Faster Realization of that ROI
Since you have shippable software at the end of every iteration you can start realizing the ROI whenever you like instead of having to wait. If the Agile team is also releasing frequently, they are able to capture the value that they created sooner than they would have otherwise been able to do.

Higher Quality
In a traditional project, the elapsed time from start to finish from the perspective of any individual work item is very long. There is no consistent process applied to each and every work item. Instead, work items are fused together into “the release” and the quality of “the release” is measured.

One consequence of this is that QA gets a big dump of functionality near the end of a release and has to figure out how to best use the time remaining which is often less than originally planned. That means taking shortcuts and the "most important" new features get very thorough testing and the rest get a "spot check." Another problem with traditional projects is that since much of the test writing and execution is left to the end, problems hide out for long periods of time and thus there is a long time between introduction of a problem and the detection and fixing of that problem.

In an Agile project, the elapsed time from start to finish from the perspective of any individual work item is very short. Test plans and automated tests are created throughout the project and each work item is given the amount of QA resources that is appropriate for it. Because problems are found and fixed faster, there is less chance of the quality of a project being poor for long stretches of time. Code is written on a stable base and is more likely to be stable itself because there will be accurate and timely feedback on the results of the changes.

Higher Visibility
Time after time, with traditional development, progress is measured based on progress against a plan. The problem with that is that customers don’t buy Gantt charts, they buy shipping software and the progress that is measured by Gantt charts is not directly connected to shippable software.

Just because you have finished 100% of the requirements which is 20% of your plan, that doesn’t mean you are 20% done. In fact you are 0% done from the perspective of the customer because they can’t benefit from that progress until you ship. In a traditional project, you only know how much ahead or behind plan you seem to be based on the progress that people claim, but you don’t really know how much ahead or behind plan you actually are from an “is this shippable” perspective.

Short iterations solve the problem of false progress reports. With short iterations, you know at the end of each iteration exactly how much progress you have made. Instead of the traditional "we're still on target" all the way up to just before the end of the release, you know at the end of every iteration exactly where you are.

Whatever work was done during that iteration is done and potentially shippable. You know exactly how many work items were fully completed, how many weren’t started, how many test cases remain to be written and automated, how many test failures remain to be fixed, and how many defects were filed on work done during the iteration. You get the kind of information that is usually only available just prior to release on a monthly or even weekly basis.

Next: Sustainable Pace

Top Ten Agile Analogies

I like analogies. Sometimes folks say I use them too much. I know that at least once I’ve come up with a really “out there” analogy. For instance, there was the one about cats riding motorcycles… Anyway, in a recent post I used two analogies and I previously posted about how Agile was like driving a car and like sales. So I was wondering what other analogies for Agile development were out there and did a search for “Agile Development is like” and “Agile is like.” Here is my top ten list. There actually weren’t many more than ten to choose from, surprisingly enough. If you know of another good one, please let me know!

10. “In many ways agile is like dieting or quitting smoking: you know it's good for you but some find it easier to fall back on old ways than to stick with the program” – Junilu Lacar

9. “Agile is just like driving a car” - various

8. “Agile is like sales” – yours truly

7. “Agile is like golf” - Joe Little (link)

6. “Agile is like religion” - various

5. “Agile is like a box of chocolates, you never know what you’re going to get” - Tim Coulter (link)

4. “Agile is like Churchhill’s democracy, the worst possible solution until compared to the alternatives” – David Starr (link)

3. “Agile is like being pregnant, you either are or you aren’t.” - anon (link)

2. "The puck stops here." (Agile development is like hockey, not ballet --- Tim Lister).

1. Agile is like any other newly introduced popular concept. “…Everybody is talking about it. Very few are doing it and those who are, are doing it badly.” - James O. Coplien

What is Agile Development?

I'm often asked "what is Agile Development?" I've never really been sure how to answer that. Some people say "well, it is the 4 values defined by the Agile Manifesto." To me, that's too vague. The four values are fairly philosophical in nature and pretty far from business benefits or in the trenches implementation. I tend to start any discussion of Agile by talking about the benefits. It seems to me that these benefits are fairly widely agreed upon as the following.

In comparison to "traditional development", Agile provides the benefits of:

- More flexibility/options
- Higher ROI
- Faster realization of that ROI
- Higher quality
- Higher visibility

Recently it occurred to me that this also serves as a good definition for Agile. That is, Agile development is that which provides the above benefits. I like this definition because it doesn't require a discussion of specific practices, yet it is also straightforward and concrete.

Agile Development as Panacea

I find that people have pet issues which they are always looking for a new way to solve. Often, when a new way of doing things is introduced, people ask if this new thing will solve one of their pet issues. Agile development is a new way of doing things. Thus, people often ask if it will solve one of their pet issues. As an Agile advocate, it is tempting to find a way to answer such questions in the positive.

Unfortunately, Agile development is not a panacea. It will not cure all your software development problems. Here are some of the issues that Agile development does not directly address: technical problems, choice of which technologies to use, personality conflicts, mismatched skill sets, insufficient management skills, poor listening skills, or poor communication skills.

However, there are two aspects of Agile development which can be applied to all software development problems: frequent feedback, and reduced complexity. The frequent feedback provided by Agile development provides a better problem solving environment than traditional development. The environment surfaces more problems, surfaces them faster, and provides more opportunities for addressing problems and gaging the effectiveness of those solutions.

Agile development also reduces the complexity of your software development projects by breaking it up into smaller pieces. It is much easier to complete 6 projects which are each 1/6th the complexity of a large project than it is to complete one large project. This is especially true when you take into account the fact that the planning required for the large project must cover a larger time frame and take more variables into account.

Consider this rough analogy. You want to transport 300 people from Boston to San Francisco and you are given the choice of two methods. The first method is to create a rough plan and have a pilot who can make adjustments as needed as you go. The second method is to line up the plane, set it to hit a cruising speed and altitude and go straight for a set length of time and then start its descent at a specific time. And there is no autopilot. Basically, you set the parameters at the beginning and that's it.

In the first case, you can adjust for changing conditions as they come up. These might include weather, air traffic, or natural disasters. There is a medium degree of difficulty for this method.

In the second case, you must accurately predict every possible condition and plan for it accordingly in order to have the plane safely hit the tarmac at the destination at the right time so as not to interfere with other planes. If there is a change in one of the variables affecting the flight that you didn't take into account, it could make for a rough landing. The degree of difficulty here is pretty much incalculable. Suffice it to say that it is very high.

As I write this I'm reminded of another analogy: golf! The variability involved in golf is exactly what makes it such an interesting game. It is also why there are only a small number of people in the world that have ever gotten a hole in one. Considering that traditional development is very similar to expecting to hit a hole in one on every hole, it is no wonder that so many software projects end up being sub-par.

Next: Agile Product Management

Agile Development: The Perfect Fit for Mission Critical Applications

I was talking to Ron Morsicato of Agile Rules at dinner at last weekend's Deep Lean conference and he was saying how he thought that Agile was a good idea for mission critical applications. That struck me as an interesting thing, and I've been thinking about that all week. I think he's right and here's why I think so.

Let's say you provide mission critical software and there is a problem reported in the field which requires a fix. Well, if you are doing traditional software development, you can't exactly ask the customer to wait until you finish your current release. And if you were to put out a fix for them using your main development path, that would probably take too long too. So, you have to have a critical-fix development path. By definition, this is a development process which is not used for regular development and it is not used very often (so one would hope).

As a result, you have one process for regular development and one process for critical fixes because your regular development process takes just too darn long. That means that when you most need for things to go smoothly you are going to use the process which is the least practiced and probably also something that only a handful of folks know or are allowed to do. Doesn't that strike you as just plain wrong? Imagine trying to explain that to the media!

Part of the idea with Agile, Lean, and especially Hyper Agile is that the path from deciding to make a change and being able to deliver that same change is as short as reasonably possible. If the "overhead" from start to finish for a critical fix is 4 hours, then any task should take you no more than 4 hours plus however long it takes you to write the code and the associated tests. Once you have this capability, then you really only need one development process which you use for both regular development and critical fixes! Thus, everybody is practiced in your critical fix process and everybody has experience with doing it. This will reduce risk and increase confidence in the results.

Agile Development: The Perfect Fit for Financial Institutions

A couple of days ago while at QCon San Francisco, I had a series of conversations with a person involved in software development at a major financial institution. He really wants to start using Agile techniques, but isn't sure of the best approach to take to convince others internally. I gave him some pointers including my recent article "Software Developers Can Learn a Lot from Salespeople." As I was thinking about this I realized that Agile is actually a perfect fit for financial institutions!

Software projects are highly unpredictable. More often than not they are either late, over budget, missing originally planned contents, much lower in quality than originally planned or a mix of all of the above. Another area of the business that has similar problems with predictability is sales.

When was the last time that salespeople were expected to tell you exactly what their financial performance would be? Sales people can't predict exactly which deals will come in during a quarter, nor exactly what the amount of any particular deal will be, but they are expected to hit their quotas and they are expected to show results on a regular basis and CFOs are perfectly happy with how their salesforce goes about their process. They may not be happy with the actual results sometimes, but they approve of their process.

What sales does and what Agile teams do are exactly the same! They are both working with unpredictable quantities. Developers in an Agile team may not be able to predict exactly what will be the result of any particular release and sometimes not even the result of an iteration, but they can be held accountable to produce a regular flow of high business value software on a regular basis and to be able to demonstrate that the software is actually done on a regular basis as well.

So, if the organization is comfortable with the sales process, they should be equally comfortable with Agile development. In fact, anybody involved in the business side of software development should be very uncomfortable with anything other than Agile development and be actively working to figure out how to get their development process to be more like sales.

This post has only briefly touched on the comparison of software development and sales. If you are interested in a full comparison, please take a look at "Software Developers Can Learn a Lot from Salespeople."

Agile Seminar at QCon

If you are going to be in San Francisco for QCon, check out my session on "What's Keeping Agile From Mainstream Adoption?" . I've changed my focus a bit from when this was originally created. It really should be called "Agile: Why it Works" at this point and that's really what the talk is about. There has been so much written and said about the benefits and how to achieve them, but not so much on the mechanics of why it works in the first place. I think that without this basic framework it is harder for people to take the plunge. It is definitely why I only got into Agile by accident rather than via an understanding of how it works at the outset.

Agile Bazaar Deep Lean Trip Report

Deep Lean was a 2-day event held at MIT, organized by the New England Agile Bazaar, and sponsored by AccuRev. Speakers included Jeff Sutherland, Nancy Van Schooenderwoert, and Mary & Tom Poppendieck. Here is a quick overview of the weekend.

Day 1
What a great first day at the Agile Bazaar's Deep Lean event. Jeff Sutherland started the ball rolling. All of the speakers were excellent (all weekend) at stopping right at the allotted time.

Broken Development – Jeff Sutherland
First Jeff showed us what a mess traditional development really is. Then he started to show us how Scrum can help. He didn't get through all of his material due to all of the questions from the audience, which was fine because the discussion was riveting.

Thrashing – Mary Poppendieck
According to Mary, even when development teams are doing Agile well, there is almost always one problem remaining: thrashing. Thrashing is when you are spending more and more time to do less and less. She showed us the root causes of thrashing which include excessive context switching. For instance, if you have two tasks and keep switching back and forth between them to show progress to multiple stakeholders, you will actually finish both slower than if you had focused on one and then the other. My favorite part was seeing a video of how the Toyota Type G loom could stop itself when it detected a defect in the fabric it was producing… without modern electronics!

Lean Principles and Scrum – Jeff Sutherland
A very straightforward subject which was well presented. Basically, Scrum overlaps with much of Lean by default and a deeper understanding of Lean will improve your experience with Scrum.

Value Stream Mapping – Mary and Tom Poppendieck
During this talk there was a point at which the audience broke up into teams to try value stream mapping on real world examples. While this was a great idea in concept, I don’t think there was enough time allotted to make it worth it. Considering this was one of the very few (only?) hitches in the whole weekend, I really didn’t mind. Besides, it is well covered in their first book on Lean which you should just go out and buy anyway.

At the end of the last session about 25 of us went to the nearby Cambridge Brewing Company for dinner and conversation. The CBC is a local favorite for the tech crowd and well worth checking out if you are ever in Kendal Square for an hour or more.

Day 2

Coming of Age of an Agile Company - Nancy Van Schooenderwoert
Nancy observed that Agile adoption has a high mortality rate. That is, well intentioned projects start up but then fail. This is a topic that is near and dear to my heart because part of my mission is to help make Agile mainstream. That is, to get Agile to the point that any team anywhere working on any project can pick it up and be successful quickly.

Nancy used a number of (painful) stories to illustrate her points. One of the things that struck me about the circumstances was that there were many problems which had nothing to do with Agile, but rather just plain old good development practices. For instance, don’t use your production environment to test code whose only claim to fame is that it compiles!

Leadership in Software Development – Mary Poppendieck
Mary took us through a fascinating tour of leadership and management styles and methods from the early 1900s to the present day: from Taylor to Charles Allen to Taiichi Ohno to Deming to Kaoru Ishikawa.

Real World Experiences in Creating Agile Companies – Jeff Sutherland
Jeff showed graph after graph on Scrum benefits. The only thing keeping me from being hypnotized by the rapid succession of graphs was Jeff’s take-no-prisoners style. Summary: Scrum works.

Panel Discussion
Lots of great Q&A too numerous to summarize here.

In summary, this event was a rare treat and if you have the opportunity to attend one of these events in the future, I highly recommend that you go. As both a sponsor and attendee, it was well worth it.

Are You Hyper Agile?

Something I've noticed a lot lately is that people often come away from a brush with Agile development thinking "gosh, we're pretty much already doing that now!" After thinking about it for a while I realized that there is a big difference between "pretty much" and "actually." :-) This got me thinking about how to make this distinction more obvious. I haven't come up with the perfect explanation yet, but I did come up with a "test" which can be used to see if you are Hyper Agile or not.

1. Did everything in the iteration come from the top of your backlog?
2. Is your backlog in highest-business-value order?
3. Are your iterations time-boxed?
At the end of each iteration...
4. have you written all of the tests that will be run at release time?
5. have all tests (not just new tests) been run?
6. do all tests (not just new ones) pass to the level that you would have at release?
7. are you as confident about releasing as you would be at an actual release?
8. there is no work in progress?
9. have you demoed all new functionality?
10. did you do root cause analysis on all problems?

If you answered "yes" to all of these, then you are getting the key benefits of Hyper Agile development. If you answered "no" to any of them, then you are not.

That doesn't mean that you have to do all of these in order to start experiencing secondary benefits. However, the primary benefits come when these are all happening in concert.

What's Keeping Agile Development from Mainstream Adoption?

Come and discuss this topic at a BOF session that I will be hosting at SD Best Practices 2007. Wednesday, September 19, 7:30 PM – 9:00 PM. The BOF will be held in the Exeter Room on the 3rd floor of the Sheraton. More details here.

Agile Development: It's Just Like Sales!

Here's the introduction to an article I wrote recently called "Developers Can Learn a lot From Salespeople" on CM Crossroads:

I know what you are thinking. Engineers don't need salespeople because truly great software will sell itself and the only thing sales people are good at is wining and dining management in order to push unwanted products onto unsuspecting users and running off with their cash.

I used to think that as well until I experienced the benefits of good salespeople first hand. It turns out that salespeople are much better than developers at predicting the future. In fact, successful businesses rely on sales forces to predict the future with a fairly high degree of accuracy.

On the surface this may seem impossible. It would seem that developers have much more control over their own destiny. Whether somebody will be interested in buying a product on a given date or how much they will be interested in buying seems completely random and unpredictable. Similarly, whether there is a sufficiently well performing algorithm to solve a problem and exactly how long it will take to find it, implement it, and verify it also seems completely random and unpredictable.

Read the whole article on CM Crossroads.

Natural Law

Laws, rules, process. They are all related and they are all around you. You don’t consciously think about them from moment to moment, but you are reacting to them moment to moment. They are second nature to you and you don’t have to think about them. On your way to work you drive on the correct side of the road, you stop at stop signs, and you stay in your own lane. When you go to lunch, you pay for your food. After work you go to your house instead of going to somebody else’s house.

It is easy to remember the laws, rules, and processes because you interact with them every day and you get reminders and possibly even unpleasant consequences when you go astray. A great example of this is the rumble strip that is commonplace on the side of highways these days. If for some reason you start to drift off the road, you’ll hear a loud warning noise as your tires hit the rumble strip. If you are really violating the law, you’ll hear an even louder sound accompanied by flashing lights. This is also known as feedback.

We get feedback our whole lives. When we first come into the world, we need constant feedback in order to learn how to survive. Touching something hot is painful, going without food is painful, falling down stairs is painful. In school we get feedback as to which subjects we are good at and which subjects we are not so good at. We can use this feedback to decide what to do more of, what to avoid and as an aid to improve, if we are able to.

An important part of absorbing all of these laws, rules, and processes is that they are clearly defined, easy to understand, and easy to remember. Once you have mastered basic physical laws such as running at full speed into a door is painful, it is easy to understand the reasoning behind having an upper limit on the speed of vehicles on public roads. You may disagree with that limit, but you can generally guess what the limit is for a given stretch of road and it is posted at regular intervals to help you score your guesses.

Somehow, the powerful combination of clearly defined process and reinforcement through immediate feedback are absent in most software development organizations. This is surprising considering that two of the main ingredients in software itself are rules and feedback.

One of the ways that Agile development helps is by providing frequent feedback, both to you as a software developer, and to the customer which in turn provides feedback to you. This makes it easier to learn about your market and your internal process and to make constant improvements in your product and your product development process.

Next: Agile Development is People Oriented

The Role of Defect Management in Agile Development

There are some who recommend against using a defect tracking system. Instead, it is recommended that when a bug is found, it is fixed immediately. While that is certainly one way of preventing an ever growing inventory of defects, the tracking of an inventory of defects is one of the smallest benefits of a defect tracking system. Overall, a defect tracking system serves as a facilitator. It simplifies the collection of defect reports from all sources. It isn’t just the developers responsible for fixing the defects that find problems. Customers, developers working on dependent systems, and testers also find defects. Even if you have a policy of fixing defects as soon as they are found, it isn’t always logistically possible to do so. For instance, if you are currently working on fixing a defect and in the process of doing so you find another one, you don’t want to lose track of it. Thus, a defect tracking system coordinates the collection of defect reports in a standard way and collects them in a well known location, insuring that important information about the functioning of your system is not lost.

A defect tracking system also manages the progress of work through the development life cycle from reporting, to triaging, to assignment, to test development, to completion, to test, to integration, to delivery. It simplifies the answering of customer questions such as “what is fixed in this release” and “what release will the fix appear in.” A defect tracking system also allows for the collection of metrics which aids in the spotting of trends. I have heard from multiple sources that metrics collected from a defect tracking system are worthless because developers will just game the system. That may be true in an unhealthy environment where the metrics are tied to compensation. However, in an environment where developers are actively participating in the improvement of the process, they will want this information in order to help to find and fix problems, including the root cause of individual problems.

Next: Designing Software is the Same Thing as Predicting the Future

Podcast: Open Source Technique for Agile Development

Check out this podcast to hear about how "meritocracy" allows developers to contribute outside of their traditional areas, to build trust in their capabilities, and to allow them to naturally gravitate to the areas where they are the most effective.

Podcast: Mainline Chaos

Here's a podcast which covers these topics:

• More on the practice of a "development hierarchy"
• Checkpoints
• Gatekeepers
• Atomic Transactions

Podcast: Hyper Agile Development for Compliance and Distributed Development

It might seem like doing Agile Development is not compatible with compliance requirements. However, Software Configuration Management (SCM) plays a vital role in any compliance effort that involves software development.

This podcast covers the following topics:

• How SCM and Issue Tracking can help with compliance in an Agile project
• Process is a dirty word
• The Hyper Agile practices of "development hierarchy" and "one virtual site"

Podcast: Winning the Jolt Award Using Hyper Agile Development

Here are some thoughts about winning the Jolt Award as well as a discussion of Hyper Agile, the new methodology that enabled AccuRev to deliver AccuWorkflow on time with high quality.

Primary vs. Secondary Agile Development Benefits

Agile is generally talked about as a single package: if you adhere to the principles, you get the benefits. I believe there is another way to look at Agile. Agile introduces a whole new set of practices to the development toolkit. These practices include: product backlog, pair programming, on-site customer, continuous integration, refactoring, test-first development and many others.

While all of these practices have been either associated with or created as a result of Agile, their application and resulting benefits are completely independent of your chosen methodology. Use Scrum or "Waterfall" and you still get the benefit.

For instance, refactoring is a standard feature in Eclipse. Eclipse is an IDE and is completely orthogonal to your methodology. The benefits from these practices are secondary benefits when practicing Agile. In my opinion, the primary benefits of faster turnaround, realizing business value sooner, and quality increases above and beyond the quality increases from the individual practices comes from the single practice of short iterations.

You don't have to do pair programming, use 3x5 cards, collocate, only use small teams, or have a customer on site to get to short iterations. These may help, but they are not the only practices that will. For instance, test driven development is very easy to adopt, contributes greatly to enabling short iterations and does not require any experimentation with personal boundaries.

Let's say you give me the benefit of the doubt and are willing to believe that there are benefits to implementing Agile. Why should you personally spend any time thinking about it?

Next: Agile Development, What's in it for Me?

The Dangers of Moving to Agile Development

My previous post received this comment:

"We have use the scrum process for about 6 months and I feel we are still practicing mini waterfalls. I really need help to defining the role of QA from start to finish. Are we suppose to write test plans? While we are waiting for development, we are unable to write test cases because the specs are not fully define and items that are being added are changing. In the end, we have the product not fully testing and catching up with the next sprint. No test cases are done and no automation is done. Seems like it is what we were doing before."

To me this sounds like a case of going to short iterations too fast! Most of my thoughts on the role of QA in an Agile project (Scrum or not) are articulated in the two recent posts I've already made on this topic, but here are a couple more thoughts...

Opinions may vary on this, but IMHO, if there are no test cases and no automation, you're just not going to succeed with any Agile methodology, Scrum or otherwise. One of the major principles of Agile is that you have working software at the end of every iteration. If it hasn't been tested, how do you know if it is working? It sounds to me like you've got complete chaos.

Do you have an Agile coach on the team? Do you have certified Scrum Masters on the team? If not, I'd highly recommend getting one or both of these.

I believe that for any Agile project to succeed, each story must go through a full and complete mini-lifecycle prior to the end of the iteration. That is, whatever process it would have gone through in your old methodology, it should still go through and it should happen during the term of the iteration. My preference would be that it go through those steps in immediate succession. That is, before code begins on any story, you are clear what you are setting out to accomplish (requirements), you have a plan for doing it (design/spec), and you have created as many test cases as possible. I'm not saying that all of the requirements are known at the start, but that you do as much as you can. Same for designs, specs, test cases. Worst case, all of these should be done for whatever code is done at the end of the release.

For instance, if story 1234 has code associated with it that performs 4 functions, then you better have the requirements, design, and test cases rock solid at the end of that iteration and those test cases better all be automated and passing. By "rock solid" on the requirements and design I mean that at a minimum, the coder better be able to articulate it to you. If they avert their eyes and hem and haw, how will anybody be able to use it?

If at the end of the iteration you don't have near 100% of what you have come to expect just prior to starting your release process (delivery to customer), then you need to stop immediately, do a retrospective, and take immediate corrective actions!!

Good luck and be sure to keep post updates on your progress.

Passing the Baton from Iteration to Iteration

My experience is that with automation, branching, and the supporting process, the goals of determining the quality of an iteration and creating stable baselines can be achieved in parallel. The first step is automation. Without a high degree of automated testing, I'm not sure it makes sense to try to do this in parallel. However, with continuous integration which includes full builds and full test suite runs, there should be few if any problems at the end of the iteration.

Some might say that their automated build and test takes a very long time. That’s a whole separate topic. In the meantime, let's assume that the full automated build and test cycle takes on the order of hours.

By branching, you can start the next iteration without disturbing the previous one. If QA finds any problems, development can fix them on the previous iteration branch. At some point that branch will be declared "done." Those changes are then merged into the current iteration. At that point, there is very little if any difference between where things are and where they would have been if development had started from the "done" baseline.

The advantage here is that development did not have to stop, and you still have a stable baseline. My experience is that once a team gets into a rhythm, the amount of time spent on "touch-up" is minimal.

This also applies to iterations that become releases. The "previous iteration" branch becomes work towards the release and then hopefully soon thereafter the release itself. Meanwhile, work continues towards the next release.

The Role of QA in an Agile Development Project II

At a high level, you should work to keep the ratio of development to QA balanced. The ratio will vary from project to project, so I'm not commenting on that, but once you have a ratio there are several ways to maintain it. In an Agile project, you may quickly find out that the ratio itself is off and need to adjust the ratio. In a project with long iterations, you might not find out that the ratio is out of whack until it is "too late."

In any case, once you find that you are short of QA resources, there are a couple of options. The first option of course is to get more QA resources. Failing that, you can use development resources. It may also be that you have a short-term need for more QA and in that case using development resources might be exactly the right short-term solution.

I think a common rule of thumb, which especially applies in the context of compliance, is that different people do the development and testing of a particular requirement. That is, if Sue is responsible for issue 12345, then Sue doesn't do the testing of it.

As to the exact use of development resources, I advocate that developers take on a sub-set of QA activities, thus allowing for better leveraging of the existing QA resources. For instance, creating automated unit tests or the general generation of automated test cases.

You can of course set whatever policy makes sense for your organization. Having people only write test cases for other people's code is always a good idea. Beyond that, I would assume that QA would take what the developers had done as a starting point, supplementing with additional tests, reviewing what the developers had done, running the official test cycle according to the test plan and reporting on the results.

Next: Multi Stage Continuous Integration

An Introduction to Hyper Agile Development

Hyper Agile (aka Hyper) is a development methodology designed to smoothly scale out from small teams to large teams and to maximize the potential of the development organization from a return on investment (ROI) perspective. I’m still working on documenting Hyper in full. In the meantime, here is yet another preview. Except for the practice of short iterations, each of the individual practices are best practices in and of themselves and can be implemented in conjunction with any development methodology.

Short Iterations - It has been widely stated that the benefits of Agile Development are producing value, maturity, and feedback faster; allowing early feedback to inform requirements to produce a better product, and increasing business flexibility. In my opinion, all of the benefits of traditional Agile Development can be directly attributed to the practice of short iterations, everything else is an enabler.

Automate Everything - The practice of automating everything is an extension of the Agile Development practice of writing automated test cases and fully automating the build process in support of the practice of continuous integration. If our main reason for existing as software developers is to automate manual processes, shouldn’t we be using automation more and more instead of less and less? Of course it makes sense that if we are going to be doing something new, there may not yet be off the shelf automation that fits the bill, but when there is, shouldn’t we use it?

Massively Parallel Virtual Pipelining - this is the process of breaking the lifecycle into as many separate steps as possible and then applying that lifecycle to each user requirement on an individual basis in parallel. This enables a high degree of asynchronous activity which in turn facilitates short iterations. I say a virtual pipeline because various stages of the pipeline may overlap with the same stages in other pipelines and because the various stages may be performed at different locations.

Development Hierarchy - A development hierarchy is simply a representation of the dependencies between groups that includes process steps such as integration, quality assurance, and code reviews. It encompasses a number of best practices including multiple isolation levels, gatekeepers, checkpoints, and always moving from a known good state to a known good state. It provides the structure for the parts of the virtual parallel pipelines that are associated with actual code. The fundamental building block of a development hierarchy is a branch.

Work Item Ranking - Work Item Ranking is done by putting all of the planned items into a single list. The first item is the highest priority item and the last item is the lowest priority item. This is also known as backlog in the Scrum Agile methodology. This practice vastly simplifies project planning, keeps you focused on your target market, and helps to prevent feature creep.

Estimation - At first glance, it might seem like estimation is a relatively simple and mundane task. Think about the task, compare it to your experience, give a rough guess and add a little padding. Actually, there is a surprising amount of literature that focuses solely on estimation. In the end, I think the most important part of estimation is to do it. I suggest you use the PERT method. It is widely used, and it is very simple. Take the least possible amount of time, add 3 times the expected, add in the most time it could possibly take, and divide the sum by 5. That’s it.

Writing Tests Early - Tests and requirements are very closely related. In order to write either one, you need to know the answer to the question “what is this supposed to do?” If the person responsible for writing the tests is struggling, it is a good early warning that it is not clear what the software is supposed to do. If it is not clear what the software is supposed to do, it is unlikely that anybody will be able to write the software to satisfy the end user. Thus, it makes sense to write the tests first to reduce the chance of writing code that does not reflect the needs of the end user.

Quality Quotient - One of the frequent objections to Agile (believe it or not) is related to quality. Since there is no good way to determine the exact quality of a product, it is difficult to empirically compare one iteration against another iteration to help to determine whether or not an iteration is ready for release. A quality quotient is a systematically calculated approximation of the quality of a particular iteration. The data that is compiled as part of the calculation of the quality quotient can also be used to guide future quality and process improvement efforts.

Meritocracy - Meritocracy allows developers to contribute outside of their traditional areas, to build trust in their capabilities, and to allow them to naturally gravitate to the areas where they are the most effective. An efficient way to implement meritocracy is to allow developers to create branches where they can implement their ideas. If they are successful in implementing their idea and the result is accepted, it is then a simple matter to incorporate the change into the development process.

Watered Down Agile Development

I find myself torn between being a supporter of Agile and being associated with the unprofessional aspects of it. For example, we all (or at least most of us I hope) know that discussing religion and politics in a professional environment is not a good idea. Yet that is often what many discussions about Agile seem like.

Actually, that just gave me idea. Next time you find yourself in such a discussion, you could say "hey, let's keep religion out of this" with a smile (or smiley) and try to steer things back to a professional level.

In the end, Agile is either a good idea or it is not and it will be adopted at its own pace in its own way. Trying to force it won't help. Representing it in a professional and appealing way will help.

Some folks complain that Agile is becoming “watered down.” Agile may evolve in ways that we as individuals don't care for. Oh well! But if it is a good idea, it will be adopted. It may be that it is adopted in a watered down fashion. Oh well! If it is adopted in a watered down fashion world-wide, that's actually progress. From that will come the next wave of adoption. And so it goes.

Consider refactoring and continuous integration. These are very widely adopted practices and aren't really even associated with Agile by many (most?) of the people using them, yet they did originate with Agile. You could say that people that are using only those practices are using "watered down" Agile. Is that a bad thing? I would say it is a very good thing. It provides more support for adoption of additional practices and helps to accelerate adoption of Agile in general.

The Role of QA in an Agile Development Project

An interesting question in Agile Development is “how do you incorporate QA?” I guess that all depends on what you see as the role of QA. For instance, if you have your QA people involved from start to finish, then you really wouldn't have to change a thing. For instance, a QA person can:

• help to determine if your stories are well defined
• suggest adding stories related to testability
• make sure that people are creating good unit tests and/or write unit tests
• increase your code coverage by adding more automated tests
• write tests for stories before (test first) or after they are completed
• organize usability testing
• add stories for improving usability
• do exploratory testing on early builds
• verify the completion of stories as they are completed

If they are involved from start to finish in this manner, then you eliminate the need for a well-defined QA stage. If you find at the end of an iteration that QA is "falling behind" then it could be argued that your ratio of development to QA is off. If so, then you can either add more QA resources or have developers take on some portion of the above tasks to restore the balance.

For AccuWorkflow, I had the partial use of multiple QA resources over time, so I had a "virtual QA person" which was different people at different times including developers doing the various tasks I mentioned.

With short iterations, it is likely that there will still be some touch-up work to be done on the previous iteration when the next one starts. Perhaps QA has not finished evaluating the iteration, or perhaps somebody wants to do a demo but runs into a bug that needs to be fixed first.

We were doing variable length iterations where the iterations were between 2 days and a week, so we ran into this hand-off period on a regular basis. The way we handled it was to create a stream for the “hand-off” period.

As an iteration ended, we created a new stream based on the existing development stream. Depending on the SCM system you are using that probably translates to either creating a branch or a label. In any case, the new stream then represented a stable version of the “previous iteration.” If somebody needed to do a little follow-on work, for instance to get things ready for a demo, they used another configuration called “demo” based on the “previous iteration” stream. To prepare for the demo they made changes in the demo stream without affecting the previous iteration or the current one. Once they finished the demo they then merged any changes into the current stream.

Having streams representing multiple iterations also made it possible to easily find out what had changed between iterations which aided in tracking down problems.

Of course, there are all of the standard things that a good QA organization brings that operate at a higher level than stories, but I'll just take those for granted for the purpose of this post.

Next: The Role of QA in an Agile Development Project Part II

Agile Development Drives Better Business Results

Imagine a world where you are predictably delivering high quality products to happy customers faster than ever before and realizing more revenue and higher profits as a result. A world where compliance is automatic, audits are a breeze, and expenses are kept under control.

In this world builds don’t break, developers are happy and productively working on the right things. People are working smoothly towards common goals at multiple sites across the globe. Using parallel development, multiple projects are worked on all at the same time, customer updates are made quickly, and there are fewer regressions due to fixes that didn’t make it into a new release.

More and more folks are experiencing this world as they upgrade their development environment. By development environment I mean the combination of development tools and development methods. Just as C replaced assembler, C++ replaced C, and Java and C# are replacing C++, new development tools are replacing the old ones and Agile development is replacing traditional development methods.

Change is in the air. The terms “Agile Development,” “Extreme Programming,” “continuous integration,” and others can be heard more and more. They are in the titles of conference sessions and books, they are showing up in new publications and web sites, and more and more companies are adopting these new methods.

It has been widely stated that the benefits of Agile Development are producing value, maturity, and feedback faster; allowing feedback to inform functionality to produce a better product, and increasing flexibility. Getting these benefits is much easier using more modern software development tools.

A lot has changed in the software development world in the last 10 years. If you’ve only ever used the older tools such as Bugzilla, CVS, ClearCase, ClearQuest, PVCS, VSS, SCCS, Synergy, or Dimensions, you are missing out on a whole new world of options. There are many exciting new entrants in the world of software development including AccuRev, Ant, Cruise Control, Jira, Mojo, Perforce, Surround SCM, Subversion, Team Foundation, and Team Track. As a simple example, none of the older SCM systems provide atomic transactions, but all of the newer systems do!

There are really only four ways to increase the profitability of a business based on software development: reduce costs via outsourcing, reduce headcount, reduce other expenses, or increase revenues. In the end, reducing expenses can only go so far. By far, the most expensive part of software development is the people. Compared to your people costs, the cost of migrating to a new open source tool or purchasing a new commercial software development tool is a drop in the bucket. The most successful way to increase profits is to increase revenues or increase productivity. These new tools and development methods can help you increase your revenues by helping you get your products to market faster with higher quality.

Refactoring and the Law of Unintended Consequences

Refactoring has the advantage that over time it makes code easier to understand and maintain. This can contribute to the maturity of the code. It has the disadvantage that it requires changing the code and any change has the potential to reduce the maturity and stability of the product.

The so-called “Law of Unintended Consequences” states that any human action will always have at least one consequence that was not intended. The bad part of this is when the unintended consequence causes a problem. Another way to look at this is that each change will have an effect; one of the effects is the reason for making the change in the first place. All of the others can be classified in two different ways: expected or not and reasonably predictable or not. When making a change, there may be effects that are expected even though they were not the reason for making the change. As long as they are considered to be beneficial, that’s probably ok. If there is any reason for not wanting the effect, even if some observers might consider it beneficial, then it is probably best to think of the effect as non-beneficial.

The other classification is whether or not an effect can be reasonably predicted. Some effects might not be immediately obvious. As with chess, the effect of moving a queen into a capture square of a pawn is immediately obvious. Generally, it means that the queen is going to be captured. However, it might also be the case that the queen was moved there in order to guarantee checkmate in five moves. Unless the other player is looking ahead at all possibilities five moves in advance, there is a very good chance that they will miss this. Thus, the worst part of the law is unintended effects which are harmful and cannot be reasonably expected to be predicted.

Some changes have a non-linear effect on a system. That is, the change was very small – perhaps a single character was changed, but the impact was very large, such as destruction of large amounts of data. Because the amount of change often has no connection to the potential impact, it is always best to keep the amount of change to the absolute minimum required.

Even if a change could be reasonably expected to be predicted, sometimes the effort to do so is not invested. In any case, any effects that were not determined prior to incorporating the change are considered to be unforeseen consequences.

There are some strategies for minimizing the risk of unforeseen consequences. The first is to make a reasonable effort to determine effects other than just the intended one. The second is to do an impact analysis and either verify that the current requirements and code coverage are sufficiently high to mitigate the risk or to increase coverage to a reasonably high level.

The Agile Development Waterfall

These days, there is a lot of criticism of the “waterfall” model of development. I believe that is because it is used as a blanket term that actually has three important subcomponents. In a classic waterfall project, the duration of the project is however long it takes with the resources provided to produce the software which meets all of the requirements. There are set process stages that each work item goes through such as requirements gathering, requirements specification, design, development, integration, etc. Lastly, there is the characteristic that all of the work items go through the process stages in lock step.

I submit that all software development projects share an essential part of the “waterfall” model: the steps of the waterfall itself. This is out of necessity. When was the last time you saw code spontaneously pop into existence without being written, or code being written without any goal in mind, even if it was as vague as “make it better?”

All changes to software go through a waterfall of stages. Even if it is a fuzzy idea, change starts with a concept for what needs to be done. From this concept comes a plan. The plan may exist only as a flash of insight in the mind of the developer and they may not even be conscious of it, but there is always a plan. From the plan comes the code. Lastly, the proof of the original concept comes from trying the result. Thus, even in the least rigorous of environments, working on a single issue, there is at least a 6 step waterfall.

All of the stages described may be intermingled, have no clear start or finish, and be repeated multiple times, but they are there. Another way to describe them would be: need, requirements, specification, design, development, test.

The major differences between the classic waterfall model of development and other models of development come in the number and duration of iterations, number of steps in the waterfall, degree of detail in the steps, and the degree of synchronicity between the steps and between the work items.

For instance, the classic waterfall has a single iteration where the duration of the project is however long it takes with the resources provided to produce the software which meets all of the requirements, there is a very high degree of detail including verbose documentation every step of the way, and all of the work items progress together from one step to the next.

In the Agile model, there are frequent short iterations which consist of self-contained projects representing the highest value work that can be done during a single iteration, there is a minimum of detail and documentation, and the work items may or may not progress together from one step to the next, but those steps are still there.

I suggest that the value of the waterfall model, when applied to a single work item, is in producing a well defined series of steps which when followed is known to detect and filter out problems. For any particular development team, those steps may vary a great deal, but by refining the steps and applying them consistently, you have a higher likelihood of producing consistent results.

Next: Sustainable Pace: Supply vs. Demand