Multi-Stage Continuous Integration Part I

I’m a big fan of continuous integration. I’ve used it and recommended it for many years. But it has a dark side as well. When it is combined with the practice of mainline development, especially for a large development effort, it can turn into “Continuous Noise.” In this case, you get notified every 10 minutes or so (depending on how much building and testing is going on) that the build and/or test suite is still failing.


In the diagram, the stability of the mainline is graphed over time. In this example, the graph starts when the product is released and the mainline is 100% stable. It builds without errors and all tests pass. As new development starts, developers make changes that break the build or cause tests to fail. Because they are all working against the mainline, it is likely that just as one developer fixes the problems they caused, another developer is checking in more changes that will create one or more new problems.

As the stability of mainline goes up and down, everybody that is working against mainline is affected. If the mainline doesn't build or there are tests that don't pass, then everybody has to wait until the problem is fixed.

It could be argued that the problem in this case is that the developer that broke the build should have updated their workspace, done a full build, and then run the full test suite after that. Basically, a full integration build and test. In practice this doesn’t work very well.

For any meaningful project, a full integration build and test will take time. Even if you somehow get that time down to 10 minutes, a lot can happen in 10 minutes, such as other people checking in their changes and invalidating your results (whether you realize it or not). It is more likely that this full cycle will take on the order of an hour or more. While you can certainly do something else during that hour, it is inconvenient to wait and necessitates task switching which is a well-known productivity killer.

Even though developer integration build and test has its problems, many shops have implemented it. While this practice is not ideal, it is an idea that is headed in the right direction.

Next: Multi Stage CI Part II .

There is no Bug. It is not the Bug that Bends, it is Only Yourself

In the movie “The Matrix,” when Neo goes to visit the Oracle, he has a conversation with one of the potentials who is bending a spoon with his mind. The boy explains to Neo that “there is no spoon” and that “It is not the spoon that bends, it is only yourself.” While this is rather mystical, it occurred to me that this is a good way to think about bugs!

Often, getting a bug to bend to your will (go away), is about as easy as bending a spoon with your mind. For instance, consider the problem of coloring maps such that no two contiguous regions are the same color. Let’s say that in your experience, you’ve only ever needed 3 colors. You need to color maps using software, so you write it with the assumption that you only ever need 3 colors. You may encounter some bugs which end up putting the same color on two contiguous regions in a map that can be colored with just 3 colors. Eventually you will find and fix those bugs. But if you run your program on a map that requires 4 colors, no amount of bug fixing your program that uses 3 colors will ever work. You must realize that your assumption that 3 colors will always work is wrong.

Another problem with bugs is that sometimes fixing the bug doesn’t actually fix the root problem. You think you know what the bug is, so you make a change so that the software now behaves as desired under the given circumstances. However, there are still 5 other ways that the problem can manifest itself and you didn’t fix any of those. Plus, you’ve introduced a new problem. You don’t know about these 6 problems yet, but they are there. You’ve only fixed the symptom, not the cause. You have not bent the spoon.

Even when you find and fix the root cause of that bug, you will write more software using your existing process and it will have new bugs to replace the old bugs. The progress was an illusion. The spoon is still unbent. If anything, it is even more unyielding than before.

It is tempting to think of a bug as something that creeps into the software after it is written, something that is separate that can be isolated and removed. But really, bugs are introduced during the process of developing software. They are not separate and isolated, they are actually flaws in the process itself. The problems lie within. Only when we acknowledge this ("there is no bug") and accept it can we hope to affect real change. Agile development is an excellent platform for seamlessly incorporating the required introspection and continual improvement.

I’m not saying that real software with zero bugs is an attainable goal, only that you consider a change of perspective in order to bend the spoon a bit. The ultimate goal is to make behavioral and process changes such that no bug is found in the software that you deliver.

Next: Apply Elegant Architecture to Your Dev Team

What is Your Impression of Agile?

I'm interested in hearing people's first and/or current impressions of Agile development. Here's mine.

My first real encounter with Agile development was at the Software Development Best Practices conference in September of 2005. Up until then I had managed to be all but ignorant of it. I was at the speaker’s reception, and unbeknownst to me I had been dropped into a den of Agilistas. They all looked like perfectly normal people to me. Conversely, they all seemed to assume that I was “one of them.” But then they started talking to me about 3x5 cards, pair programming, collocation and the like.

I wondered why these crazy people thought it would be a good idea to send poorly tested software to customers on a weekly basis. And what were they thinking doing their planning by shuffling around 3x5 cards with notes scratched on them? These folks were talking about writing software while sweating hip to hip with a fellow programmer all day as part of a two-person programming team in a small cramped space in close proximity with lots of other sweaty two person teams. And when I objected I was treated as a heretic that “just doesn’t get it.” To top it all off, this stuff is all written up in a Manifesto!!

I wanted to run as far as I could as fast as I could. In talking to other people about their first contact with Agile, I have found that this initial negative reaction is quite common. My reaction was so strong I decided to put together a whitepaper pointing out all of the problems with Agile and introducing an alternative to Agile which could be loosely described as “process and tools done right.” I felt that Agile, and especially Extreme Programming, was an overreaction to unwieldy project management, process, and tools. The more research I did for the paper, the more I found to dislike.

Along the way, I finally realized why Agile is a good thing. But, it took me close to six months of active research motivated by my dislike of Agile to see the light.

There really is something significantly new, different, and worthwhile lurking in there and it doesn't require small teams, pair programming, working in an open area, 3x5 cards, or fanaticism. This blog is an attempt to examine the benefits and inner workings of Agile from a technical perspective. That is, what exactly are the benefits, how exactly are they achieved, and why is it different than traditional development.

Describing Agile is a bit difficult, let's take a look at it at a high level to start with.

Next: Agile in a Nutshell

Agile Development is People Oriented

[This post is a remix of an earlier post and also a repost from my WordPress blog]

There’s a lot of literature available on the usability of both software and everyday objects. But have you ever considered the usability of the process of software development itself? Do you feel like you know exactly what to do next at every step of the process? Or do you feel pulled in a million different directions at once and lose track of what you’ve already done and what you need to do next?

In my experience, traditional software development involving long iterations suffers from poor usability and hinders the ability of otherwise intelligent people to get their work done. On the other hand, there is something special and people-oriented about Agile: short iterations leverage human nature and thus allow us to fall into a more natural rhythm in which we make fewer mistakes and work more productively.

While it is true that people can succeed on projects that have very long timeframes and involve a high degree of stress at the end, people are much more productive and much less prone to error when they work at a constant and sustainable pace. This also means that when there are unforeseen circumstances, people are more likely to be able to respond well.

Consider the following statements. I’ll bet that you’ll agree with most of them.

• When working on software for personal use, people make changes and then use the new version right away.

• One of the things that developers enjoy about software development is the fact that they can make a change and see the result right away.

• When working on a hotfix, people are more likely to cross all of the “t”s and dot all of the “i”s and take extra care to make sure that everything is done just right.

• It isn’t easy to create something that the customer thinks is exactly right the first time.

• People prefer to delay making important decisions as long as possible.

• Towards the end of a release when a new bug is found or a great idea for an enhancement comes in, it is tempting to just put it into the current release rather than the next release because the next release won’t happen for quite a while.

Each of these is a part of human nature. All of the positive points are reinforced with short iterations, and all of the negative ones are reduced with short iterations. Conversely, the exact opposite is true of long iterations. Long iterations go against the grain of human nature and thus create usability problems.

Next: The Agile Waterfall

Desiging Software is the same as Predicting the Future

Software Design Basics
Designing software requires a myriad of skills and knowledge to do well including: domain knowledge, market knowledge, technical aptitude, up-to-date knowledge of the technology relevant to the domain, and the ability to spot patterns and trends. If you have all of these, you have the basics required for the knowledge-based decisions that you must make as you design your software. But designing software also involves dealing with the unknown.

Predicting the Future
It is impossible to know exactly how many customers or users you will have, exactly what features will bring in the most revenue, which implementation technologies will work the best for you, the effective useful life of various implementation technologies, or which of many possible implementations will best satisfy your users. For these factors, you must make guesses. In other words, you must predict the future!

Predicting the future is hard. The more information you gather and the more variables you consider, the better your prediction will be, but the more time your prediction will take. Also, the more possibilities you take into account when you design your software, the bigger and more complex your design will become and the longer it will take you to produce the design. The bigger and more complex your design becomes, the more likely it is that you will have difficult design problems to solve. Solving difficult design problems can take a long time. Once you've finished this big and complex design, it will take a proportionately long time to implement that design.

When you put all of this together it is obvious that the more accurately you try to predict the future, the more time and effort it will take. The question is, how do you decide when your design is good enough? If you take it too far, you'll either never finish your design, or you'll never ship your software. But if you don't spend enough time on design, you may end up with unhappy users or software that needs to be rewritten.

Consider the problem of predicting the weather. Modern forecasting software is pretty good, but the further out you look, the longer it takes for the forecast to run and the more likely it is for the forecast to finish after the day you are forecasting has already come and gone. The other factor to consider when designing software is that technologies and user needs change almost as frequently as the weather.

Iterative Design and Iterative Implementation
What can we do to solve this dilemma? There are two choices: spend lots and lots of time on up-front design and the software that results from that large design or do iterative design on a smaller set of features. When you do big up-front design for a large set of features, it is inevitable that you will have to rewrite large parts of your software in the future to take into account changes in technology or user needs. Of course, you don't have to accommodate change, but then it is also less likely that you will maintain or increase your revenue stream. Since it is inevitable that you will have to make changes to keep your software relevant, why not just accept this fact and start building the skills and implementing the practices to get good at iterative design and iterative implementation? The alternative is that when you are forced to change your software, it wasn't written with change in mind and your team isn't used to making changes on a regular basis.

It isn't that hard to make this decision. After all, everybody knows that predicting the future is hard. What's more important, getting good at predicting the future, or getting good at adapting to change?

Next: The Simplest Thing That Could Possibly Work

It is Better To Find Customer Reported Problems as Soon as Possible

First, many thanks to Isaac Rodriguez whose comments on my previous post elicited a comment from me which I have re-written as a post here. I've also updated my previous post a bit to better state the original point.

Let's say that you develop your software in 30-day iterations. At the end of every iteration all functionality introduced during that iteration had all of its tests written during that iteration instead of at the end of the one year cycle. All of those tests pass. You take testing very seriously; you've got very high test coverage numbers, you are using decision based coverage instead of line coverage, and you fix every bug you find.

Let's say that you could release every month but instead you choose to release every year. Now at the end of this year you release your product. I guarantee that your customers will find problems. Let's say for the sake of simplicity that they find exactly 12 and each one is linked to functionality introduced in one of the 12 iterations. This means that for the bug in functionality in iteration 1, you waited 11 months to find the issue that your customer would have found right away.

I'm not saying that you should forgo testing or rely on your customers to be part of your QA department. I'm only saying that despite your best efforts, it is inevitable that there will be issues that you only find after you release, so keep on testing, don't stop that. But release as often as you can.

Also in this (contrived) example, your customers would still be exposed to the same number of bugs, just not all at the same time.

Moral to this story: "In addition to keeping your testing standards high, it is better to find problems that you are likely to only find by releasing to customers as soon as you possibly can. Therefore, release as often as you can."


Next: Customers Don't Want Frequent Releases

Frequent Releases Improve Code Quality Faster

There’s a saying that “you can’t test quality into a product.” That may be true, but it is very difficult to assess the level of quality of a product without testing it. I find that one of the problems with quality is the way that it is talked about. There are all different categories of testing such as regression, black-box, white-box, exploratory, automated, manual, etc. Each of these categories serves a useful purpose, but what is the overall purpose of testing?

Some people argue that testing is how you verify that the product works. Other people argue that the purpose of testing is to uncover what does not work. My view is that the purpose of testing is to show that the software works. By “works” I include that it correctly handles bad input, bad data, and diabolical usage. If you try to use the product in a way that was not intended and it crashes or gives no result at all, then you have found a way in which it does not work.

Quality Assurance and testing are not the same thing. You can implement all of the QA you want: good process, code reviews, coding standards, coding for testability, and whatever else you want to assure quality short of actual testing, but without testing, you have no idea how the product will actually perform. Let’s say you produce a product with absolutely no test plan or you have a great test plan but you don't execute it. I suppose that it is possible that the quality that the customer experiences will be exactly the same as if you had a test plan and executed it. The difference is that your knowledge of the quality of the product will come from your customer after you have already shipped your product. The purpose of testing is to derive knowledge about the quality of the product prior to shipping it so that you can make good decisions about what to do now that you have that knowledge.

So, testing itself does not produce higher quality. It only produces knowledge of current quality. There are only two things that produce higher quality: changes to the process that you use to create the product itself (not including testing), and the responses taken based on the knowledge gained from testing. Testing can influence quality, but it does not directly affect it.

Product quality is an ongoing effort. No matter how much testing you do, users will find ways to use your product that find holes in your test plan. For any change, whether it is a bug fix or a new feature, you don't really know the impact until customers start using it. That means that the maturation of a change doesn't really start until the change ships. If you have a long release cycle, you will have changes that are made early that are basically gathering dust until you ship. You can't really say that they are tested until you test the whole product as it will ship, so again, true maturation doesn't start until you ship.

The moral of this story? Release as often as you reasonably can; which may be once a week, once a month, or once a year, but always strive to release often.

Next: The Role of QA in an Agile Project

Book Recommendations for Agile Development

Continuous Integration, Paul M. Duvall
There's much more to Continuous Integration (CI) than just kicking off lots of builds and this book proves it. I've always been a big fan of CI, but even so I was skeptical that there could be enough material to fill a whole book (283 pages). There's really much more in this book than just CI. The author uses the context of CI to cover lots of software development best practices. Topics include: Continuous Build, Continuous Test, Continuous Inspection, Continuous Database Integration and many more. Release Engineers in particular will want to recommend this book as it eloquently and effectively covers many topics which are RE pet peeves such as "it works on my machine" and creating a consistent directory structure.

Patterns of Agile Practice Adoption, Amr Elssamadisy
This is a no-nonsense book for those that are looking to adopt some Agile practices, but aren't sure where to start. The book will help you to identify which problems you have that Agile practices can help solve and then gives straightforward advice on how to adopt those practices. It is available from InfoQ for free here.

Extreme Programming Explained - 2nd Edition, Kent Beck
Whether you agree with the idea of 3x5 cards or not, Kent's book is a short and easy read that is well worth the effort. A seminal book on Agile development.

Lean Six Sigma, Michael L. George
I read this book because I was interested in learning more about Six Sigma. However, I selected this book because the idea of "Lean" was appealing. The way that the problem and solutions for finished product sitting on a factory floor is explained immediately suggested parallels to finished features sitting in the source repository waiting for release.

Lean Software Development, Mary Poppendieck and Tom Poppendieck
These folks have done an excellent job of taking the lessons of Lean manufacturing, pioneered by the Toyota Production System, and translating them to software development. I read "Lean Six Sigma" first and on my next trip to the book store "Lean Software Development" became an obvious choice. Reading this book was the tipping point for me.

Agile Project Management with Scrum, Ken Schwaber
If you are already familiar with Scrum, this is a treasure trove of examples of how it has been applied in the real world.

Software Teamwork: Taking Ownership for Success, Jim Brosseau
Independent of your chosen methodology, the people and the personal interactions are a big factor in the success of any software development project. On an Agile project, the tight feedback loop not only surfaces technical problems faster, it will also make people issues much more apparent. This book offers a great framework for looking at the interactions of individuals, groups, teams, and stakeholders as well as many practical approaches for identifying and addressing specific issues. This is a terrific book to have at your side as you take the Agile plunge. [Link to this book on amazon.com]

Unconsciously Agile

The funny thing is that most shops are unconsciously flirting with Agile Development. They do a release on a regular basis which is very similar to what Agile teams do without even realizing it and then feel ashamed that they had to do it. Afterwards, they hope they won’t have to do another one. Those feelings of guilt and shame get associated with the need to do the release and it doesn’t occur to anybody that they are missing what is right in front of them. I know that I’ve been looking the evidence right in the face for years and I never saw it.

I am speaking of maintenance releases, patches, and hotfixes. Forget for the moment that these are created out of necessity due to problems with “the big release” and that they are often done under intense pressure. Forget for the moment that sometimes you have to redo a hotfix with another hotfix. The need for the second hotfix is often found by the intense scrutiny that is in place during the aftermath of the first hotfix and due to the fact that the hotfix probably didn’t get the full spin cycle of the normal QA process.

Think for a moment about how you later felt about that maintenance release. Didn’t you think something like “too bad we had to do it, but we really pulled together and I’m really proud of that release?” Wouldn’t you say that the type of release that spends the most time in production is actually a maintenance release or patch release?

A lot of the time, those maintenance releases aren’t just maintenance releases. Usually, there are a smattering of small enhancements thrown in for good measure for a customer or two.

So, let’s review. Your maintenance release has the highest quality, was produced in a short time, and it has a fair amount of new features in it. Guess what? This is very close to what Agile teams do on a regular basis. Surprise! But then of course, you revert back to your old habits.

I believe that the reason that most shops drift towards and then away from Agile development is that it is the natural rhythm of software development and it is hard to resist it. The challenge is to sustain it. In order to sustain it, you need two things: understanding of Agile and conscious intent. Without conscious intent and the synchronizing signal of short iterations it is hard to notice the difference between being on the path and off the path unless you stay on the path for a while. The other problem is that while everybody in an organization might from time to time all be synchronized into an Agile rhythm by pure chance, the synchronizing signal of the project plan, habits, and entrenched process interferes with the natural rhythm. Thus, the natural rhythm doesn’t get a chance to take hold.

Let me leave you with this thought: if your best releases are your maint/patch releases, and they are similar to what Agile teams do on a regular basis, perhaps you should take a closer look at Agile.

[Note: many thanks for Soon Hui's comments on an earlier version of this post.]

Next: The Agile Waterfall

Top Ten Reasons that Agile Produces Higher Quality

The best way that I currently know to convey why I think that Agile produces higher quality is via a 1 hour PowerPoint presentation. I thought it would be worthwhile to attempt to put it into a blog post. But, I couldn't figure out how to condense it down to a reasonable length. There's lots of diagrams and animation, etc. But while I was working on it I created a list of points to make. Looking at the list I thought "Hey! Here's yet another top ten list!"

Please note that in this list I am referring to truly Agile projects. See “No Really, What is Agile?" for details.

10. More appropriate distribution of test coverage. Typically, QA gets a big dump of functionality near the end of a release and has to figure out how to best use the time available. That often means that the "most important" new features get very thorough testing and the rest get a "spot check." In an Agile project, test plans and automated tests are created throughout the project and each work item is given the amount of QA resources that is appropriate for it.

9. Because testing is done in the same timeframe as the coding, problems are found earlier.

8. Writing tests early catches requirement and design problems earlier.

7. Because problems are found and fixed faster, there is less chance of the quality of a project being poor for long stretches of time. When there are lots of tests that don’t pass, it is difficult to get accurate feedback on new code. In contrast, code written on a stable base is more likely to be stable itself because there will be accurate and timely feedback on the results of the changes.

6. It is hard to succeed in an Agile environment without automated testing. Automated testing helps to increase the consistency of testing.

5. One effect of short iterations is the evening out of resource demands. That means that testing is done consistently and on a regular basis and there is no need to take shortcuts in contrast to typical development which compresses most of the testing to the end of the process which then requires taking shortcuts due to schedule pressure.

4. More frequent customer input on direction. Part of quality is usability and match of features to needs.

3. More frequent customer input on results. Customers are the ultimate arbiter of quality and their level of expectation is often different than you expect.

2. The Development and QA organizations must be integrated for Agile success. Integrated development and QA is far better than the typical “separation of church and state”.

1. Especially when doing one piece flow, there is significantly more opportunity to detect process problems, diagnose them, try corrective action, and gauge the results of the corrective action.

Many of the above deserve further elaboration, but I wanted to get this out there for discussion.