Jim Coplien's New Book on Agile Architecture

Did you know that Jim Coplien will be publishing a new book on Lean Architecture called "Practical Agile Production?" Find out more by attending Deep Agile 2008 at MIT on Nov 8-9 2008 and receive an autographed manuscript of the book, a once-in-a-lifetime opportunity! Find out more at the Agile Bazaar's Deep Agile 2008 web page.

SD East - Oct 30th - Multi-stage Continuous Integration Presentation

Interested in Continuous Integration? Come see my presentation at SD Best Practices this Thursday at 8:30am in room 102 to see how it scales to teams from 20 to 500 with Multi-stage Continuous Integration. See the presentation conference page for more details.

Oct 23rd - Presentation on Multi-Stage Continuous Integration at MIT

Come hear my presentation on Multi-Stage Continuous Integration!

I'll be speaking at this month's Agile Bazaar meeting.

MIT Tang Center
Building E51-Third Floor
(Room number TBD - We will meet outside Room 335)

6:30 - 7:30 Presentation
7:30 - 8:00 Q & A, Networking

We usually walk over to the CBC or similar venue afterwards to continue the networking.

Continuous Integration is an increasingly popular technique for discovering and fixing problems early. As larger groups start to adopt it, a problem arises. A full integration build and test cycle can easily take 30 minutes. The chances rise exponentially with project size that during that time, other people's check-ins will silently invalidate your results. Multi-Stage CI is an extension of the common practice of shielding others from our changes by only checking-in when we have tested our changes and only updating our workspace when we are ready to absorb other people's changes. With Multi-Stage CI, each team does a team-based CI first and then cross-integrates the team's changes with the mainline on success. This limits project-wide churn and allows CI to scale to large projects. This session will introduce Multi-Stage CI, cover the pros and cons, and give examples of how to
implement it.

Scaling Agile to Large Teams

Read what Scott Ambler and I have to say about scaling Agile for large teams in "Agile Development: It isn't just for small projects" from SearchSoftwareQuality. Scott has an interesting point about bureaucracy versus discipline which I haven't heard anybody make before.

Deep Agile 2008 - Reminder

In case you haven't heard about it, or forgot to register for this great event, you need to register for the Deep Agile 2008 event before the price goes up! If you register before October 14th, you'll get the early bird price of $500, which is $100 off the standard price. And for readers of my blog, when you register, enter the discount code of DA20087UPAN for a further $50 off. You’ll pay only $550 – a full $150 savings off the standard price!

What: Deep Agile 2008: Not as Easy as you Thought!
When: November 8-9, 2008, 9:00am to 5:00pm (Registration opens at 8:30am)
Where: MIT, Cambridge, MA - Room E51-345 Hosted by Jay Conne

Space is limited to 90 in this MIT seminar room and it has been selling very well. So register early! Details are on the Agile Bazaar website.

Deep Agile 2008 - Not as Easy as You Thought!

A 2-Day Intensive Conversation With James Coplien and Bob Martin

The sparks will fly when two passionate professionals - Jim Coplien and Bob Martin - square off to make the world safe for software development.

In the corner of architecture, patterns and agile is Jim Coplien.
Driving the necessity of test based design is "Uncle Bob" Martin.

Each will use their long track records, numerous case studies, and success stories to argue that they have the answers you need to deliver successful projects and products. The difference here is that we are presenting both sides of the story, and working with Jim and Bob to show how both approaches meet in the arena of professional software development.

Come prepared to be surprised and have your assumptions questioned! Our goal is to get well beyond the buzzwords and introductory agile ideas, and to get you thinking.

What: Deep Agile 2008: Not as Easy as you Thought!
When: November 8-9, 2008, 9:00am to 5:00pm (Registration opens at 8:30am)
Where: MIT, Cambridge, MA - Room E51-345 Hosted by Jay Conne

Only $495 at the Super early bird rate. That expires on Sept 23 so don’t put off registering! And for readers of my blog, when you register, enter the discount code of DA20087UPAN for a further $50 off. You’ll pay only $445 – a full $205 savings off the standard price!

Space is limited to 90 in this MIT seminar room and we are expecting to sell out early. So register early! Details are on the Agile Bazaar website.

Software is Indistinguishable From Magic

Plush red curtains withdraw to the sides of the wide stage, only the dark of midnight can be seen beyond them. A tall figure with a top hat strides into the spotlight at the center of the stage. The weight of anticipation presses us into our seats like breathless astronauts at take-off as we wait for something “never before seen by any audience.”

The magician raises both hands into the air as though preparing to pull himself up on an invisible bar. After a pause just long enough to hear your heart beat loudly twice, the magician pulls down hard on nothing until his knuckles rap the floor. Lightning flashes and thunder cracks. The stage is filled with a pyramid of elephants, maidens juggling bowling pins, and a flock of doves darting over our heads and towards the back of the theater.

Software is indistinguishable from magic. It gives us the power to conjure new capabilities at the click of a mouse. Magic springs from our fingertips. We move electronic mountains of information, we uncover patterns that were previously invisible. New revenue streams well up out of the internet. Users eagerly try new software, hoping for a magical experience. Developers demonstrate new software reveling in the reaction of the users, journalists, and bloggers.

Timing is everything. The only magic of poor timing is a disappearing audience. When the search engine Cuil was launched, users of Google everywhere wondered what sort of new trick Cuil had up their sleeves. 120 billion indexed pages seemed unfathomable. Unfortunately, Cuil’s timing was off and the effect fell flat. Just as it would be if you saw a wire during a levitation act, Cuil was revealed to be ordinary software with bugs and scalability problems.

When the timing is right, as it was with the introduction of the first iPhone and its amazing user interface, software can send a tingle up and down your spine. I still remember my first experience with software. It was a simple Tic-Tac-Toe program at the Boston Museum of Science in the early 70’s. There were just a few stations and you had to wait your turn in line. It was amazing to me that a computer could play Tic-Tac-Toe against a person and win.

I kept going back for more until Dad asked me if maybe I’d like to visit some of the other exhibits. I suggested we go to the gift shop because I knew they had a plastic mechanical computer kit for sale. It was called a DigiComp and it could be programmed to do simple computation including playing the game of Nim. It was pure magic.

What was your most magical software experience? Post your comment below!

Part 2: The Magic of Demos

The Magic of Demos

When I started writing software, I enjoyed the thrill of showing people something they hadn’t seen before. Even today, one of the main reasons I enjoy working in the software industry is the thrill of demoing new software. When you demonstrate new software, you become a magician, conjuring feats of computation that dazzle the imagination. The audience starts out skeptical, wondering if you are just a two-bit side-show act. You slowly build up to the main event and then, when you’re lucky, they gasp in amazement as you show them something that they’ll no longer be able to live without.

One of my favorite demos was many years ago when I was showing an early version of a product to some folks for feedback. As part of the demonstration I interrupted the power to the laptop (with the battery already removed), and showed them that the software continued as though nothing had happened when the power was restored.

Even though we had told them that the software wouldn't ship for at least six more months, they called us the next day to place an order anyway. For them, the value outweighed the risk. We decided to accept the order. That early exposure to a real customer changed the way we thought about things. Even though we considered the product to be pre-release, we made sure that every new feature worked as it was developed instead of waiting until the end game of the official release. As a result, the end game was much smoother than we had expected.

The Magic of Agile Development
From a business perspective, the main reasons I appreciate Agile development are an increase in quality and ROI, more options, and higher visibility into progress and status compared to traditional development. But from a purely personal perspective, the reason I enjoy Agile development is because it made my job more fun.

Today, thanks to Agile development, I interact with customers more than ever before. As a product owner, I do more demos and am able to provide new features that hit closer to the mark faster and more frequently than ever before. This in turn means more oohs and ahs from customers which is more fun for me and more profitable for the business.

Related:

• Software is Indistinguishable From Magic
• Zero to Agile in 90 Days or Less

The Simplest Thing That Could Possibly Work

One of the principles of Agile, mostly related to design and architecture, is “The Simplest Thing That Could Possibly Work.” This is sometimes taken as a license for cowboy coding. But that is not the intention. A better way to express it would probably be something like “The Simplest Solution That Could Possibly Satisfy Your Requirements.” For instance, if you have a requirement to create the back end for a web site like amazon.com, then while a perl/cgi solution on a single core machine could possibly “work,” it doesn’t work from the point of view of high availability, fast response time, or reliability.

From Oversimplification To Rube Goldberg
On the one hand, there is a wide spectrum of complexity of construction ranging from doing nothing to Rube Goldberg level complexity. On the other hand, there is the set of solutions that work, meaning that they meet all of the requirements. TSTTCPW refers to the solution which works and which is lowest in complexity.

Part of being simple means simple to read, maintain, use, design, understand, and implement balanced against the time it takes to get the job done. Spending too much time to create the ultimate in simplicity starts to get you into a different kind of trouble.

As somebody that struggles to apply this principle on a regular basis, I was happy to stumble upon an example of this principle which can be captured in a picture and kept in mind as I am working on a new design. Perhaps you will find it to be useful food for thought as well.

A Bridge Too Far
There’s a construction project that you’ve probably heard of which is affectionately called the “Big Dig.” Part of this project was the construction of the “Leonard P. Zakim Bunker Hill Bridge” aka the “Zakim bridge.” This part suspension bridge, part cantilever bridge is an enormous one of a kind architectural marvel. It supports five lanes of traffic in either direction for a total of ten lanes. It was built at a cost of approximately $11M per lane.

Running parallel to the Zakim (to the left in the photo) is the Leverett Circle Connector Bridge. It serves a total of four lanes of traffic. It was built at a cost of approximately $5M per lane.

Part of the requirements for the Zakim bridge were clearly “create a stunning new Boston landmark.” On the other hand, the Leverett Bridge is a very simple but also very strong bridge. It could have been made even more simply, but not without a safety risk and/or a shorter lifespan. In other words, it is “The Simplest Thing That Could Possibly Work.”

Next: The Faberge Egg

TOC: Zero to Hyper Agile in 90 Days or Less

[Note: revised 7/3/08 to reflect comments on reddit. Clearly the original post didn't work. :) ]

Many Hands Make Light Work, But I’ve Only Got Two

When choosing how to allocate resources, it can be difficult to do an effective cost benefit analysis in a short period of time. A technique that I stumbled upon is to look at things from the perspective of “what would I do if I was the only person on the team?”

In 1992, I convinced my manager at the Open Software Foundation (OSF) to create the position of tool smith and allow me to work on the OSF Development Environment (ODE) full time. As I now had much more time for doing development and made many more changes to ODE than I had ever made before, there was much more testing to be done.

Initially, I didn’t realize just how much testing was fully required for ODE. Not only were there no automated tests for ODE, there were no documented test cases either. The test plan consisted entirely of “round up as many volunteers as you can and have each volunteer test on one of the nine platforms.” The testing that each volunteer did was entirely up to their discretion. It was different volunteers every time, and there was no record of what they did, only a “seems fine to me” or “I found these problems.” Once the number of problems got down to an acceptable level, we’d ship the new version.

After a couple of releases it started to sink in that I had to do something differently. My first attempt was to document a standard set of test cases. At first this seemed to work really well. Testers commented that it was much easier and took much less time. I felt like I had gotten a consistent set of results from a consistent set of tests. But a test/fix cycle requires running the same tests over and over again until things stabilize. Following the same steps over and over again can become pretty mind-numbing. Pretty soon I couldn’t get the volunteers I needed and I was starting to suspect that people were skipping steps.

I also discovered another problem. As bug reports came in from production use, and as I thought of test cases that were missing, the list of test cases mushroomed. I was very careful not to include overlapping test cases, but still the list grew and grew and grew.

Since the QA of the tool was ultimately my responsibility, I had to pick up the testing slack. The combination of the ever increasing list of test cases and the diminishing pool of volunteers soon made the QA of a new release pretty much impractical. I would end up spending much more of my time testing than coding.

But then I remembered how I had gotten the job of tool smith in the first place, by automating myself out of my previous job of manually kicking off builds on all platforms. Test cases are basically a set of steps to take and the expected results. Automating test cases is basically coding, and that was much more fun than manual testing. A couple of inspired weekends later I had automated all of the test cases and added many more new ones as well.

That was the last time I ever relied on manual tests.

Read more from: "Zero to Hyper Agile in 90 Days or Less"

Reinvest in Your Engine by Improving The Work Environment

There are really only five ways to increase the profitability of a business based on software development: reduce costs via outsourcing, reduce headcount, reduce other expenses, increase productivity or increase revenues. Reducing expenses can only go so far. The most expensive part of software development is the people. Thus, one of the most successful ways to increase profits is to increase the productivity of the software development team.

The Agile Workplace
At Litle & Co., developers like the fact that Agile provides the additional challenge of solving business problems instead of just technical problems which requires thinking at a higher level. Developers at Litle report that they have a higher level of job satisfaction than in previous companies that were not using Agile because they see the results of their software development efforts installed into production every month. Also, they like the fact that there is much less work which amounts to “implement this spec.”

Your development infrastructure is really no different than the general company infrastructure which includes your cube or office, the carpet, the artwork on the walls, the company cafeteria, your phone, your computer, and the company location. These are all part of your work environment. If you have a computer that is 5 years old, your work environment is not as good as if you have a computer that is only 2 years old. If you are writing in C rather than C++, C# or Java, your work environment is sub-optimal.

The closer that your development infrastructure is to the ideal environment for your circumstances, the more productive your team will be. This principal extends to all aspects of the development environment, from development language, to build system, to build farm, to issue tracking system, to the process that you follow.

Next: Your Development Process is Part of Your Work Environment

Your Development Process is Part of Your Work Environment

Your development process (regardless of how it is implemented), is also part of your work environment. If as a result of your development process you regularly end up redoing work because problems weren’t discovered until just before the release, or projects get cancelled or shelved, then this is also likely to reduce productivity and job satisfaction. As this process improves, so does your work environment. The smoother it operates, the more pleasant your working environment will be.

There are many problems which you may think of as being unrelated to your development process. For instance, broken builds. Broken builds are simply the result of somebody making an idiotic mistake, right? Perhaps that’s true some of the time, but most of the time it is due to the complexity of integrating many changes made by many people for software that has many interdependencies.

To be sure, a “perfect” process does not guarantee happiness, success, or the absence of problems. You still have to debug complicated problems, port to new platforms, deal with unforeseen circumstances, etc. However, the state of your process impacts the efficiency with which your effort is applied.

If your process is perfect and completely frictionless, then 100% of your effort will be applied to the work that creates value. If it is rife with problems, it may mean that only 50% (or less!) of your effort will be applied to work that creates value. If there are problems with the process, then you are already expending effort which is essentially wasted. You would be better off investing some of that effort in removing the problems permanently instead of losing it to friction on a regular basis.

Next: Quick Summary of The Benefits of Adopting Agile

The Benefits of Adopting Agile

The benefits of moving to Agile development can be split into two categories: benefits to the organization, and benefits to you personally. As a result of increased productivity, higher quality, and responding more rapidly to market demands, Agile can provide the following benefits to the organization:

• Increased revenues
• Reduced costs
• Increased market share
• Higher customer satisfaction

Each of these benefits lead to a stronger organization which is then in a better position to reward you for your efforts both directly and indirectly. Some of these benefits include:

• Getting a raise and/or bonus
• Having more discretionary income to buy cool stuff
• Improving your working conditions
• Actually using all of your vacation time
• The opportunity to spend more time working on cool stuff

In addition, Agile can provide the following direct benefits:

• A less stressful environment
• Less canceled or shelved work
• Career advancement due to learning new skills
• Having the resume that gets you your dream job

Many of these same claims have been made in the past about tweaks to traditional development, but nothing ever came of it. Why will this be any different with Agile? Let's take a look at why attempts to "fix" traditional development haven't panned out.

Next: Having Dev Team Performance Problems Using Traditional Development? Try Niagra!

Agile Development Scenario

Here's an Agile development scenario. For simplicity, let’s use two month iterations. Marketing says the three features with the highest ROI are a Facebook plug-in, a Second Life plug-in, and an RSS feed plug-in.

You start with the Facebook plug-in. You plan. You design. You create a test plan while the code is being written. You discover potential problems and deal with them. You automate those tests while the code is being written. As the code is written, it is integrated, built and tested continuously; problems are found and fixed immediately. At the end of development, the only problems that remain are the ones that could only be found at the end of development. If you wanted to, you could cut a new release with very little overhead.

Marketing announces that the Second Life plug-in is not as marketable as they had hoped and iPhone support is showing signs of becoming very lucrative. So, you start on the RSS feed support. Now marketing says the Second Life plug-in is worthless but iPhone support is hot, so you add iPhone support. During the whole process, you kept your options open. At the end you have produced more business value than using the traditional process, and you were in a position to start realizing that value much earlier.

To get the primary benefits of Agile, as described above, you don't have to use 3x5 cards or pair programming, and you don’t have to do frequent releases. These things are optional and you can use them or not according to your preference and circumstances.

Next: Achieving the Primary Benefits of Agile

Traditional Development Scenario

A useful way to describe Agile is by contrasting it with traditional development. To do that, let's consider a typical development scenario: adding new features to stay competitive. First we'll look at the scenario using traditional development and then using Agile development.

In a traditional project, you have a known timeframe for major releases. Too short and you’ll spend too much time on overhead, too long and you’ll miss opportunities. For the sake of argument, let’s pick a timeframe of six months and say that allows you to provide three “big features.” Marketing says the three features with the highest ROI are a Facebook plug-in, a Second Life plug-in, and an RSS feed plug-in.

Midway through the design process, marketing announces that the Second Life plug-in is not as marketable as they had hoped and iPhone support is showing signs of becoming very lucrative. You think, oh well, nothing we can do about that now. Just after you finish coding marketing declares that the Second Life plug-in is going to be a complete flop and when can they get iPhone support?

Now that the functionality has settled down, QA starts writing tests and running them. Planning and development took longer than expected and the release deadline is looming. Testing time is compressed, QA concentrates on the most critical stuff and gives the rest a spot check. Here’s a mystery. If your full test cycle takes two days, why does testing take a month? The answer is that you aren’t really doing a month of testing. Problems have been creeping in all along the way that you are just now finding out about and it takes many test/fix cycles to expose them and fix them. Once the find/fix rate gets down to an acceptable level, you declare victory and deliver the new release.

Next: Now Let's Try Agile

The "Faberge Egg" Widget

There was a developer that worked for me once, I’ll call him George. He wrote a lot of really good code. But one day he decided he wanted to make his mark on things. George believed that the way to do it was to create a beautiful widget. He wanted it to be so beautiful and so useful that it could be used generically and would be something that we could sell separately on its own merits.

The widget was part of a Java/Swing user interface for an issue tracking system. It was responsible for the data model used by a form, a query editor, and a handful of other objects. So, it did need to be fairly general purpose, but it didn't need to be so generic and so functional that it would be something that people would want to buy separate from the application.

George referred to this widget as his “Faberge Egg.” It was an apt name for the widget. It was overly ornate, intricately detailed, and supported a very wide range of functionality that we had no immediate use for and still don’t to this day.

I was very clear with George that I only wanted a “Cadbury Egg” and tried hard to convince him that he could provide much more value to the company in other ways and the company would reward him for that, not the Faberge Egg. I like folks to have the freedom to grow and explore. Even though we were under deadline pressure, I gave George some wiggle room while at the same time trying to guide him towards the simplest design that would work. Unfortunately, after weeks of work and many conversations, George’s desire to produce a masterpiece of a widget prevailed.

Over time, that widget has been the source of more than its share of problems, both in bugs and in making it more complicated for other developers to maintain it and extend it. It has been partially refactored several times, but it seems there has never been the time to really do the full job required.

The same functionality that the widget provides was needed in our new web interface. Instead of reusing the code as we have done for other functionality, we decided to start from scratch. Doing just what was needed for the job at hand took only two days and that code has been very reliable right from the start.

Next: Frequent Releases Improve Code Quality Faster

Related: The Iterative Design of a Transforming Lego Sports Car

Do You Need a Standup Meeting?

Stand-up meetings are a great way to reduce delays in communicating important information. Another benefit of stand up meetings is the elimination of time-wasting status and progress meetings.

Stand up meetings are most closely associated with Scrum and are called “Daily Scrum Meetings” within Scrum, but have become populare independent of any particular methodology which is a good indicator of suitability for mainstream use.

A stand-up meeting is simple to implement. There are just a handful of guidelines:

• Limit the time to fifteen minutes.
• Pick a regular time for the team to meet, preferably in the morning.
• Start on-time regardless of who is absent.
• Each person answers these three questions:
• What have you accomplished since the last meeting?
• What are you working on next?
• What impediments do you have?
• All discussion and problem solving is deferred until the end of the stand-up meeting.
• Follow-up as needed in smaller groups

Although it is called a stand-up meeting and standing is encouraged, the time limit is the most important part and standing is optional.

The point of a stand-up meeting is to improve communication and to discover and resolve impediments, not to have a meeting just for the sake of having a meeting. If the team feels that other practices make the stand-up meeting redundant, then by all means reduce their frequency or even discontinue use until such time as it appears to be necessary again.

To help make this decision, let’s take a look at the expense side of stand-up meetings. First, people have to get to it. And then they have to get back to their computers. Scrum discusses how to minimize this time, but practically speaking, there is more overhead than just the ideal 15 minute meeting. If you are at a larger company, somebody has to book the room and let people know where it is. Let’s call the cost of the meeting 20 minutes per person. If you have 12 people in a stand-up meeting, that’s 4 person hours per day. That’s the equivalent of half of a person. Those meetings had really better be worth it!

Now let’s take a hard look at the stand-up meeting itself. One of the basic ideas of Agile (and Lean) is continual self improvement. If the value of the meeting exceeds the cost, then there’s no problem with the meetings, especially if they are eliminating other meetings. If the stand-up meeting is the only remaining meeting, that seems like a good thing. However, continuous improvement means we’re never satisfied. Now that you are down to just the one meeting, you should still ask the question: “is it providing more value than the cost? Is there a better way?”

What is the purpose of a stand-up meeting? To quickly find out if people are getting their assigned work done and if not why not. If it is more efficient to do that via e-mail, IM, an issue tracking system, or other means, then use those means. Someone might say “but seeing folks face to face is worthwhile.” Ok, so why not just do that then? Go out to lunch together or something like that.

Or perhaps the stand-up meeting is needed because otherwise folks wouldn’t complete their work, or people wouldn’t speak up when they run into an impediment. In that case the stand-up meeting isn’t a solution at all, it is a crutch. For instance, perhaps somebody isn’t completing their work because they don’t like it, but the constant peer pressure of the standup meeting is goading them into completing their work anyway. So then the real problem is lack of job satisfaction or low moral or something along those lines. Until you fix that problem, the stand-up meeting is just acting as a band-aid.

The real measure of project status and health is having an increment of shippable value at the end of every iteration. A standup will only expose problems that are on people’s minds, but the forcing function of the increment of shippable value is where you will get the true picture of how things are going. A one month iteration interval is good, but if you can get it down to 2 weeks or even 1 week, that may do far more to expose real problems than a standup will.

Next: How Agile Helped Litle & Co. Get to #1 on the Inc. 500

Sustainable Pace: Supply vs Demand

In a traditional project, the demand for resources from the four major aspects of software development see-saws dramatically over the course of a project. These aspects are project management and planning, architecture and design, development, and QA. You need resources on hand to serve the peak demand level, but during periods of low demand those resources will either be idle or used for activities which have a lower ROI.

A common circumstance is that there are insufficient resources on hand for the peak demand level and so people end up working in “crunch mode.” During crunch time, people tend to make more mistakes. Agile levels demand out over time and removes this see-saw effect which simplifies resource planning and removes the need for crunch time.


In the figure above, the straight green lines represent the resource load in an Agile project and the zig-zagging purple lines represent the resource load in a traditional project.

With traditional development, delays during development compress most of the testing to the end of the process which then requires taking shortcuts due to schedule pressure. I used to think that one way of compensating for insufficient QA resources was to delay the release until QA finishes. On the surface it seems to make sense. But only if the folks writing code sit on their hands while QA does their work. Ok, so you have multiple projects and the developers work on another project. But then they finish that. Now QA starts on the second project and the developers move to the third. The problem is still there.

On the other hand, as a result of the need for increased QA resources during testing, you may have two other problems. If you have enough QA resources to handle the pressure of the endgame, you may have too many QA resources during the rest of your development cycle. Alternatively, you may bring on additional QA resources on a short-term basis to compensate. Both of these options are obviously undesirable.

There’s a natural balance between the amount of effort required for developing something and the amount of effort required to QA it. No matter what you do, if you have the wrong ratio of development resources to QA resources, it will cause problems. If development creates more than QA can absorb, you will create a backlog of QA work that will always grow.

There are six options for dealing with a QA backlog: do less QA than needed and thus shift the burden of finding problems that you could have found onto your customers, increase your QA capacity, decrease your development capacity, have development idle, have development help with QA or allow the backlog to grow. The larger your testing backlog, the longer it will take to ship it and the greater your opportunity cost.

The imbalance may be in either direction. After you transition to Agile development, you may find that you have more QA resources than are needed. In that case, you have the option of having QA take on some of the work currently done by developers. See The Role of QA in an Agile Project for more on that topic.

This natural balance holds between all four aspects of software development. Depending on your organization, there may be an imbalance between supply and demand at any stage in the pipeline. Wherever there is an imbalance you have the same six options as described above. For example, you may end up with project plans that are never used, developers idle because the design isn’t ready yet, etc. To the extent that some of the resources are actually the same people you can use that fact to manage this problem.

When using short iterations, resource imbalances are easier to detect and correct. Having balanced resources means that all development activities are done consistently and on a regular basis and there is no need to take the shortcuts that are typical of traditional development.

Next: The Usability of Short Iterations

TOC: Zero to Hyper Agile in 90 Days or Less

Agile Adoption Stage 2: Establishing a Natural Rhythm

The first stage in adopting Agile is Preparing for the Transition to Agile.

The next stage is to make some early and measurable progress to encourage adoption. As it is, you have actually already experienced some success. You’ve planted the idea of process improvement, you’ve got a map of the areas of general consensus and contention, and you’ve provided a forum for people to learn more about the current process, uncover misunderstandings and correct them.

One of the areas of consensus you’ve uncovered is very likely a problem which everybody agrees should change, there is a consensus or near consensus on how to improve it, enough pain to easily instigate a change, and a sufficiently small scope that the change can be made in a relatively short amount of time. That should be your first area of attack. It doesn’t really matter what it is or if it has anything at all to do with Agile. At this point, you just want to establish the idea that process improvement is worthwhile, doable, and worth doing on a regular basis.

Short Iterations
The main point of this stage of adoption is to discover the natural rhythm of your team and the project that you are working on. It may be a month, it may be a week, or it may be some other timeframe between 1-6 weeks.

For your first iteration, start with a goal of a one month iteration time. It is better to make it a goal rather than a hard requirement. Making it a hard requirement will encourage rushing to finish or taking shortcuts at the end which will defeat the purpose of establishing a rhythm. It will take time to get used to short iterations.

Don’t plan to do a release from the first iteration. Whichever iteration you do plan to do a release from, keep your release process the same for now. The purpose of the first couple of iterations using Agile is to identify obstacles in order to start removing them.

The Backlog
To determine what to work on for the first iteration, make a conservative estimate of how much work your team can take on. Create a backlog of the highest priority items, adding estimates to them as you go in order to determine the cutoff point. You don’t need to have a product owner or ScrumMaster or make any changes to how you plan a release at this point. Just do whatever you would normally do to plan the contents for a release, but keep the timeframe to a single iteration.

If there are one or more work items which are too large to fit into a one month iteration, you have a variety of options at this point. First, you can postpone going to short iterations until they are done and concentrate on the other aspects of the first stage of adoption. I don’t recommend this. If you’ve gotten to this point, this is your chance, don’t postpone it, find a way to move forward.

Second, if the work items would fit into a slightly longer iteration, use that as your iteration time. It isn’t ideal, but it is a better alternative than giving up!

A third option is to delay the oversized work items until after you’ve got 1-2 iterations under your belt. If they are in your backlog, somebody thought they were important, but if your release isn’t for many iterations anyway, perhaps it won’t matter to the overall release.

Done Means Done, But What Does "Done" Mean?
You should have a clear definition of what “done” means for a work item. At a high level, any definition of done should include whatever you believe is necessary for that work item in isolation to be considered shippable. Think about your entire release cycle, not just what it takes to get something ready to check-in. Do you eventually do a code review? Then that should be a pre-requisite for “done.” Other candidates are: documented, tests written and all pass, demoed by a QA person to a third party (that is, not just the developer and the QA person themselves).

Iteration Retrospective
Before you get too giddy with the success of your first iteration, be sure to schedule a retrospective. This is exactly same as the traditional “post-mortem” but doesn’t have the same air of death about it. The purpose of the retrospective is to take a look back at the iteration and look for opportunities to improve your next iteration. What went wrong? Why did it go wrong? What went well? What ideas do people have for improving the next iteration?

Most likely, the duration of your first iteration overshot your goal. This should be the focus of the retrospective. If in fact you did make the goal of 1 month, what could you do to reduce the time to 2 weeks? What could you do to have a higher ratio of value-producing activities such as adding more new functionality? You should not change your goal of 1 month to 2 weeks, even if you made the goal. At this point, it is more important to establish a rhythm rather than reduce the time of the interval.

At the end of the retrospective, decide as a team which changes you want to make in the next iteration. Don’t try to change everything at once, concentrate on the top 3-5 ideas. Make sure that somebody is taking notes. The notes will be invaluable in your next retrospective for detecting trends.

Repeat
Breaking down entrenched habits and ingrained beliefs while simultaneously establishing new habits and beliefs takes time and patience. The most important things at this stage are to establish a rhythm via the short intervals of the iterations and to find and remove obstacles. Because you should already be experiencing positive results, there is no need to rush to the next stage and endanger the progress that you’ve made. Remember that there are lots of people watching the success or failure of this process and support for a new way of doing things is still shaky at best. That includes you and everybody else on the team! I recommend you simply repeat the process with another 2-3 iterations, finding and removing obstacles as you go.

Considering a transition to Agile? Read more of "Zero to Agile in 90 Days or Less."

Preparing for the Transition to Agile

Once, when I was just starting to snowboard, I was at Sugarloaf for the weekend and they had very little cover and very few trails open. But then Saturday night, they got 33” of powder. A friend and I came to a trail that was closed. It looked like a great trail; endless powder with no tracks. The problem was that it had been rocks and grass the day before and there was no base underneath, so it was just the same as riding on rocks and grass. It was not a pleasant experience. Adopting Agile without understanding it and without creating a proper ecosystem for it is destined for a similar fate.

Adopting Agile development requires breaking down mental barriers and building up new skill sets. There is nothing particularly hard about actually doing any of the Agile practices, but many of them are counterintuitive and do take a bit of practice to get used to. That said, don’t underestimate the amount of effort required. The effort required is at least on the order of taking a team which is very used to writing in C++ and moving to Java. There’s nothing particularly difficult about such a transition, but there are many subtleties which must be learned and it takes a while to build up the same base of experience.

Self Education
Before getting too far along, make sure that you have done your homework. Read other books on Agile, find other folks in your organization that have done Agile development before. Go to conferences, join the local Agile user group, become a certified Scrum Master, do whatever you do to find people that you can lean on when you need it.

Scope
Determine the best scope of the adoption. As with most things, it is best to think big, but start small. Is there a small project with no more than 12 people that is amenable to piloting something new? There are two advantages in starting small: minimizing disruption and leveraging what the pilot group learns about doing Agile in your environment when transitioning other groups.

Scouting
Agile development has certain perceptions related to it. One of the most prevalent perceptions is that it is “for small groups.” That was certainly my perception when I first started hearing about it. Another perception is that small iterations aren’t a good thing because customers don’t want frequent releases, there’s more overhead involved, the quality will be lower that way, and it makes marketing’s life more difficult.

If you just advocate Agile without knowing the landscape, you run the risk of alienating the people whose support you need in order to go Agile. Find out how receptive your organization is to going Agile. Think about who is in a position to help or hinder its adoption. Those are the key stakeholders. You will need to find out where they stand, what they like about the idea of going Agile, and what their objections are. This information will come in handy later in the adoption process.

Prepare Your Organization
Once you have a basic lay of the land, see what you can do to raise people’s awareness and understanding of the advantages and potential pitfalls of Agile. Do a presentation for folks that are interested, invite in somebody from the Agile community to do a presentation or workshop. Recommend books and websites that you found helpful.

Transition Development and QA Together
The most important component of reducing the rework period that comes from long iterations is improving your testing process. For many if not most organizations, this is the hardest goal to achieve in practice.

If you don’t have any automation at all, it is a good bet that there is an ingrained belief that automated testing is either a bad idea, doesn’t work as well as manual testing, is too expensive, or that the tools are too expensive. As a result, it may be that there are no QA automation experts in the building and possibly nobody with scripting skills in the QA group. The best course of action in this case is to concentrate on introducing the idea of QA automation.

If it is clear that there is a bias against automated testing that is too strong to overcome any time soon, another tactic is to have the development organization champion automation with an eye towards handing it over to QA once the idea catches on. A good place to start is with unit tests. It should be clear from the start that your goal is to have QA own test automation. Developers will write good tests, but they are too optimistic by nature. Developers start from the assumption that “it will work.” QA people start from the assumption that “it doesn’t work and I’ll prove it to you.” Pessimism is a good trait for a person creating tests.

Keep Your Healthy Skepticism
Think carefully about the value of each practice that you plan to adopt and make absolutely sure that it is appropriate for your exact circumstances before you adopt it. You shouldn’t be adopting practices simply for the sake of saying that you are adopting Agile practices. Every practice you adopt should be because now that you know about it you simply can’t imagine getting by without it.

Don’t Throw Out the Baby With the Bath Water
I’ve seen many first-time Agile projects fail because they threw out everything they already knew about developing software. Most of the individual steps of developing software with an Agile methodology are the same as traditional software development. You still have to talk to customers, decide what you are going to do, write code, write tests, do testing, etc. Agile development is simply a different framework for those steps. You have a business to run, and you don’t really need to introduce large and sudden risk factors. Before you decide to chuck everything that you know and start from scratch, spend some time developing your knowledge and understanding of Agile. Look closely at your existing practices and see which ones will fit well within Agile, and which ones may cause problems. Create a game plan and start out gradually.

Next: Agile Adoption Stage 2: Establishing a Natural Rhythm

How Does Choice of Methodology Influence Strategy and Tactics?

Once More Unto the Breach, Dear Friends
As Helmuth von Moltke (the Elder) said, “No battle plan survives contact with the enemy.” In the case of software development, the “enemy” is reality. What we think we need to do to satisfy customers and how we think we need to implement it generally changes as we implement and as we get customer feedback.

Similarly, Dwight D. Eisenhower said, “In preparing for battle I have always found that plans are useless, but planning is indispensable”. Planning (short and long, tactical and strategic) is still important in the Agile world. The difference is that Agile is specifically designed to take advantage of the fact that planning, design, and development have a learning component and to accommodate and embrace the resulting need to change plans on a regular basis. As you discover new information during an iteration, you can easily adjust your plans to take what you have learned into account in future iterations.

Too Busy to Plan
Short iterations and other Agile practices such as “You Aren’t Going to Need It” (YAGNI), may seem to indicate that Agile is tactical in nature. However, I would say that it is when looked at through a Waterfall lens. For instance, with Waterfall the planning horizon and iteration length (the full start to finish time) are the same. So if you look at Agile with this perspective you might think that the longest that Agile folks look ahead is 30 days. You might then conclude from this that Agile can only be used tactically. But in fact there is no connection between the planning horizon and the iteration length.

Many teams use very short iterations; two weeks or one week. What happens within an iteration is whatever has been planned for that iteration. Different Agile methodologies use different approaches for planning iterations. In any case, you can plan as far out as you would like. As an Agile product owner I can tell you that some of my products have plans going out multiple years.

Sometimes there is work that can’t be fit into a single iteration (whatever iteration length you have chosen). In that case, you can use multiple overlapping iterations. For instance, a one-week iteration for most stuff and an overlapping one month iteration for the exceptions. Over time though, teams generally get better at avoiding the need for exceptions.

Wherever You Go, There You Are
In my experience, whether the software development of a company is done tactically or strategically is connected to the planning culture of an organization, not the methodology. If an organization thinks and plans strategically, that will be reflected in their software development whether they are using Agile or Waterfall.

I’ve seen plenty of companies that use Waterfall but think tactically, packing mostly tactical features into a large release. Feature creep is a good example of how tactical thinking is fairly common in Waterfall projects. While of course it is true that “that shouldn’t happen” the facts on the ground are that it does.

I’ve also seen plenty of Agile projects that were strategic in nature. For instance, an Agile company that thinks strategically will have a strategic plan which they carry out using multiple iterations. As they discover that they need to do something tactical in the middle of their strategy, or they need to change their tactics in support of their strategy, it is simple to do with short iterations. They just change their plans for future iterations. As soon as their current iteration is done, they begin the next iteration which has now changed to take into account the new information.

It may well be that folks that think and plan strategically are less inclined to look deeply at Agile because it appears on the surface to be tactical in nature. I know that was true in my own case. I believe that people and organizations that think strategically will bring that thinking with them into the Agile world. Having crossed over I can report that I have been able to very happily and successfully bring strategic thinking with me to the other side.

Read more about Agile in "Zero to Agile in 90 Days or Less" .

[Note: this post was inspired by Jordan Bortz’ post, Agile and Waterfall are Two Sides of the Same Coin where he states that ‘ “Waterfall” is basically a Strategic method of attaining goals, and “Agile” is basically a Tactical method for achieving goals.’ ]

An Agile Case Study: Litle & Co.

As part of the book that I am working on, "Zero to Agile in 90 Days or Less" I decided to include some case studies to illustrate how Agile is different, the problems that people run into and the solutions that they find, as well as the results. One of the companies that I discovered in my search for case studies was Litle & Co. I was immediately struck by their high level of expertise and success with Agile. I couldn't wait to hear how they did it, their trials and tribulations, and I am very grateful to them for allowing me to use them as a case study. The case study speaks for itself, so without further ado...

Case Study: Litle & Co.
Litle & Co. is a leading provider of card-not-present transaction processing, payment management and merchant services for businesses that sell directly to consumers through Internet and Multichannel Retail, Direct Response (including print, radio and television) and Online Services. They interface with the world’s major card and alternative payment networks, such as Visa and Master Card, and PayPal, and must be available 24/7 with no downtime.

During the founding of the company in 2001, the initial team of six developers wanted to use Extreme Programming (XP), an Agile development methodology. Some of the developers had used it before and the rest had read about it and liked the ideas. When they talked to Tim Litle, the founder and Chairman, about using Extreme Programming (XP) they told him “you’ll have to accept that we won’t be able to tell you exactly what we will deliver in 9 months.” Tim said, “That’s fine with me, I’ve never gotten that anyway!” He agreed not only to use it, but to insist on it as the company grew and brought in additional management. He liked the idea that he could change his mind about prioritizing feature development for merchants without cancelling or shelving work in progress.

In 2006, Litle & Co. landed at No. 1 on the Inc. 500 list with $34.8 million in 2005 revenue and three-year growth of 5,629.1 percent. In 2007 Inc. magazine cited Litle & Co.’s 2006 revenue as $60.2 million, representing a three-year growth rate of 897.6% over its 2003 revenue of $6.0 million. How has Litle achieved these impressive results? One factor that they cite is their use of Agile development.

Litle uses many of the XP practices including pair programming. The director of software development, David Tarbox, said “at first I thought I would hate pair programming, but I quickly came to enjoy it and depend on it.” Some of the side benefits that they see in pair programming are better on-boarding of new developers due to the mentoring and training that is inherent in pairing. They also like the fact that all code was developed by two people which provides both redundancy for knowledge of all code and real time code review. It may be that one of the developers for a piece of code is on vacation or otherwise unavailable when somebody needs to talk to them, but it is rarely the case that both are.

They started with weekly iterations then moved to 2 weeks then 3 and are now monthly. They gravitated to a month because it was the easiest cadence for everybody to work with and it meant that the whole company was able to adopt that same cadence. Agile development is part of the psyche of the whole organization. It permeates every aspect of the organization.

One thing that other areas of the organization had to get used to was that they had to be very clear about what they wanted from the development organization because there is no slack time. If you want something and you expect to have it done in the next iteration (monthly cycle), you have to be able to state it very clearly or it won’t even get started. It also means that development is very closely connected to the corporate strategy. For every task, developers ask “is this aligned with our overall direction?”

Agile has had a very positive effect on their hiring. Developers like the fact that Agile provides the additional challenge of solving business problems instead of just technical problems which requires thinking at a higher level. Developers at Litle report that they have a higher level of job satisfaction than in previous companies that were not using Agile because they see the results of their software development efforts installed into production every month. Also, they like the fact that there is much less work which amounts to “implement this spec.”

Litle’s software is hosted at multiple off-site datacenters and they upgrade their software every month like clockwork. In reality, they are doing ~7 week iterations which overlap such that they are releasing monthly. The first week of the development cycle is planning, the next four are development, and the final two are production acceptance testing (PAT) and deployment. Each iteration starts on the first Monday of the month. Some iterations are 4 weeks and others are 5 weeks, but there are a total of 12 iterations a year.

Although updating such a mission-critical application might seem risky to do every month, a lot of the perceived risk comes from the typical lack of fully-automated testing. Litle has a very large body of automated tests (over 30,000) and also has a test system which simulates the usage patterns and load of organizations like Visa and Master Card. Every change is subjected to the same sort of testing that typically takes traditional development organizations multiple person years of effort to achieve. As a result, the quality of their monthly upgrades is unusually high even compared to traditional product release cycles of six months to a year or more.

Their development teams are entirely collocated with no offshore or offsite development teams. For each project they assign a team consisting of multiple pair programming teams. Over time they have grown to a development team of 35 including QA and a codebase that contains 50,000 files including test cases and have had to add practices in order to scale XP. The biggest problem they ran into as they have grown was integration of newly developed code into the existing codebase. To address this, in 2007 they added the practice of Multi-Stage Frequent Integration to their implementation of XP. They do frequent integration instead of continuous integration because a full build/test cycle takes 5 hours.

Prior to implementing Multi-Stage Frequent Integration, they would have to manually pore over all build and test failures to determine which change caused which failures. This was done by very senior developers that were familiar with all aspects of the system to be able to understand complex interactions between unrelated components of the system.

Using Multi-Stage Frequent Integration, each project works against their own team branch, merging changes from the mainline into their team branch every day or two, and doing their own build/test/cycle with just that team’s changes.

Thus, any failure must be a result of one of their changes. When the build/test cycle is successful, the team then merges their changes into the mainline. As a result, any team-specific problems are identified and removed prior to mainline integration and the only problems that arise in the mainline are those that are due to interactions between changes from multiple teams. The isolation also simplifies the job of figuring out what changes may have caused a problem because they only have to look at changes made by a single team, not the changes made by all developers.

Tests and code are written in parallel for each work item as that work item is started. They do not distinguish between enhancement requests and defects. All assigned work is tracked in their issue tracking system and all work that developers do is associated with a work item in the issue tracking system. This is done to automate compliance as they need to pass biannual SAS70 Type 2 audits and annual Payment Card Industry (PCI) audits.

The PCI audits go smoothly. According to Dave: “You have to do them yearly, we’ve been doing them for 5 years. We’ve never had a problem. If you’re doing development the right way, you’re not going to have a problem with audits, because all they’re checking is that you follow the best practices of software development. You must use source code control, track why changes are made, ensure that the work done aligns with the company’s goals, and that your app meets data security requirements. On the auditors’ first visit, they said passing it was going to be a big problem because of our size (they figured, ‘small’, meant insufficient), and that we would likely have to make a lot of changes to pass. When the audit was complete--the auditors were surprised. They said they had never seen anyone do this good of a job the first time out.”

Dave also sees their long track record of successful use of Agile as a competitive advantage: “Because we have our monthly releases, our sales people are able to work with our product people to adjust the company priorities for what makes the most sense for the business. Rather than having to pick everything we need to fit in for the year now and then get together again in a year, it becomes a regular monthly meeting where they look at the priorities together. So if there is a vertical we’re not in yet or there’s a vertical that we are trying to expand in or there’s something that makes sense for the business, if we decide as a company we should go after that business, we bid on it.”

More Information
I look forward to any questions about the case study. By the way, if you are an AccuRev user, keep a look out for your invitation to the local user group meeting. David Tarbox will be speaking about his experiences using AccuRev for Agile Development.

In the meantime, you may be interested in learning more about Litle & Co., Multi-Stage Continuous Integration, or AccuRev.

If you would be interested in participating in an Agile case study for the book, please contact me.

TOC: Zero to Hyper Agile in 90 Days or Less

Upcoming Conference Sessions

Agile 2008 - August 4th - 8th
I will be doing three sessions at Agile 2008 in Toronto. If you are going, I hope you will consider attending one of my sessions. In any case, I look forward to meeting you there. Agile 2008 promises to be a truly great conference. The planning process was amazing with lots of great discussion of potential proposals.

Here are the three sessions that I will be doing:

Questioning the Perception of Agile (Full Listing)
There are many misperceptions of Agile. This session is mainly for Agile advocates. It is a facilitated discussion about common misperceptions/misconceptions and ways to alleviate them.

Calling All Agile Skeptics, The Curious, and Die-Hard Non-Agile (Full Listing)
If you are looking for a place to either air your skepticism/questions or respond (helpfully) to skeptics/questions, this will be a facilitated forum for both. Please bring an open mind, your sense of humor, and at least one anecdote to share. Participants will be expected to share the floor and help to keep the atmosphere fun and relaxed.

See Large Scale Multi-Stage Continuous Integration in Real Time (Full Listing)
Multi-Stage Continuous Integration is a way to scale CI to larger teams as well as a way to make CI more effective for teams of any size. For a preview, I have a series of posts on the topic. This session will show a real-time environment using AccuRev, Cruise Control, and Jira with hundreds of scripted developers resolving issues, making changes, causing builds to fail and pass, etc.

SD Best Practices, October 27th - 30th

Introduction to Multi-Stage Continuous Integration (Full Listing)
Multi-Stage Continuous Integration is a way to scale CI to larger teams as well as a way to make CI more effective for teams of any size. For a preview, I have a series of posts on the topic.

Agile Development Practices, November 10th - 14th
A full list of sessions is TBD soon, but all other details are available now and registration is open.

Introduction to Multi-Stage Continuous Integration (Details TBD)
Multi-Stage Continuous Integration is a way to scale CI to larger teams as well as a way to make CI more effective for teams of any size. For a preview, I have a series of posts on the topic.

Calling All Agile Skeptics, The Curious, and Die-Hard Non-Agile (Details TBD)
If you are looking for a place to either air your skepticism/questions or respond (helpfully) to skeptics/questions, this will be a facilitated forum for both. Please bring an open mind, your sense of humor, and at least one anecdote to share. Participants will be expected to share the floor and help to keep the atmosphere fun and relaxed.

The Iterative Design of a Lego Sports Car that Transforms Into a Robot, Part II

In Part I, I talked about how the design of a Lego car that can transform into a robot illustrates the process of iterative design. At the beginning of the project, I had the exact requirements and they didn't change at all from the start of the project to the end. However, just having the right requirements did not mean that I could immediately design something that would meet the requirements. At the end of Part I, I had a Lego car that met many of the basic requirements, but still didn't transform. I was stuck because I didn't have the technology that would meet the rest of the requirements.

Part of iterative design is keeping your eyes open, always looking for ways to intelligently expand the solution space. If the information you need is not already at hand, trying variations of things you already know may not lead to a good solution.

Thinking Outside The Box
I had always eschewed the Bionicle Legos as “not really Lego,” so I hadn't originally considerd those. But then two things happened that breathed new life into my Lego project. My son’s third birthday was coming up, and my wife suggested that we make it a Lego birthday. I was responsible for filling the gift boxes for the guests, and I decided to start by visiting the Lego store to see what they might have.

The Lego store had a wide variety of gifts for $5 or less including Duplo dinosaurs, a pony and princess set, knights on horses, and most importantly (for the project) small Bionicle sets. I had never bought a Bionicle set before, so I had never noticed that many of them had articulated joints. The joints were ball-and-socket parts which were small and had enough friction to make them poseable. They were perfect for the project.

There is another kind of Lego set called Exo-Force which has a completely different technology for doing articulated joints and also has a part which when combined with a Bionicle part provides a third technology for articulated joints.

Creating limbs which can fully fold up is a difficult proposition at best. But, I managed to create a couple of variations that allowed me to create the next version of the project. It didn’t really meet the goals of looking like a sports car or hiding the fact that there was more to it than met the eye. It was quite boxy, had odd proportions, and some of the joints were exposed. It was also fairly fragile. But, it could transform!

Sometimes Flexibility is Constraining
Sometimes software that is too generic or two flexible creates another set of problems such as taking too many resources or being too difficult to configure for the task at hand. The knee and elbow joints of the robot form had the same problem. Because they were fully articulated they were just too darn big and difficult to hide. I knew that these joints didn’t need to have the same degrees of freedom as the shoulder, hip, and ankle joints, but had not previously been able to create a hinge that could fold flat and have the full range of motion required. I repeated the exercise of determining all of the available hinge options and found a couple of new ones. The most promising one was a dual Technic hinge which was perfect for the knees.



For the arms, this hinge was too large. But then I realized that less resistance was required for the arms and a single hinge would do. Perfect! Now I had the robot skeleton, I just needed the car exterior.

Inspired by Elegant Design
Many months later, when looking for a new Lego set for my son, I saw a new car set (#4939) that was perfect for the project. I’m not an artist by any means, so one of the challenges has been finding Lego car designs to emulate. Previously they had always been either too big or too small. At one point I had been very excited about the Ferrari that comes with both red and yellow exterior pieces, but it was double the size I needed. Set #4939 seemed to be the perfect size. It didn’t hurt that it was very cool looking and reasonably priced. I had the feeling that this was it, I was finally going to achieve the goal.

Using set #4939 as inspiration, I was able to layer a car exterior onto the robot skeleton without too much trouble. As a bonus, the head, which had previously never really worked out well, was just there. I was within reach of the goal. But still one problem remained. I couldn’t cover the shoulder and hip joints without preventing transforming. I needed something that would allow the top and bottom parts to slide out and then lock in place. I had created parts that slid from place to place before for other Lego projects, but never something that could also lock in place.

Sliding Into Home
After trying many variations using small hinges to act as the locks, I had something that worked, but it was just too big. It was also very clumsy. Luckily, I was in denial. I forged ahead anyway. I wasn’t sure what to do about it being clumsy, but I was determined to create more room in the model. The most obvious step was to do what they do to create stretch limousines: take a regular limo and stretch it out. I made the whole model a bit longer, wider, and taller while keeping the same overall proportions. I also removed all unnecessary internal parts or replaced them with parts that would still do the job but took up less space.

It wasn’t enough, and I couldn’t stay in denial about the sliding part being clumsy forever. But while I was stretching things out I observed that some of the technic parts would slide around a little bit when I exerted too much pressure on them. Of course, the part that would give depended on the amount of friction. Whichever had less friction was what moved and the other one did not. From that came the solution: two bricks together to anchor the axle, and one brick that moved freely but with enough friction that it would stay wherever it was slid to.


The slider was the last major challenge. Once I had this part, it was just a matter of trial-and-error shifting things around until it all worked. There are two sliders; one for the lower body, and one for the upper body, the following picture shows the car with the upper and lower body pulled out as the first steps of transforming.



After the car has been extended, the legs are unfolded.



Lastly, the arms are extended. The head is also fully articulated.



There is still room for improvement, but at this point I think it is more a matter of aesthetics than mechanical design. I’m happy with the current design and hope that perhaps a Lego fan with more art skills than I will take the design to the next step. My other hope is that perhaps the amazing folks at Lego will start producing transforming Lego sets. And of course, if they were Transformer branded, that would be a wonderful (but not necessary) bonus.

[Get the LDraw model for the car here . ]

Motivation, skill, and experience all play important roles in the design process, but they are no guarantee that you are going to be able to design, let alone build what you have set out to do. It almost always requires trial-and-error, serendipity, creative inspiration, research, and an open mind.

Next: Frequent Releases Improve Code Quality Faster

The Iterative Design of a Lego Sports Car that Transforms Into a Robot, Part I

In the lead-up to the debut of the movie “Transformers,” there was already plenty of Transformers merchandise available. My son, who was 2 ½ years old at the time was instantly entranced. While my wife was strolling him through the bookstore he grabbed a Transformers calendar without her noticing. When she got to the register he then used his charms to convince her to buy it.

The Challenge
This got me to thinking. Wouldn’t it be fun to build my son a car out of Legos that could transform into a robot? After all, how hard could it be? I decided to give it a shot and I set myself the following design goals and constraints:

1. Build a Lego model that has two forms: a sports car and a robot
2. Folks looking at it agree that each form is clearly recognizable as that form
3. On casual inspection, it doesn’t look like it transforms
4. 100% Lego, no other parts, and no glue
5. Easily held in one hand
6. Strong (parts don’t fall off easily during use)
7. Robot form can stand on its own
8. The model can easily transform from one form to the other
9. Model is completely connected and transformation does not require the addition or removal of any parts
10. Fully articulated

Now, a year and several generations of Lego models later, I've finally created a Lego model which meets these deceptively simple goals.

As the project progressed, I wasn’t thinking of it as an exercise in iterative design or a formal project, it was always just something I did for fun in my spare time. I didn’t realize the link to iterative design until I started talking to people about the history of the project.

Transforming Requirements Into Working Software
When working with software, even if you have clear requirements that you have validated with users, it is not always clear how you will implement them. The search for a solution has two possible components: thinking about how to do something and implementing something. When thinking about how to do something, that’s really just fantasizing. Of course, it is fantasizing that is focused on producing something real, and it is more likely to produce something real than fantasizing about winning the lottery, but it is still fantasizing.

Requirements, analysis, specifications, and design are all examples of fantasizing. Yes, it is useful to do it, and you will often hit on solutions or eliminate problems much more quickly and much more economically than creating something real without giving it the same amount of thought, but fantasizing is only an approximation of reality, an interlocking web of educated guesses. Not until you actually attempt to create something real will you find out how all of the individual requirements interact and the real costs and difficulty involved. You don’t always find a clear match between requirements and implementation that is cost-effective to implement. In this case you have four paths forward: implement with the closest match and hope for the best, search for new possibilities, invent new possibilities, or shelve the project.

Transforming Lego Car, Iteration 1
The first attempt to build the transforming Lego car didn’t go very well. I was able to create a reasonable looking car which had lots of interior space for the mechanics needed to transform, but creating the sliding parts and the joints proved to be more difficult than I had imagined. I had lots of different hinges, flat parts, special joints, and Technic parts, but all of the possibilities I came up with were either too big, too fragile, too loose, or didn’t support articulation well enough. I was stumped. After a quick trip to the nearby Lego store to see if there was something I had missed, I reluctantly decided to shelve the project.

From a requirements perspective, I had satisfied requirements 2-6, which is half of the requirements. Of course, it was easy to satisfy "doesn't look like it transforms" because it didn't transform at all. :-) From an Agile perspective, the car was fully functional as a car in its own right, and I had discovered lots of valuable information that would help with later iterations of the car.

Little did I know that a hidden bias had kept me from seeing what was right in front of me. It wasn't until a couple of months later that serendipity broke through that bias and brought together the necessary ingredients for the first transforming version of the car.

Next: Part II (of 2)

Applying Usability to the Software Development Process

In order for something to become widely adopted, it has to provide an advantage that is significantly greater than whatever it replaces, and the advantage has to be easily accessible to a mainstream audience. Another way to describe “easily accessible” is to use the term “usability.” That which has high usability is more likely to become mainstream.

There’s a lot of literature on the usability of both software and everyday objects. But what does usability mean when applied to the process of software development itself? If we think of the myriad steps we use to produce software as the implementation of a software development process, then we can think of that implementation as software and consider its usability.

To consider the usability of a particular process, we first have to have a measuring stick for usability. Here’s a set of categories loosely based on some of the chapters from Joel Spolsky’s excellent book “User Interface Design for Programmers” and also adding in “visibility.”

Feedback
We get feedback our whole lives. When we first come into the world, we need constant feedback in order to learn how to survive. Touching something hot is painful, going without food is painful, falling down stairs is painful. In school we get feedback as to which subjects we are good at and which subjects we are not so good at. We can use this feedback to decide what to do more of, what to avoid and as an aid to improve, if we are able to.

The software development process, along with its rules, is just one process that you are a part of. Process, rules, and laws are all around you. You don’t have to consciously think about them but you are constantly reacting to them and they are second nature to you. On your way to work you drive on the correct side of the road, you stop at stop signs, and you stay in your own lane. When you go to lunch, you pay for your food. After work you go to your house instead of going to somebody else’s house.

It is easy to remember the laws, rules, and processes because you interact with them every day and you get feedback in the form of reminders and possibly even unpleasant consequences when you go astray. A great example of this is the rumble strip that has become commonplace on the side of highways. If for some reason you start to drift off the road, you’ll hear a loud warning noise as your tires hit the rumble strip. If you are really violating the law, you’ll hear an even louder sound accompanied by flashing lights. This is also known as feedback.

With traditional development, the development timeframe is much longer and thus the feedback loop is much longer. With Agile, the feedback loop is usually no longer than a month and often on a weekly basis.

Consistency
An important part of absorbing all of these laws, rules, and processes is that they are clearly defined, easy to understand, and easy to remember. For instance, once you have mastered basic physical laws such as the fact that running at full speed into a door is painful, it is easy to understand the reasoning behind having an upper limit on the speed of vehicles on public roads. You may disagree with that limit, but you can generally guess what the limit is for a given stretch of road and it is posted at regular intervals to help you score your guesses. Sometimes there is even a referee.

In traditional development, the long timeframe makes it difficult to absorb what the actual process is and usually everybody has a different mental model of that process. When people have different mental models for the same process, bad things happen. In Agile development, the process is far simpler and much easier to absorb. You have a chance to see the full process from start to finish on a very regular basis.

Expected Behavior
When something happens unexpectedly, it interrupts our train of thought and slows us down. It is a little bit like taking a wrong turn and becoming lost. We have to take time to figure out where we are and how to get back to familiar territory. Most of the time, when working on a software project we are concentrating on a single task that we alone are responsible for.

Consider software development from your own perspective. I’ll bet that you identify with most or all of the following statements:

• When working on a software project on your own, you make changes and use the new version right away.
• One of the reasons that you got into software development is the fact that you can make a change and see the result right away.
• When you have something new working for the first time, you want to show it to somebody.
• It isn’t easy to create something that the customer thinks is exactly right the first time.
• Even when creating something for yourself, it isn’t easy to create what you want the first time.
  

Most software projects are part of a larger scope, even if we are the sole developer. Somebody has to test it, and there are usually organizational standards that must be adhered to. In any case, you will spend a fair amount of time developing according to the rules of the organization (“the process”) rather than the way you would do it naturally if it was an individual project just for yourself. Whenever the way that you would develop software naturally conflicts with “the process,” the more you have to slow down and think about the correct next step and the greater the usability problem.

With traditional development, there is generally much more process to cope with the greater complexity of trying to develop a much larger increment of functionality over a much longer time period. Because of the large gap between how one would develop software on one’s own and how one develops software as part of a much larger process, the steps in the process are often counter-intuitive and thus it is much harder to establish a natural rhythm.

Visibility
It is hard to know what to do next if you aren’t sure where you are. Visibility is akin to project status. Do you feel like you know exactly where you are and what to do next at every step of the process? Or do you feel pulled in a million different directions at once and lose track of what you’ve already done and what you need to do next? If you do feel like you’ve finally “got” how software is developed in your organization, how long did it take to get to that point?

In a traditional project, it is almost impossible to see at a glance what the real project status and progress are. You know that you won’t really know until sometime after code freeze. In an Agile project, the next milestone is much closer, so it is much simpler to see where you are. Granted, if your real goal is six iterations out, you don’t have the same sort of visibility into the next five iterations, but the chances of risks hiding out until code freeze is substantially reduced.

The Usability of the Software Development Process
Based on just these few criteria, it is clear that traditional software development suffers from poor usability and hinders the ability of otherwise competent people to get their work done. While it is true that people can succeed on projects that have very long timeframes and involve a high degree of stress at the end, that mode of working does not align well with our strengths as human beings.

People are much more productive and much less prone to error when they work at a constant and sustainable pace. This also means that when there are unforeseen circumstances, people are more likely to be able to respond well.

Agile Development provides frequent feedback via short iterations, a simple process which is more in keeping with natural human rhythms and capabilities, and significantly more visibility into project status and progress. This creates a highly usable and people-oriented software development environment.

Next: It is Not the Bug That Bends, it is Ourselves

How Agile Works in a Nutshell

Yet Another Fad
We’ve heard it all before. The Segway, UML, Virtual Reality, TQM, object-oriented databases, Artificial Intelligence. Agile’s no different, right? Just the latest fad? That’s what I thought.

Back in 2005 at the SD Best Practices conference, I had my first run-in with Agile development. At a speaker’s reception, my blissful ignorance of Agile came to an end. I was surrounded by Agilists. They looked perfectly normal to me, but then they started to talk about the advantages of keeping track of requirements on 3x5 cards, pair programming, and lots of other crazy sounding stuff.

3x5 cards! Are you kidding me? What is this, the seventies? How can people who make a living writing software advocate the use of 3x5 cards? And what about pair programming? Two people sitting together all day sharing the same keyboard and mouse switching between coding and sitting on their hands, having to hear phone calls about the results of medical tests. Yuck!

I wasn’t just skeptical, I was outraged that all these people were trying to push this nonsense. Keep in mind that I work at a company that provides software tools to companies that are looking to improve their process. On the surface, Agile seems to be diametrically opposed to that. I felt that it was time for somebody to step forward and do something about it. I nominated myself. To gather evidence I read the books, talked to the experts, and attended the seminars. But then one day a funny thing happened. I had an aha moment and I realized there might actually be something of value buried under all of the rhetoric.

To illustrate what I realized, let's compare traditional development and Agile development using a typical development scenario.

Next: Traditional Development Scenario

New Reddit for Agile Development

Reddit is a great way to surf the net for things that you are interested in, and you can participate in determining what's good and what's not. Now you can surf using reddit for all things Agile! Check out the new Agile reddit!

Is Your Dev Team Having Performance Problems? Try Niagra!

Have you ever thought “why do the companies I work at have so many problems with developing software?” Well, I can tell you from interacting with literally thousands of software development organizations that most development organizations have problems with reliably and predictably producing high quality software. What I hear over and over again is “if only we could hire the right people” or “if only people could be more disciplined.” The other thing I hear over and over again are various tweaks to traditional development that people believe will fix things but that “other people just don't get it.”

Well, there are only so many people out there to hire so unless we find some magical place to hire more of “the right people” from, we are going to have to figure out how to use the people we have. And I hate to break it to you but you just aren’t going to get more discipline out of people. Whatever discipline that exists today, that’s all you are going to get. So what’s left? Changes to the way we do things.

Let’s say there was some combination of techniques which would mean that most development groups could use traditional development to reliably and predictably produce high quality results. Let’s call it the “Niagra” method. Furthermore, you (or whoever wants to make this claim) get to define the Niagra method. It works every time. Just use Niagra and performance improves overnight (when used as directed).

If the Niagra method existed, it would have become widespread by now. It would have become the #1 prescribed solution to project performance problems. If it existed and produced the claimed results, people would start referring to it, and it would spread rapidly. That’s how C++, Java, and HTML became mainstream. People rapidly adopted them because they provided benefits that anybody could realize. In fact, there are many proposed Niagra methods, but none of them have become mainstream because they only work under special conditions.

This raises an obvious point. If traditional development is so bad and is such a big failure, then why has the software industry done so incredibly well? The software industry is an incredible success, that’s absolutely true. It is incredible how much of our daily lives runs on software and how much software has positively influenced our quality of life. The benefits of software, when it does finally ship and it does finally have a high enough level of quality are worth waiting for. Otherwise, of course, there would no longer be a software industry.

There have been incremental improvements to the software development process which have produced incremental improvements and have become widespread. Examples are: the use of software development tools such as source control and issue tracking, breaking down large tasks into small tasks, nightly builds, one-step build process, moving from procedural programming to object-oriented programming and many others. The important point here is that each of these improvements have become widely adopted and made things somewhat better, but still didn’t solve the root cause. The process is still unpredictable and unreliable.

This is akin to the O(n) approach to problem solving. If you have an algorithm which takes 100*n^2 operations, then getting that down to 50*n^2 is a good thing, but changing the algorithm to (for instance) be 200*n is much better. To date, changes to the software process have been of the first variety, shrinking the constant.

Perhaps Agile development is the answer. But isn’t Agile “yet another fad” which will blow over any day now? After all, we have had CMM, 9001, TQM, Critical Chain, and many other “next big thing” ideas in the past. What makes Agile any different? Previous ideas have all been centered around the idea that the framework underlying traditional development is fine, it is the implementation, discipline, people, or management of the people that is the problem. No, it is the underlying framework that is the root problem.

Certainly, people issues are important. Getting a team gelled and working well together with the right combination of skills is essential. But the success rates of these teams isn’t exactly stellar either! Yes, it is better, but they have similar problems with predictability and quality. A root problem remains. I do not claim that solving this problem means that suddenly teams that have insufficient skills and hate each other will start pumping out high quality software on a regular basis. But I do claim that Agile development has the ability to dramatically increase the potential of any team. And to truly succeed you have to do both: create a good team, and use Agile development.

Let's take a deeper look at the fundamental problems with traditional development.

Next: Traditional Development is a Chinese Finger Trap