Traditional Development == The Game of Telephone?

In theory, traditional development is a good idea. The work of development is broken up into stages in order to reduce risk and increase efficiency. But in reality a significant proportion of projects are shelved, cancelled, or delivered with low quality. In part, traditional develelopment has problems for the same reasons that it is fun to play the game of telephone.

I remember fondly playing a game called telephone in elementary school. In the game of telephone, a message is whispered from person to person. The fun part is that the last person says what he heard to the whole group and everybody gets a laugh out of how a phrase such as "Send reinforcements, we're going to advance" gets inadvertently translated into "Send three and fourpence, we're going to a dance". This is very similar to what happens in traditional software development projects.


Figure: The Traditional Development Telephone Game

The figure above represents a typical traditional development process and some typical times between handoffs. At each stage, instead of whispering the information along, the participants create and use a series of documents, each of which serves a different purpose. As the information moves from stage to stage it needs to be translated into an appropriate form for each stage of development. The documentation also serves as a record of information that was discovered along the way and the decisions and justification for decisions that were made along the way.

The product manager talks to customers and those conversations are translated into Marketing Requirements Documents (MRDs) which are translated into engineering requirements which are translated into specifications which are translated into designs which are eventually translated into code. The end result is then given to the user who often says something pithy like "that's not what I wanted."




At each stage of development there is a risk that the intent from the previous stage is mistranslated. Another way to think of the mistranslation is data corruption. The larger the amount of work that is flowing from stage to stage, and the longer it takes for work to pass from stage to stage, the more likely it is for information to become "corrupted" and the greater the extent of the corruption. Of course, people don't just blindly translate information without interacting with people from earlier stages, they ask for clarification when they think they need it.

When clarification is needed, such as when somebody is writing a test and has a question about how a feature is supposed to work, it requires a conversation with the author of the documention. There is no guarantee that the author is available when needed, knows the answer themselves, or can find the answer in the supporting documentation that they themselves used. That then requires another conversation with the architect, and then the architect with the product manager and then the product manager with customers. The customers may have given their feedback based on ideas they had during the discussion with the product manager but have long since forgotten. The further back in the chain you go, the harder it will be for people to remember the details required to answer clarifying questions.

People aren't very good at remembering the exact details of the work that they did weeks or months in the past. That’s one of the main reasons for creating the documents in the first place. When people are routinely put in the position of having to do something they are not good at, it leads to stress, lower morale, and mistakes. [See "Guilty Confessions of a Software Developer"]

Ok, so now we have a good idea of how customer intent can get lost in translation and how the people developing the software can get stressed out. While that's all very interesting, it isn't very useful unless we can figure out how to act on this information.

Next: Leveraging Human Strengths in the Development Process

Do It Yourself Agile

Originally, my blog was focused on Software Configuration Management. Then I described in a blog post how I realized that I'm not really an SCM person, I'm a software development process person that had been focused on SCM. So, I changed the name of the blog to "Agile Development Thoughts." Today I realized that this blog has changed its focus from random thoughts about Agile Development to a blog about Agile for the Agile DIY'er. So as of today, this blog is now "Do It Yourself Agile."

I still believe that the quickest route to Agile success is to get people involved who have done Agile before. But, that option is not always available and even when it is, it may not be possible to retain an Agile coach for the multiple years that it will take to reach your full Agile potential.

So, the idea of this site is to give the Agile DIY'er a resource to lean on. A good place to start is the web-only book "Do It Yourself Agile." This is an evolving book which is expanding and changing every week in response to questions and discussion from readers like you. If there is a topic that you don't see covered or that you would like to explore more fully, let me know! Just post a comment wherever you see fit or use the Q&A page. I'll answer your question or create a new blog post on the topic as soon as I can.

Check out the web-only book "Do it Yourself Agile" .

Agile Exposes Scaling Problems

One of the first questions that seems to come up in any discussion of Agile is "How Well Does Agile Scale?" Sometimes this is asked explicitly, but more often there is an underlying assumption that Agile does not or can not scale very well. When I was first exposed to Agile, my first impression was that Agile didn’t scale beyond a small team, say 1-12 people.

I used to try to tackle the question head on, but then I realized that there's something else going on here. Let’s take a step back. What do we currently assume about traditional development? I think we come to the discussion thinking that just because there are teams of 100 engineers, 500 engineers, and even 2,000 engineers doing traditional development, that traditional development is a proven quantity when it comes to scaling. Let’s first ask the question: “How well does traditional development scale?”

To answer that question we first have to define "scaling." I think a good working definition is that as you add new resources, there is a linear increase in effective output with little or no decrease in efficiency. For software, output and efficiency translate to new functionality and productivity. That would mean that if you have a 50 person engineering team and you add 50 more people you get twice the output. But when was the last time you saw that? Have you ever seen it? In my experience, after talking to hundreds of development organizations doing traditional development, productivity falls and thus output does not increase linearly with additional resources. In many cases, output actually decreases as more resources are added.

What do you actually do to "scale" your development organization? Do you have meticulously updated Gannt charts and estimates? Do you schedule lots of meetings? Do you spend a lot of time making sure that requirements and designs are just right? Do you reserve 30% of the development schedule for testing (and fixing)?

In my experience, after talking to hundreds of development organizations, the answer to the question “How well does traditional development scale” is “not very well.” We've just suffered along with this problem, in large part because while the pain was there, the root causes were difficult to trace, let alone address.

The main point here is that if you are in the process of rolling out Agile to a large organization, don't be discouraged when you run into scaling problems. The problems were always there, they just weren't as obvious. Now, instead of wondering why things aren't coming together as expected at the end of the project, you may find out right away that your organization doesn't really have a good way to coordinate API changes for APIs that are used by multiple teams. As Agile exposes problems like this, you can take steps to solve the problems and thus create a more scaleable development organization that just happens to be doing Agile development.

Is Agile Really Any Better? Free Talk with Pizza and Beer

If you will be in the Boston area on June 16th, I'll be talking about how and why Agile works, pitfalls to watch out for, and why it goes well with beer and pizza. There will also be an opportunity to learn how AccuRev helps in an Agile environment. You can read more details here.

Ken Schwaber to Speak on "Flaccid Scrum"

Come hear Ken Schwaber's talk on "Flaccid Scrum" at the Agile Bazaar in Burlington, MA on June 18th. Visit our web site for more details.

How Continuous Integration Promotes Healthy Agile Practices

Come learn "How Continuous Integration Promotes Healthy Agile Practices" in a webinar I'm doing for Version One at noon EDT on June 17th . See you there!

Abstract: Healthy, successful Agile teams rely on a robust Continuous Integration implementation as the hub of their Agile infrastructure. Many of the Agile practices depend on Continuous Integration (CI) in order to provide their full value. For instance, unit tests are the most valuable when all unit tests are run automatically on a frequent basis. Scaling Agile to large and/or distributed teams also depends on CI. Conversely, CI can be used as a leading indicator of the health of your Agile teams. This session will cover best practices for transitioning to Continuous Integration as well as show the strong link between CI and one-piece-flow, unit tests, short iterations, whole teams, refactoring, user stories, shared code ownership, distributed Agile, and large scale Agile.

Product Managers and Product Owners - Newton MA, Thu Feb 26th, 6pm

Rich Mironov of Enthiosys will be speaking on "Product Managers and Product Owners: Which Do We Need, and What Do They Do". Hosted by the Agile Bazaar.

For more details, please check out the full meeting announcement.

Poll - Short Iterations, The Benefits

Short iterations are the practice of breaking up a release into 1-4 week development cycles that each produce a shippable increment of work. Some of the benefits of short iterations include working software that you can show to people in order to get their feedback and having nothing in progress at the end of the iteration so you can change your plans to take into account market changes, feedback from customers, or information you discovered while implementing that iteration. Also, all of the tests for the functionality implemented during the iteration have been written and run and you have a well known starting point for the next iteration.

There are some assumptions hiding in this practice. It assumes that you believe that incremental design is possible and that you believe that any refactoring you have to do in subsequent iterations is worth the benefits as described above.

So, if you were starting a company with your own hard-earned cash, would you use short iterations or wouldn't you?



Next: Poll - Whole Teams, The Benefits

Poll - Product Backlog, The Benefits

A product backlog is a list of high-level product requirements (or user stories), prioritized by business value (typically determined by a product manager or product owner). It contains entries for any and all work that is performed, including bug fixes. As new requirements are gathered or submitted, they are inserted into the backlog at the position that best represents their business value with respect to requirements that have already been entered. As requirements are met, they are removed from the product backlog.

A backlog gives a clear context to discussions of priority. Anybody can go and look at the backlog to see the position of something they have a stake in. If it is lower than they think it should be, they can examine the backlog to see what is higher in order to help frame a discussion around either moving it higher or moving other things lower.

So, if you were starting a software company with your own hard-earned cash, would you use a product backlog or wouldn't you?



Next: Poll - Your Preferred Methodology

Poll - User Stories, The Benefits

User stories are a description of what is needed from the user’s perspective. User stories help to separate business value from implementation and focus all parties on the desired outcome.

User stories are different than requirements. When using requirements, it is likely that the developer implementing the requirement will be presented with an implementation task or a design document and be constrained to implementing as specified or as designed. A user story removes invisible constraints by focusing on the outcome desired by the user. The developer doing the work will see the user story, will be able to better understand what the user needs, and will be able to participate in or even own the specification and design of that story. User stories provide engineers more freedom to utilize their creativity and ability to innovate without the risk of implementing something that the user doesn’t want.

So, if you were starting a software company with your own hard-earned cash, would you use user stories or not?



Next: Poll - Product Backlog, The Benefits