Thursday, November 29, 2007

Agile Product Management

One reason that companies fail is that their software isn’t really differentiated from their competition. Just adding all of the bells and whistles that your customers ask for is not a way to differentiate. Your competition is being asked for exactly the same things.

While the appeal of rapidly responding to customer requests is very tempting, there are a few cautions in order. Customers don’t always know exactly what they want and don’t always express their high-level requirements. Often, they will ask for something because they have established a manual process to work around the fact that you are missing a whole feature set and there is one feature that would really help their manual process which wouldn’t even be needed if you had the feature set that they really need. By responding rapidly to customer requests as though they were well thought out requirements, you run the risk of creating Frankenstein’s monster: lots of bits and pieces that don’t really work well together and don’t form a very intelligent solution.

Also consider that unless you have a very big team or a very small stream of requests from users, it is unlikely that your development organization has the bandwidth to keep up with all of the requests from users. And of course, the more you provide, the wider your market appeal will be and the more customers you’ll have and the more requests you’ll get.

Each of the requests that you implement has both a cost and a benefit associated with it. If there is not enough benefit to justify the cost, it is debatable if you should even consider fulfilling the request. Remember though that the benefit may come in the form of customer satisfaction which can lead to more customers. In any case, you should do at least a rudimentary cost/benefit analysis to at least rank a particular request among all of the other requests you could do. Those with the best ROI should bubble to the top.

Even doing an ROI analysis is not really enough. What you really want to do is to add new capabilities to your product or new products to your product line with a good ROI while maintaining the integrity of those products. That is, you want to maintain the quality, usability, and performance of your products.

Next: The Role of Defect Management in Agile Development

Wednesday, November 28, 2007

More Agile Kool-Aid Please!

I got an interesting response to my post on "No Really, What is Agile?" today. For the complete response, see the comments on that post. It is fairly vitriolic and anonymous, but I have no problem with that because the benefit is that we get to see what is really on the poster's mind. The poster has clearly confused me as a spokesperson for "the Agile movement" which I certainly don't claim to represent. I would hope that anybody that has read a couple of my posts would realize that I'm not in the "Agile can do no wrong" camp. There is also clearly a high level of frustration apparent in the post. Hopefully, the venting was cathartic. :-)
I have read enough to know that the Agile Kool-Aid drinkers think they're immune to such concerns. Sadly, that is not the case. Instead, the Agile types keep moving the goalposts: "Oh, that's not Agile!" or "They're doing Agile wrong!"

Guess what? Eventually you'll move the goalposts so far that you'll hit the stands.

There are plenty of issues with Agile adoption. Some of them can certainly be pinned on the overzealousness of some Agile advocates, but some are due to the adopters themselves. As the Dilbert cartoon points out, adopting something without understanding it and without creating a proper ecosystem for it is destined to fail.

I am reminded of a day long ago when I decided to ignore a "trail closed" sign because 33" of powder had fallen during the night onto the trail which was rocks and grass the day before. It looked great, but there was no base, so it was just the same as riding on rocks and grass.
...where is your movement going to be when they totally bastardize it and projects fail? Well, Kent Beck and his gang started this movement with a huge failure of a project at a big company, but you always conveniently ignore that history.

Well first, it isn't my "movement" any more than my use of AJAX technologies means that I am part of the "AJAX movement". For me, Agile is neither a movement nor a philosophy, but rather a tool like any other. Agile is either a good idea or it isn't. I happen to think it is a great idea, but I agree that we have a long way to go before it is mainstream. As for ignoring history, I'm not sure what that means. Does that mean that when software development projects fail we should stop doing software development? When we fall down when learning to ski should we give up? Does it also mean that traditional development is the best way forward? If it was, there would be no such thing as Agile development because everybody would be loathe to change. Agile development has come to be precisely because of the frustration that people have with traditional development.
Just try going to Slashdot and posting your nonsense. They can be quite vicious.

I suppose my point is: don't be afraid to be self-critical about Agile. Don't make excuses.

Dear anonymous, one of the best ways to move things in a positive direction is to pull things in a positive direction, leading by example. I invite you to read more of my blog and respond to specific points instead of responding generally to Agile just because I have the word Agile in my blog.

Tuesday, November 27, 2007

No Really, What is Agile?

[Revisions: 6/11/08 general updates, 1/9/08 responding to comments]

Giving a concise definition of Agile is far from easy, probably because Agile is actually an umbrella for a wide variety of methodologies. Some will say that "it isn't about methodologies, it is a philosophy; it is the 4 values defined by the Agile Manifesto." Philosophies and Manifestos are a fine place to start, but at the end of the day you have to get down to brass tacks in order to actually bring about change.

It may be pointless to define Agile development. It may be better to just talk about the concepts and the practices and say "these things are worth learning about regardless of what you call it or how you define the overarching concept." I still think it is worth a shot so as to establish a starting point for discussion. So, let's discuss the definition of Agile a bit and then move on to the nitty-gritty.

Because there are so many practices associated with Agile development, a simpler way to define Agile is to do it in terms of the benefits. It is not a perfect way to define something, but it is currently the best way that I know. There would be no point in doing Agile development unless it was better, so I define the benefits in relation to traditional development. I consider waterfall to be part of traditional development, but I use the phrase "traditional development" instead of waterfall because there are many shops which would say "we're not doing Agile, but we're not waterfall either."

I define Agile development as that which, in comparison to traditional development provides the benefits of more flexibility, higher ROI, faster realization of ROI, higher quality, higher visibility, and sustainable pace. Let’s take a brief look at these benefits.

More Flexibility/Options
With traditional development the software may look like a construction site until just before the release: some parts are done, others are not, but the building is not currently usable at all. Changing plans during the development cycle means ripping out work in progress and possibly work that is done but depends on other work that is not done.

With Agile development, you develop your software in short iterations where “short” means a month or less. At the end of each iteration you have software which is shippable. That means that at the end of each iteration you have the option to easily change plans before the beginning of the next iteration in order to take into account new business requirements or information gained during the previous iteration.

Higher ROI
Because you have more flexibility you can change your plans to take into account the current market conditions instead of what seemed like a good idea when you started working on the release. So, if traditional development would have released after 6 months and had features A, B, and C which had an ROI of 9 during planning at release has an ROI of 7, Agile might produce A, B, and D in the same timeframe which has an ROI of 11. The idea is that while C looked like a good idea at the beginning it turned out later to be worth less than expected and D came along which had much more ROI so we did that instead.

Faster Realization of that ROI
Since you have shippable software at the end of every iteration you can start realizing the ROI whenever you like instead of having to wait. If the Agile team is also releasing frequently, they are able to capture the value that they created sooner than they would have otherwise been able to do.

Higher Quality
In a traditional project, the elapsed time from start to finish from the perspective of any individual work item is very long. There is no consistent process applied to each and every work item. Instead, work items are fused together into “the release” and the quality of “the release” is measured.

One consequence of this is that QA gets a big dump of functionality near the end of a release and has to figure out how to best use the time remaining which is often less than originally planned. That means taking shortcuts and the "most important" new features get very thorough testing and the rest get a "spot check." Another problem with traditional projects is that since much of the test writing and execution is left to the end, problems hide out for long periods of time and thus there is a long time between introduction of a problem and the detection and fixing of that problem.

In an Agile project, the elapsed time from start to finish from the perspective of any individual work item is very short. Test plans and automated tests are created throughout the project and each work item is given the amount of QA resources that is appropriate for it. Because problems are found and fixed faster, there is less chance of the quality of a project being poor for long stretches of time. Code is written on a stable base and is more likely to be stable itself because there will be accurate and timely feedback on the results of the changes.

Higher Visibility
Time after time, with traditional development, progress is measured based on progress against a plan. The problem with that is that customers don’t buy Gantt charts, they buy shipping software and the progress that is measured by Gantt charts is not directly connected to shippable software.

Just because you have finished 100% of the requirements which is 20% of your plan, that doesn’t mean you are 20% done. In fact you are 0% done from the perspective of the customer because they can’t benefit from that progress until you ship. In a traditional project, you only know how much ahead or behind plan you seem to be based on the progress that people claim, but you don’t really know how much ahead or behind plan you actually are from an “is this shippable” perspective.

Short iterations solve the problem of false progress reports. With short iterations, you know at the end of each iteration exactly how much progress you have made. Instead of the traditional "we're still on target" all the way up to just before the end of the release, you know at the end of every iteration exactly where you are.

Whatever work was done during that iteration is done and potentially shippable. You know exactly how many work items were fully completed, how many weren’t started, how many test cases remain to be written and automated, how many test failures remain to be fixed, and how many defects were filed on work done during the iteration. You get the kind of information that is usually only available just prior to release on a monthly or even weekly basis.

Next: Sustainable Pace

Top Ten Agile Analogies

I like analogies. Sometimes folks say I use them too much. I know that at least once I’ve come up with a really “out there” analogy. For instance, there was the one about cats riding motorcycles… Anyway, in a recent post I used two analogies and I previously posted about how Agile was like driving a car and like sales. So I was wondering what other analogies for Agile development were out there and did a search for “Agile Development is like” and “Agile is like.” Here is my top ten list. There actually weren’t many more than ten to choose from, surprisingly enough. If you know of another good one, please let me know!

10. “In many ways agile is like dieting or quitting smoking: you know it's good for you but some find it easier to fall back on old ways than to stick with the program” – Junilu Lacar

9. “Agile is just like driving a car” - various

8. “Agile is like sales” – yours truly

7. “Agile is like golf” - Joe Little (link)

6. “Agile is like religion” - various

5. “Agile is like a box of chocolates, you never know what you’re going to get” - Tim Coulter (link)

4. “Agile is like Churchhill’s democracy, the worst possible solution until compared to the alternatives” – David Starr (link)

3. “Agile is like being pregnant, you either are or you aren’t.” - anon (link)

2. "The puck stops here." (Agile development is like hockey, not ballet --- Tim Lister).

1. Agile is like any other newly introduced popular concept. “…Everybody is talking about it. Very few are doing it and those who are, are doing it badly.” - James O. Coplien

What is Agile Development?

I'm often asked "what is Agile Development?" I've never really been sure how to answer that. Some people say "well, it is the 4 values defined by the Agile Manifesto." To me, that's too vague. The four values are fairly philosophical in nature and pretty far from business benefits or in the trenches implementation. I tend to start any discussion of Agile by talking about the benefits. It seems to me that these benefits are fairly widely agreed upon as the following.

In comparison to "traditional development", Agile provides the benefits of:

- More flexibility/options
- Higher ROI
- Faster realization of that ROI
- Higher quality
- Higher visibility

Recently it occurred to me that this also serves as a good definition for Agile. That is, Agile development is that which provides the above benefits. I like this definition because it doesn't require a discussion of specific practices, yet it is also straightforward and concrete.

Monday, November 19, 2007

Agile Development as Panacea

I find that people have pet issues which they are always looking for a new way to solve. Often, when a new way of doing things is introduced, people ask if this new thing will solve one of their pet issues. Agile development is a new way of doing things. Thus, people often ask if it will solve one of their pet issues. As an Agile advocate, it is tempting to find a way to answer such questions in the positive.

Unfortunately, Agile development is not a panacea. It will not cure all your software development problems. Here are some of the issues that Agile development does not directly address: technical problems, choice of which technologies to use, personality conflicts, mismatched skill sets, insufficient management skills, poor listening skills, or poor communication skills.

However, there are two aspects of Agile development which can be applied to all software development problems: frequent feedback, and reduced complexity. The frequent feedback provided by Agile development provides a better problem solving environment than traditional development. The environment surfaces more problems, surfaces them faster, and provides more opportunities for addressing problems and gaging the effectiveness of those solutions.

Agile development also reduces the complexity of your software development projects by breaking it up into smaller pieces. It is much easier to complete 6 projects which are each 1/6th the complexity of a large project than it is to complete one large project. This is especially true when you take into account the fact that the planning required for the large project must cover a larger time frame and take more variables into account.

Consider this rough analogy. You want to transport 300 people from Boston to San Francisco and you are given the choice of two methods. The first method is to create a rough plan and have a pilot who can make adjustments as needed as you go. The second method is to line up the plane, set it to hit a cruising speed and altitude and go straight for a set length of time and then start its descent at a specific time. And there is no autopilot. Basically, you set the parameters at the beginning and that's it.

In the first case, you can adjust for changing conditions as they come up. These might include weather, air traffic, or natural disasters. There is a medium degree of difficulty for this method.

In the second case, you must accurately predict every possible condition and plan for it accordingly in order to have the plane safely hit the tarmac at the destination at the right time so as not to interfere with other planes. If there is a change in one of the variables affecting the flight that you didn't take into account, it could make for a rough landing. The degree of difficulty here is pretty much incalculable. Suffice it to say that it is very high.

As I write this I'm reminded of another analogy: golf! The variability involved in golf is exactly what makes it such an interesting game. It is also why there are only a small number of people in the world that have ever gotten a hole in one. Considering that traditional development is very similar to expecting to hit a hole in one on every hole, it is no wonder that so many software projects end up being sub-par.

Next: Agile Product Management

Sunday, November 11, 2007

Agile Development: The Perfect Fit for Mission Critical Applications

I was talking to Ron Morsicato of Agile Rules at dinner at last weekend's Deep Lean conference and he was saying how he thought that Agile was a good idea for mission critical applications. That struck me as an interesting thing, and I've been thinking about that all week. I think he's right and here's why I think so.

Let's say you provide mission critical software and there is a problem reported in the field which requires a fix. Well, if you are doing traditional software development, you can't exactly ask the customer to wait until you finish your current release. And if you were to put out a fix for them using your main development path, that would probably take too long too. So, you have to have a critical-fix development path. By definition, this is a development process which is not used for regular development and it is not used very often (so one would hope).

As a result, you have one process for regular development and one process for critical fixes because your regular development process takes just too darn long. That means that when you most need for things to go smoothly you are going to use the process which is the least practiced and probably also something that only a handful of folks know or are allowed to do. Doesn't that strike you as just plain wrong? Imagine trying to explain that to the media!

Part of the idea with Agile, Lean, and especially Hyper Agile is that the path from deciding to make a change and being able to deliver that same change is as short as reasonably possible. If the "overhead" from start to finish for a critical fix is 4 hours, then any task should take you no more than 4 hours plus however long it takes you to write the code and the associated tests. Once you have this capability, then you really only need one development process which you use for both regular development and critical fixes! Thus, everybody is practiced in your critical fix process and everybody has experience with doing it. This will reduce risk and increase confidence in the results.

Saturday, November 10, 2007

Agile Development: The Perfect Fit for Financial Institutions

A couple of days ago while at QCon San Francisco, I had a series of conversations with a person involved in software development at a major financial institution. He really wants to start using Agile techniques, but isn't sure of the best approach to take to convince others internally. I gave him some pointers including my recent article "Software Developers Can Learn a Lot from Salespeople." As I was thinking about this I realized that Agile is actually a perfect fit for financial institutions!

Software projects are highly unpredictable. More often than not they are either late, over budget, missing originally planned contents, much lower in quality than originally planned or a mix of all of the above. Another area of the business that has similar problems with predictability is sales.

When was the last time that salespeople were expected to tell you exactly what their financial performance would be? Sales people can't predict exactly which deals will come in during a quarter, nor exactly what the amount of any particular deal will be, but they are expected to hit their quotas and they are expected to show results on a regular basis and CFOs are perfectly happy with how their salesforce goes about their process. They may not be happy with the actual results sometimes, but they approve of their process.

What sales does and what Agile teams do are exactly the same! They are both working with unpredictable quantities. Developers in an Agile team may not be able to predict exactly what will be the result of any particular release and sometimes not even the result of an iteration, but they can be held accountable to produce a regular flow of high business value software on a regular basis and to be able to demonstrate that the software is actually done on a regular basis as well.

So, if the organization is comfortable with the sales process, they should be equally comfortable with Agile development. In fact, anybody involved in the business side of software development should be very uncomfortable with anything other than Agile development and be actively working to figure out how to get their development process to be more like sales.

This post has only briefly touched on the comparison of software development and sales. If you are interested in a full comparison, please take a look at "Software Developers Can Learn a Lot from Salespeople."

Monday, November 05, 2007

Agile Seminar at QCon

If you are going to be in San Francisco for QCon, check out my session on "What's Keeping Agile From Mainstream Adoption?" . I've changed my focus a bit from when this was originally created. It really should be called "Agile: Why it Works" at this point and that's really what the talk is about. There has been so much written and said about the benefits and how to achieve them, but not so much on the mechanics of why it works in the first place. I think that without this basic framework it is harder for people to take the plunge. It is definitely why I only got into Agile by accident rather than via an understanding of how it works at the outset.

Agile Bazaar Deep Lean Trip Report

Deep Lean was a 2-day event held at MIT, organized by the New England Agile Bazaar, and sponsored by AccuRev. Speakers included Jeff Sutherland, Nancy Van Schooenderwoert, and Mary & Tom Poppendieck. Here is a quick overview of the weekend.

Day 1
What a great first day at the Agile Bazaar's Deep Lean event. Jeff Sutherland started the ball rolling. All of the speakers were excellent (all weekend) at stopping right at the allotted time.

Broken Development – Jeff Sutherland
First Jeff showed us what a mess traditional development really is. Then he started to show us how Scrum can help. He didn't get through all of his material due to all of the questions from the audience, which was fine because the discussion was riveting.

Thrashing – Mary Poppendieck
According to Mary, even when development teams are doing Agile well, there is almost always one problem remaining: thrashing. Thrashing is when you are spending more and more time to do less and less. She showed us the root causes of thrashing which include excessive context switching. For instance, if you have two tasks and keep switching back and forth between them to show progress to multiple stakeholders, you will actually finish both slower than if you had focused on one and then the other. My favorite part was seeing a video of how the Toyota Type G loom could stop itself when it detected a defect in the fabric it was producing… without modern electronics!

Lean Principles and Scrum – Jeff Sutherland
A very straightforward subject which was well presented. Basically, Scrum overlaps with much of Lean by default and a deeper understanding of Lean will improve your experience with Scrum.

Value Stream Mapping – Mary and Tom Poppendieck
During this talk there was a point at which the audience broke up into teams to try value stream mapping on real world examples. While this was a great idea in concept, I don’t think there was enough time allotted to make it worth it. Considering this was one of the very few (only?) hitches in the whole weekend, I really didn’t mind. Besides, it is well covered in their first book on Lean which you should just go out and buy anyway.

At the end of the last session about 25 of us went to the nearby Cambridge Brewing Company for dinner and conversation. The CBC is a local favorite for the tech crowd and well worth checking out if you are ever in Kendal Square for an hour or more.

Day 2

Coming of Age of an Agile Company - Nancy Van Schooenderwoert
Nancy observed that Agile adoption has a high mortality rate. That is, well intentioned projects start up but then fail. This is a topic that is near and dear to my heart because part of my mission is to help make Agile mainstream. That is, to get Agile to the point that any team anywhere working on any project can pick it up and be successful quickly.

Nancy used a number of (painful) stories to illustrate her points. One of the things that struck me about the circumstances was that there were many problems which had nothing to do with Agile, but rather just plain old good development practices. For instance, don’t use your production environment to test code whose only claim to fame is that it compiles!

Leadership in Software Development – Mary Poppendieck
Mary took us through a fascinating tour of leadership and management styles and methods from the early 1900s to the present day: from Taylor to Charles Allen to Taiichi Ohno to Deming to Kaoru Ishikawa.

Real World Experiences in Creating Agile Companies – Jeff Sutherland
Jeff showed graph after graph on Scrum benefits. The only thing keeping me from being hypnotized by the rapid succession of graphs was Jeff’s take-no-prisoners style. Summary: Scrum works.

Panel Discussion
Lots of great Q&A too numerous to summarize here.

In summary, this event was a rare treat and if you have the opportunity to attend one of these events in the future, I highly recommend that you go. As both a sponsor and attendee, it was well worth it.