Do It Yourself Agile

Announcing Valtivity!

2012-01-26T16:56:00.000-05:00

You may have noticed that I haven't been very active here in a while. Over the last year I've been spending more time doing Agile presentations than blogging. Well, that's about to change. I've just started a new company, called "Valtivity". I will now be devoting my time to Agile full time which means that I'll not only be doing presentations, but also blogging in tandem with focusing on Agile training and coaching. Need to get started with Agile or move to the next step in your Agile journey? Check us out at Valtivity .

Valtivity enables companies with a software development component to find and implement their ideal way of working that maximizes value to the company and its employees and allows them to find and serve new market opportunities faster. Valtivity does this by using unique training and coaching methods to rapidly teach the latest Lean and Agile techniques.

For the curious, Valtivity is short for "Valuable Activity." It comes from Lean's focus on activities that provide value.

Valtivity will be hosting its very first Webinar, "Agile - A Focus on Value" on Feb 1st at 1pm EST. Register now!

2012-01-04T16:23:00.000-05:00

Here's a little Agile Humor for you. I recognize a couple of these. :)

This comes from my friends over at SmartBear

Introducing Kanban Into an Existing Scrum Implementation

2011-10-14T10:17:00.000-04:00

By far, the talk that I am most requested to do is "Scrum and Kanban Like Chocolate and Peanut Butter". I've given the talk all over the US and Europe, most recently in Kiev for Agile EE. Invariably, people come up after the talk and ask how to get started with Kanban. I've given book and coach suggestions, but they are primarily pointers to pure Kanban resources.

Scrum and Kanban Like Chocolate and Peanut Butter
There doesn't seem to be much material specifically geared towards introducing Kanban into an existing Scrum implementation as opposed to going from non-Agile to Kanban. So, I've decided to write a short and concise guide to folding Kanban practices and principles into an existing Scrum implementation called "Scrum and Kanban Like Chocolate and Peanut Butter." I will write the guide via a series of blog entries over the coming weeks. I hope you will follow along with me, asking questions and offering feedback.

Who is "Scrum and Kanban Like Chocolate and Peanut Butter" Intended For?
The first question is: when does it make sense to introduce Kanban where Scrum is already in use? It turns out that is a very complicated question. Rather than answer that question, I'm just going to gear my posts towards folks that match a certain set of circumstances. My material for introducing Kanban into an existing Scrum implementation is geared towards folks for which the following is true:

You have already been doing Scrum for 6-9 months
There is a general feeling that things are going better than before Scrum was introduced
You are using user stories
You have a definition for "done"
Your user stories are relatively short with no stories that take longer than 1/2 of the iteration to get to done
Most stories are "done" at the end of the iteration
You have an iteration length of 4 weeks or less
You have a cross-functional (though not necessarily self-managed or co-located) team
If there is a "business unit" that previously served as "the customer" then you've integrated the business unit and software team at least to the point that there is a product owner on the team which talks to the customer of the business unit, and the business unit is no longer considered to be the customer for the software team.

If you do not have all of the above, you may want to consider getting to this point first prior to taking any of the steps that follow. Over time I'll come back and update some of the points above with links to resources for addressing these points if they are not already true. The first post will be available over the next few days. The guide will be based on my talk "Scrum and Kanban Like Chocolate and Peanut Butter". If you want to get a head start you can check out a recent video of the talk which is available on vimeo.

Agile Book Recommendations

2011-10-14T09:59:00.003-04:00

The following books are the main books that I recommend as starting points for learning about the various aspects of Agile. While their is a fair amount of overlap each of them covers a specific topic. The one on leading self-directed teams is not strictly an "Agile" book, but it is one of the few books talking about self-managed/self-organized/self-directed teams that I have found. The Toyota way is also not an "Agile" book but does a great job of explaining the concepts of Lean which I believe are the foundation for Agile.

Click to enlarge.

Sticking to Estimates

2011-09-01T11:39:00.000-04:00

I got a great comment from Kelly Waters on my recent post "Scrum: No commitment required" via his web site. After responding I realized that the response was a good basis for this follow-up to my original post.

In the original version of Scrum, the commitment of the team for the sprint is dependent on two things: a good work ethic and good estimates. If either one is an issue, then the work won't be done on time. The “no commitment required” title of my previous post was a bit tongue in cheek. Absolutely commitment is important, but I don’t think that making it part of “Scrum” will suddenly make team members with a poor work ethic suddenly have a good work ethic or team members with problems estimating better at estimating. Nor will having commitment part of Scrum automatically improve a bad team/PO dynamic.

People with a good work ethic will work hard. If you have folks with a poor work ethic, I believe Agile will expose this more clearly and then you can use your management skills to address it one way or another. Hopefully, with a win-win solution.

Estimates are just that, estimates. They are a good-faith representation of what the team believes it will take with the information they have at hand. When something is discovered, then it is important for the team to communicate that to the PO, but I don’t think it means that they have to stick to the original estimate. Is that “letting them off the hook?” I don’t think so. Let’s face it, if they estimate 8 hours, but it turns out to take 24, the only way to do it in 8 is to do 8 hours “on the books” and 16 hours off. To have the team absorb those extra 16 hours means that they will have to take it out of their own time which is violating the idea of sustainable pace.

If you have a team that consistently misses their estimates, that’s something different. Missing estimates should be the exception rather than the rule. When it becomes the rule, the question is what’s going on? Is team morale low? Why?

When using story points combined with tracking velocity, there shouldn't be a problem with estimates because story points and velocity should make the issue of estimates a moot point. What I mean is that if all estimates are suddenly off by a factor, the result is simply that velocity goes down. If estimates have always been "off" then velocity will not be going up or down, it will just be fairly constant. Since story points and velocity are relative, estimates that are "off" by a factor aren't a real problem.

Instead, the “problem” may be manifesting itself more as a general feeling of “this team just doesn’t seem to be as productive as it could be.” But then you are back to “why not?” I don’t think an enforced framework of commitment will change anything if the team has, for instance, low morale.

On the other hand, if the problem is wildly varying velocity (perhaps due to wildly varying accuracy of estimating stories) then I would venture there is a good chance that the stories are just too big in general and a good remedy to consider is to aggressively split stories into smaller stories.

Scrum: No Commitment Required!

2011-08-31T23:09:00.001-04:00

A couple of years ago I first talked about applying the decoupling principle to Scrum. Since then I've turned a bunch of that material into a presentation called “Scrum and Kanban Like Chocolate and Peanut Butter.” I’ve made many updates to the presentation and presented it to a wide variety of audiences. One of the main points of the presentation is that all of the things in Scrum that are coupled to iterations can be defined in such a way that they are no longer coupled to iterations. I argue that once you have done that with everything, there is really no value left in an anchor point which anchors nothing. That is, there is no fundamental need for iterations in Scrum.

From time to time I’ve realized that I forgot to decouple something, figured it out, and added it to the presentation. For example, burndown charts, burnup charts, and per-story timeboxes. The per-story timeboxes were tricky to even notice because in Scrum we generally think of timeboxing at the iteration level, but in effect that also imposes a timebox on individual stories. If you remove iterations without taking this into account then you are also removing the per-story timeboxes.

But recently, buried among the many feedback forms from presenting at Agile 2011, I found three little words: “What about commitment?” Ouch. I had missed a big one! The Scrum team makes a commitment that is tied to an iteration. How could I have missed that?

As I thought about it more, I realized that I do actually address this point in an off-hand way when talking about decoupling story assignment from iterations. However, I am very grateful for the comment. It helped me realize that I had been effectively dismissing commitment as though breaking a promise with a shrug.

Scrum's Commitment

It turns out that Scrum's commitment is a bit too strict. Even the version of the Official Scrum Guide as revised by Ken Schwaber and Jeff Sutherland, the co-creators of Scrum acknowledges this. In the new version of the Scrum Guide, the commitment has been replaced with a forecast. The idea is that the forecast represents the combination of what the product owner wants and what the team believes it can deliver as the next shippable increment. This is a reasonable response, but I think something is lost in the change. Let's look at this another way.

Mutual Trust

The original idea behind Scrum's commitment is that the team commits to finishing what they have signed up for for the sprint and everybody else commits to leaving the team alone. It is really two commitments based on mutual trust. Without this mutual trust, both commitments break down. But both commitments are actually completely unreasonable and don't reflect reality. The goal is good, but the implementation is poor.

The problem is that the team is not prophetic. They can't possibly know how the work will turn out until they have actually attempted it. More often than not, their estimates will be wrong. Sometimes a little, sometimes a lot. Things happen. The product owner is not prophetic either. They can't possibly know what demands of the business will come up or how customer needs may change within the timeframe of a sprint. A set in stone commitment makes it difficult for both parties.

In practice, the most important ingredient is mutual trust between the team and the product owner. This rapport allows the team to go to the product owner when they discover they have made an unrealistic promise and ask to switch out a story. It also allows the product owner to come to the team with a new request that is more important than another unstarted story and ask to swap them.

Decoupling the Forecast

If Scrum now talks about a forecast, and we are comfortable with the transition of commitment to mutual trust, then the real question is how do we decouple the forecast from iterations? First of all, ask yourself if you need it. If you don't, then there's nothing left to do.

On the other hand, if you feel the need for a forecast then the same approach can be used as for decoupling other aspects of Scrum, you simply redefine the timeframe. By default, the forecast is tied to the timeframe of the iteration. If your iteration is currently 2 weeks long, then go ahead and provide a two week forecast. It is a subtle point, but by defining a forecast period on its own, it is no longer tied to your iteration length. For instance, you may be doing four week iterations and then move to two week iterations. But if you decouple the forecast from the iteration then you can keep your four week forecast. And once again, after you have decoupled everything in Scrum from iterations, what's the value of an anchor point that has nothing anchored to it?

You may also be interested in the follow-up to this post, "Sticking to Estimates"

Scrum and Kanban Like Chocolate and Peanut Butter Slides

2011-08-31T17:15:00.000-04:00

For those of you that have attended one of my recent talks on "Scrum and Kanban Like Chocolate and Peanut Butter," here are the slides for the presentation. If you haven't already attended the session, be warned that there are no speaker notes and clicking through rapidly may result in vertigo due to the high degree of animation contained within.

Agile 2011 Highlights and Why Go to Agile 2012

2011-08-15T03:19:00.000-04:00

For the third year running now I've produced a video of highlights from the Agile 20xx conference. The intention is to give a flavor of the conference and also to motivate folks to consider going to the next one. It is amazing how much value one can get out of the conference and my hope is that more people will experience that value.

The first year was a single take done during the "open jam" session at Agile 2009. The second year I used iMovie on the iPhone 4. This year I got slightly more sophisticated using iPhone 4 for video capture and iMovie on the MacBook Pro for editing. This year I've added photo overlays, voice overs, and audio normalizing to my bag of tricks!

This video has it all. A fight scene (!), comedy, acrobats on skis, interviews with attendees, clips from the sessions, and much more. Check out the short version on YouTube! An extended edition will be available soon.

Agile 2011 - Back to the Future!

2011-08-05T22:31:00.000-04:00

Here are my thoughts on anticipating Agile 2011, filming Agile 2011, and reflecting back on the past 10 years of Agile.

Looking Forward to Agile 2011
I can't believe that Agile 2011 is upon us! A chaotic action-packed week mixed with being away from the family. And then after the event there is the week of withdrawal . No matter how wonderful and Agile your company may be, it can be jolting to move from "The Agile Future" back to whatever level of Agile you are currently at.

But I wouldn't miss it for anything. Every year there is a smorgasbord of sessions to choose from and this year is no different. Out of the literally hundreds of sessions, I've narrowed it down to 22. That includes the keynotes, and special events. If it is anything like last year, after I get over withdrawal I'll be drawing inspiration from those 22 sessions for months and hopefully it will fuel my Agile fires until "Agile and Beyond" and the Lean Conference in the Spring.

Filming Agile 2011 to Inspire Others
I strongly believe that everybody involved in software development should go to Agile 20xx every year. The ROI is just unbelievable. Of course, that will probably never happen. But my hope is that I can inspire at least a few more people to go and that it will make a difference. Towards that end, in 2009 and then again in 2010 I did a combination "Agile 20xx Highlights" and "Why You Should Go" movie completely filmed and for 2010 completely edited on my iPhone.

This year I'll be bringing along my iPad 2 which boasts an updated iMovie and will do it all again. Once again I'll be looking for people to interview about their experiences and to find out more about why they go. If you think you've got some good material for me, let me know. Also this year I'll be trying to do some video blogging for those of you back home to give you a visceral feel for what it is like and hopefully inspire you to come to Agile 2012!

I'm also open to suggestions for what to film. What questions would you like me to ask, what would you like to know about Agile 2011 that I can capture on video?

Looking Back on Ten Years of Agile
Actually, that's a misnomer. The Agile Manifesto was drafted in 2001 in Utah (thus the location for Agile 2011), but actually what they did in that meeting was to coin the word "Agile" and produce the manifesto to describe what the signatories felt was common about the various things they had already been doing for some time.

But anyway, what's the "big takeaway" of the past ten years of Agile? In my opinion, it is that there is now a struggle between Agile becoming fossilized and breaking out of the Branding Wars to reach its true potential. Let's face it, Agile was not handed down to us from a future where everything has been figured out and they have been doing it successfully for 100 years. No, a group of bright folks started talking about "Agile" 10 years ago and we've all been trying to figure out what to do with it since then. Along the way we've gotten "XP" and "Scrum" and "Kanban" and "TDD" and "Continuous Integration" and "User Stories" etc etc. It has been a true Renaissance of interpersonal and technical learning and growth.

But there is also the inevitable desire to have something definitive and unchanging and solid and secure. Scrum is currently that thing. What's wrong with that? Well Scrum is good, better than "waterfall," but it is far from perfect and it is only a subset of Agile, not the definition of Agile. Personally, I think there is much more value in the parts of Scrum than the sum of those parts. Things like product owners and backlogs are much more valuable in the long run than artificially binding them together in a particular way. More important is the process of figuring out which pieces work best in your organization and having the knowledge and skills to do that process well.

Right now, it looks like Kanban may be just the thing to break the Scrum stranglehold. So let's all go out and learn how to do Kanban exactly right and by the book to do just that, ok? Oh wait... I've got a better idea, how about we think about how to combine Scrum and Kanban like Chocolate and Peanut Butter. And while we're at it, how about a little pair programming from XP to spice things up?

Agile Training in Boston, New York City, and DC

2011-01-13T14:29:00.000-05:00

AccuRev will be providing its amazing Scrum User Training which is also at an amazing price in Boston, NYC, and DC later this month. If you are considering getting Agile training, you should definitely take a look at this unique combination of content.

AccuRev's Scrum User Training Comes to You!

2010-10-23T10:41:00.002-04:00

There are now three cities where you can get an amazing deal on AccuRev Certified Scrum User Training.

This is a unique course that not only teaches Scrum, it also covers the basics of other practices and techniques that you will need to succeed at Scrum. This includes things like user stories, story points, planning poker, continuous integration, unit tests, refactoring, and more!

We initially did these courses on site (which we still do) and are now providing them in cities across the US. The response has been tremendous. People seem to really enjoy this course and have a lot of fun learning through the hands-on activities.

These courses are for everybody involved in development: management, product managers, developers, testers, DBAs, technical writers, project managers, etc. You'll form into cross-functional teams, pick your own software product to build, and do the exercises as a team. This is a great team-building opportunity.

The course is also available for on-site delivery and can accommodate up to 200 participants at a time. Cost is based on an initial fee plus a small additional fee per person and travel and expenses. Instead of training just a small initial team of Scrum Masters, now you can train your whole enterprise for the same price. For more information, contact sales@accurev.com and tell them I sent you! :-)

AccuRev's Scrum User Training Coming to Santa Clara

2010-10-15T03:35:00.000-04:00

I'll be bringing our Scrum User Training to Santa Clara on Dec 2nd. It's a full day of hands-on activities that will introduce you to Agile and Scrum. If you've been considering Agile, this is a great way to introduce it to your whole team at an amazingly affordable introductory price.

Dallas: Scrum and Kanban Like Chocolate and Peanut Butter, Sept 15th

2010-08-27T16:19:00.000-04:00

If you will be in the Dallas area on Sept 15th, come get a brief introduction to Kanban and see how it may be able to add a little extra flavor to your Scrum implementation. Find out how "One Piece Flow" is key to both Scrum and Kanban, learn about Limited WIP, and how the decoupling principle is a key principle of Kanban that can be easily applied to Scrum. I'll be keynoting with the presentation "Scrum and Kanban Like Chocolate and Peanut Butter" which was standing-room only at Agile 2010.

Come and Re-Examine Your Beliefs at Nashua Scrum Club

2010-08-27T10:00:00.000-04:00

It has been a while since I've blogged here. Why is that? Well, I've been blogging over on the AccuRev site, doing more and more speaking engagements, and twittering like mad. I'll be blogging more here soon. In the meantime, if you are interested in what's been on my mind lately and you are in the Boston/Nashua area, why not check out the Nashua Scrum Club meeting on Sept 9th? I'll be doing a presentation that takes a look at how our beliefs and the beliefs/culture of our team/organization/customers influences the success and failure of Agile adoption. Hope to see you there!

Mary Poppendieck to Speak. Waltham, MA Jan 7th, 6pm

2009-12-12T16:28:00.001-05:00

As I write this, 52 folks have already registered for Mary's talk and there are only 120 total spots available. Her talk is titled "The Leadership Team and the Software Crisis: A Cautionary Tale" . Mary's talk will tell the story of a large company as it attempts to turn theory into practice, uncover barriers to sustainable change, and look in unexpected places for the root cause of the software crisis. She will also discuss governance and the role of the leadership team in developing software-intensive systems.

If you've never heard Mary speak, now is your chance. She's a terrific speaker who always has great material. And to top it all off, this event is not only free, dinner is included!

Registration has only been open for a couple of days. If you are thinking of going, now is the time to register!

Second Edition of "Do It Yourself Agile" Available as Free Download

2009-09-29T12:51:00.007-04:00

At the beginning of the month I released the first version of "Do It Yourself Agile." The response was tremendous and I am very grateful to the many people that blogged and tweeted about it. I am also very thankful for all of the feedback and suggestions that I have received. In total, the book has now been downloaded more than 4,000 times.

Here is the second version, which takes into account all of your feedback and also incorporates new sections based on recent blog posts. Please keep the feedback coming. If there is something you feel is missing, let me know. New material usually starts out as a blog post first, so you'll get immediate feedback.

While I believe that an Agile coach, whether recruited internally or externally, is still the best, fastest, and least expensive (in terms of ROI) path to Agile success, not everyone can do that. It may be politics, budget, or some other reason that prevents folks from using seasoned coaches. In any case, DIY Agile is here to stay, and the book "Do it Yourself Agile" is intended to be a resource you can lean on as you transition to Agile on your own.

Let me know what you think. I look forward to producing more versions incorporating your feedback.

The book is freely downloadable, no strings attached. I only ask that if you point people to the book, please point them to this blog post rather than linking directly to the pdf, copying the pdf, or providing the content in some other format.

"Do it Yourself Agile" (pdf) 180+ pages

"Do It Yourself Agile" - condensed web version

Planning Poker Reduces Risk and Waste

2009-09-28T16:48:00.004-04:00

There are three particularly valuable things that can happen during a Planning Poker session. They may also happen during any flavor of estimation meeting, but seem to be more likely when using Planning Poker.

The Bigger They Are, The Harder They Fall

I recommend that you never use a story size greater than 13. Most stories that are estimated at 8 points or above can be split into smaller stories. You should always be looking for opportunities to break larger stories into smaller stories. If you have an 8 point or larger user story, it is probably actually two or more stories in disguise. The bigger the story, the more likely your estimates are wrong and the more likely that you have many smaller stories masquerading as one large story.

For instance, you may find that a story that originally looked like a 20 point story was really two 8 point stories, one 5 point story and 2 three point stories for a total of 27 story points. Better to break that huge story down into its five constituent stories and estimate them each individually.

But remember, a story is only a story if it provides value to the user. Splitting a story up into “As a user I want the backend for X” and “As a user I want all of the UI for X” is not the right way to go. If you can’t create smaller stories that still provide user value, then the story is already as small as you currently know how to make it.

Know When to Walk Away

You may realize while discussing a story that the story contains a big unknown, something that feels like it won’t be resolved during the implementation of that story. In that case it is best to split the story up into a research story and the story itself. For instance, “As a developer I want to know how to XYZ.” That way, if you never do figure out how to do the unknown part, you haven’t invested any effort into the overall story. This technique should only be used as a last resort, but it is much better to do this than to know going in that there is a big unknown and have to pull the whole story out near the end of the iteration.

Know When to Fold’em

Lastly, you may decide that you just don’t have enough information to estimate a story. In that case, you should have a mechanism for informing the product owner such as marking the story “need more info.” There’s no point spending time on estimation if there is insufficient information to do so. You’ll just be glossing over the problem and producing a false sense of security.

Breaking out research stories and kicking stories back to the product owner may seem like procrastinating, but it tends to build good habits. It keeps the team from committing to work that includes research projects, clearly defining some stories as research stories, and keeping the product owner on his or her toes.

Counting Your Chips

In summary, if you are mostly working on small user stories that have end user value, you are reducing the chance that you are putting something into the product that never gets used and you are also reducing the chance of starting work on something that never gets finished or has to be discontinued part of the way through. The smaller your stories, the smaller your risk and the less effort you’ve wasted when you run into problems.

See Also: "The Five Essential Ingredients of Great Agile Estimation"

The Five Essential Ingredients of Great Agile Estimation

2009-09-28T16:45:00.004-04:00

Planning Poker is a useful Agile estimation technique (see previous post "Introduction to Planning Poker"), but it is even more useful when used in conjunction with whole teams, user stories, velocity, and story points. I refer to these five practices, used together, as the five essential ingredients of great Agile estimation.

Whole Teams

A key practice of Agile development is the use of whole teams. A whole team is a cross-functional team comprised of between 5-9 people. By cross-functional I mean the team as a whole contains all of the skills needed to accomplish the team’s goals, not that each team member is capable of doing any task. If you have more than 9 people, then you split people up into multiple whole teams.

Planning Poker reminds everybody that the estimate includes all of the work that needs to be accomplished in order for the story to be considered done. It includes development, testing, documentation, and anything else required for the story to be considered done. It is a reminder that nobody on the team gets credit for a job well done until the whole story crosses the finish line. Instead of people saying “well, I finished the development, I don’t know what is taking the tester so long” they are more likely to say “what problem are you running into and how can I help?”

User Stories

Because user stories are a simple and easy to understand description of the work, user stories allow you to focus on estimating rather than spending lots of time discussing what a particular enhancement request or requirement really is. To maximize the benefits of Planning Poker, you need to be good at creating and using user stories.

Story Points

In my experience, the best unit to use for estimates is story points. Two different people with two different skill sets or levels of ability in an area may take different amounts of time to perform a particular task. Estimating in hours mixes together the scope of the work that needs to be done with the speed at which a particular individual can do that work.

On the other hand, story points are a relative measure of the scope of a user story. Story points separates out the “what” from the “who.” For instance, if you have one individual that is stronger with .Net than with Java, they will estimate a Java story as taking more hours than somebody that is stronger with Java. But they will probably both agree that something that is twice as easy to implement will take half as long to do.

To use story points, you need to create a relative scale of scope. A simple approach is to find a simple and straightforward story that you use to represent a single story point. Then think of stories that are 2, 3, 5, and 8 times larger in scope. You should have a couple of examples for each story point value to take into account that some stories have more test than coding, more documentation than test, etc.

Story points are primarily used for planning, not for implementation. Story points are used to help determine the contents of an iteration by calculating a velocity.

Velocity

In Agile, the velocity of a team is simply the number of story points associated with stories that are finished in an iteration. For instance, if the team completed 8 stories that were each 5 points in an iteration, then their velocity for that iteration was 40 story points. In a stable team, a team that is comprised of the same individuals working full time as part of that team, the velocity is a good measure of the overall throughput of the team.

Knowing your velocity helps with planning. For example, if you know that the velocity of your team is 40 points, then you know you can expect 40 story points for each iteration. The team decides which stories to take based on the backlog which is maintained by the product owner.

More Than the Sum of The Parts

Regardless of which of these practices you are currently using – whole teams, user stories, story points, velocity – Planning Poker is an excellent tool. The more of these practices you use, the more they reinforce each other and the more value you will get out of Planning Poker. Conversely, the more you use Planning Poker, the more value you will see in implementing all of these practices.

Next: Planning Poker Reduces Risk and Waste

Introduction to Planning Poker

2009-09-28T16:42:00.003-04:00

Many of the techniques of Agile development combine together to provide more than the sum of their parts. One technique that definitely fits this description is the Planning Poker method for doing estimation. In part 1 of this post, I will quickly describe the mechanics of Planning Poker. I will then go on to describe how user stories, whole teams, story point estimation, and velocity all work to multiply the value of Planning Poker and how Planning Poker also reinforces the use of and value of the other practices. If you are already familiar with Planning Poker, you may want to skip ahead to "The Five Essential Ingredients of Great Agile Estimation".

Planning Poker is an estimation technique first described by James Grenning in 2002 in a paper by the same name [pdf]. When I first heard about Planning Poker, I couldn’t help but chuckle. It seemed a bit silly to mix software development and poker. After reading Mike Cohn’s excellent book “Agile Estimating and Planning” I had a good grasp of the mechanics of Planning Poker, but it felt like just a variant of the Delphi method and thus nothing new.

About a year ago we had Kenny Rubin come in to do some Scrum training for us. That was just the shot in the arm that we needed to supercharge Agile adoption at AccuRev. If you are looking for an Agile trainer, I highly recommend Kenny Rubin. It was based on Kenny’s explanation and recommendation that we decided to try Planning Poker on a real project. The results were terrific and after a single session we were hooked. Planning Poker is now a standard part of our Agile process.

The Basics
In order to play Planning Poker, each participant will need a deck of Planning Poker cards. These are easy to obtain, just Google for “Planning Poker cards” or make your own. You will need the following cards: ½, 1, 2, 3, 5, 8, 13, 20 and a card that indicates “I’ve had enough.” The numbers on the cards represent estimates. You can use story points, ideal days, or some other measure, but in this point I am assuming (and advocating) story points. Typical decks contain cards for much larger numbers than 20, but I think you’ll find that 20 and higher are rarely used once you’ve been using Planning Poker for a year or two.

The whole team gets together for a set amount of time to do story estimation. An hour is usually a good amount of time, but this is completely up to you. When I say “the whole team” I am assuming that you are using small cross-functional teams of 5-9 people (see later parts that discuss the use of whole teams).

Estimation is done on a story-by-story basis in the same order as the backlog, starting with the first story that needs estimating. Somebody reads and describes the story. This is often the product owner, but could be anybody. Some teams do story point estimation without the product owner and just skip stories which they are unable to do without the product owner’s involvement. After the story has been read, participants discuss the story for a bit and then decide on an estimate by picking one of their cards and laying it face down in front of themselves. Once everybody is ready, all of the estimates are shown. If the estimates are all the same, you are done.

Usually, some of the estimates are particularly low or particularly high. A low estimate can indicate that the estimator left something out, or possibly that they know a way to reuse existing work or have some other good information that other folks weren’t aware of. A high estimate may indicate many things, but most commonly it means that the estimator is thinking of something that other folks may not be aware of. For instance, somebody may point out that there is no test framework or tooling for a particular technology and that to adequately test the story the team will need to do a lot of setup work.

In any case, after a round of discussion, everybody chooses a new estimate. This is repeated until the team comes to consensus on the estimate. The estimate is given to the story and then you move on to the next story. You end when you hit the end of the meeting time, you run out of stories to estimate, or somebody plays the “I’ve had enough” card.

The Benefits

One of the biggest benefits of Planning Poker is the sense of team that it creates. The whole team is participating in the estimation. This creates a greater sense of team ownership and team responsibility for each story. Another major benefit of Planning Poker is that you leverage the collective wisdom of the whole team. However, this really only works well when you are using whole teams in the Agile sense of the term.

Next: The Five Essential Ingredients of Great Agile Estimation

Software Development is a Search Algorithm

2009-09-23T16:29:00.002-04:00

How we think about something can have a profound effect on how we do that thing. Thinking of the process of software development as a search algorithm rather than as the process of building software can lead to making better decisions about the products we create and the way we go about creating them.

The Search For Value
The end goal of designing software is to produce something which provides a benefit that is large enough for one or more customers to at least cover the cost of its production. Of course, the higher the value of that benefit, the more that folks will pay for it. On the other side of the equation you want to provide that benefit at the lowest cost. There is a point of diminishing returns both in looking for the highest possible value and in looking for the lowest possible cost.

As much as we might want this process to be deterministic, it isn’t. Just because you produce something doesn’t mean people will use it, and just because they use it doesn’t mean they will pay you what you want for it. The only way to truly find out is to produce something and see how much value can be realized from it, for instance by charging money for it.

The Max For the Minimum
In other words, producing software is a classic min/max search. You are searching for business opportunities with maximum value and implementations which minimize cost. You can think of it as a huge graph of opportunities and below each opportunity node is another graph of possible implementations. Complicating this is the fact that you would like the features to be close to each other from a market perspective in order to maximize the leverage of your sales and marketing departments and you would also like to have as much common implementation as possible in order to maximize the leverage of your existing implementation and engineering resources. For both you would also like to leverage your core competencies.

The faster you can traverse this graph, the faster you will unlock business value. Unfortunately, you only have a finite amount of time and resources, so you can’t run a search by producing all possible product variations and seeing how they sell. Plus, the search space is infinite. So, you have to make predictions. You look for ways to eliminate certain paths and to get more information about other paths without actually implementing them. Whatever your methods are for figuring out what to produce, they are all based around searching the solution space as fast and efficiently as possible.

Searching For Value in All The Right Places
You may think that only parts of the development process are parts of this search, but doing so will limit the effectiveness of your searching. For instance, if you arbitrarily decide that a particular feature is what your customer wants and you then spend a huge amount of time talking to your customer base about the requirements for that feature, you are limiting your search to the various ways to do that feature, but what if the perfect implementation of that feature is only worth $10? You are then searching within a low value set of possibilities. Before you dive in too deep, spend a fair amount of time at a high level so as to maximize the value of searching at the next level down.

The entire development lifecycle is part of your search. The first level of searching is to find potential high value opportunities from which to create new functionality. But don’t fall in love with a particular opportunity too quickly because implementation may cost more than the value to the customer. There are usually a variety of ways to implement something, so don’t fall in love with a particular design too quickly because there may be a cheaper implementation. And of course in order for the customer to determine whether they want to pay you or not they need to get their hands on it or at least see it in action. So, you are searching for opportunities and then within the opportunities you are searching for implementations and then you need to circle back with the customer to check the results.

Next: Specification is Planning is Design is Coding is Approximation

Specification is Planning is Design is Coding is Approximation

2009-09-23T15:32:00.007-04:00

All Phases of Software Development are Variations on a Theme

It is easy to think of specification, estimation, planning, design, implementation, and customer feedback as completely separate and unrelated activities. However, a useful way to think about them is as variations on a theme where the theme is producing a product which provides the best return on investment as measured by the ratio of revenues to cost. Each of the phases of the development life-cycle can be thought of as a combination of discovery, prediction, and design. The prediction comes into play when you make a decision. When you decide that you will schedule a particular requirement to be implemented in your product, you are predicting that the end result is something that will provide value to the customer. You have no way of being sure until the customer has actually used it as part of their normal course of business.

What does it matter if in fact it is true that all of these activities are variations on a theme? I believe it provides a conceptual framework that can help us make better decisions about the products we create and the way we go about creating them.

Requirements Gathering is Not Design
Requirements gathering is not the process of finding out what customers need, it is the process of understanding the customer. The problem with producing elaborate requirements is not just that requirements change over time or that people change their minds, it is that users can’t tell you what they want because they don’t know. Customers cannot tell you ahead of time what they will actually spend money on. At best, requirements are an approximation of what customers need and a best guess of what will provide them value at the time that they actually use what you implement for them.

Gathering requirements from customers and the market in general is much like removing compiler errors. For any particular module that you are working on, the first error is the most accurate and accuracy goes downhill rapidly from there. After you fix the first error, the results from your compiler or IDE will probably be very different. This is the same with customer requirements. For any particular functional area, the first thing customers ask for is most likely the most accurate. Once they see the new version which incorporates that feedback, what they want next is probably very different than what they would have asked for next before they saw the new version.

The most you can hope for is to gather information and ideas that are useful in producing a successful product. If you just build whatever customers ask for, then you are ceding design to them. Customers cannot design for you. If they can then they are not just customers they are also designers. It may be that in your situation your customer is also a designer, but then you should explicitly think of them that way. Otherwise, you run the risk of making the mistake of blending the process of gathering requirements and creating a design.

When you have multiple customers, it is especially true that the customer cannot tell you what they need. It is rare that you will be able to do everything that all customers asked for, let alone everything that any one customer asked for and thus you will have to predict which combination of requests that your customers asked for will produce the best result.

Specification is Design

When you write a Marketing Requirements Document, Product Requirements Document, Specification Document, or whatever combination of documents you create along these lines and whatever you call them, let's call that "specification." A specification is a design. It is a design because you are taking all of the feedback from the customer and saying, at a high level, "this is what will provide value to the customer." You are deciding and predicting that this specification rather than any other specification is the way to go forward. It is the same as deciding that this line of code vs that line of code is the right way to go.

Specification is the First Approximation
When working with software, even if you have clear requirements/specifications that you have validated with users, it is not always clear how you will implement them.The specification is an approximation in two ways. First, it is only an approximation of what users actually want. Second, it is only an approximation of the final result. This same statement holds true at every stage of the game, from specification to design, to plans, to estimates, to the code itself. From initial customer interaction to final implementation you are building approximation on top of approximation.

I'm not advocating against specification, estimation, or design. It is useful and even necessary to produce these, and you will often hit on solutions or eliminate problems much more quickly and much more economically than jumping right to implementation, but in the end they are still only an approximation of reality, an interlocking web of educated guesses.

Not until you actually attempt to create something real will you find out how close to reality all of these approximations are and the real costs and difficulties involved. The longer you take to turn these approximations into reality, the more likely it is that they will be inaccurate and the larger those inaccuracies will be.

Estimation is Design
Estimation, which is also another word for approximation, is part of design. If somebody gives you a requirement do you just shout out a random number as the estimate? The accuracy of your estimate depends on the amount of design that you put into it. You may not think of estimation as design, but it is. Let’s say that somebody says “I need a module which will calculate the sum of an array.” If you have such a module at hand, you will give an estimate that involves integrating the existing module into the overall product. Deciding to use that module is a design decision. If you don’t have such a module at hand, you will give an answer based on your prior experience doing the exact same thing. The decision to use an approach similar to a past experience is also a design decision. It is part of creating the final design.

On the other hand, if you are given a more complicated requirement, then you will probably break it down into sub-tasks and estimate those. That forces you to think about what it will really take to implement the requirement. The process of deciding what the subtasks are is actually design. The more design you do, the more accurate you can be. The accuracy of an estimation is on a spectrum from “wild guess” to delivery of product, from complete prediction to statement of fact, from “I believe it will take 10 days” to “it took 103 hours.” The other aspect here is how much is research/design and how much is counting. When well-known tasks are what is at hand, then it is a matter of counting. If you know you need to change 20 dialogs and each change will take 10 minutes then you know the task will take 200 minutes. That’s simple counting. If you need to produce something that has never been done before, then it is a research project and you won’t have accuracy until you have removed all of the uncertainty.

Planning is Design

The specific set of features that go into a product is in fact a design. By selecting that set, you are deciding (predicting) that this is the optimal set of features to achieve a chosen goal. While it is true that you may just be choosing a set of work items that sum up to the available time for a release, that is still design, but perhaps not the optimal design.

Design is Design
Obviously, the activities that we associate with “the design process” are part of product design. The design that you have prior to implementation may be a fully fleshed out design document with lots of illustrative figures, it may be in the form of UML, or it might be entirely in your head. As with requirements and estimation, the design is also an approximation. The design that you do prior to implementation is only an approximation because during implementation you have still more design to do and you will discover information that you didn’t have during your initial design.

Implementation is Design
During implementation you are doing two kinds of design: micro-design and re-design. Micro-design is all of the little decisions you make while coding. Will you use a vector or a doubly-linked list? Will you specify an index or let the RDBMS do it for you on the fly? Re-design happens when you find out that once it came time to implement something, an unexpected problem arose. For instance, the design specifies the use of a particular third-party library, but in reality that library doesn’t provide the functionality that it claimed to provide. Another example is that the implementation works fine, but doesn’t provide the expected performance and no amount of tweaking seems to be doing the trick.

Final Validation of the Design
Finally, when you deliver new features to customers, you discover how close your approximations were. You discover what customers like and don’t like, what they use and what they don’t use, what they will pay for and what they will not pay for, what provides value and what does not provide value. This information can then be used to determine what you will put more effort into and what you will discontinue. The tighter you can make the loop from discovery to delivery, the faster you can produce the product that your customers really value.

Next: Software Development is a Search Algorithm

Management: Top Ten Things Programmers Hate About Agile

2009-09-21T13:29:00.005-04:00

Sometimes Agile emerges from a grassroots initiative, but as Agile gains buzz, more and more managers and executives are initiating the move to Agile. Folks in management are often caught off guard by the reactions they get from their programmers. I’ve compiled a list of common gripes that I have heard and/or experienced firsthand to help folks in management avoid getting off on the wrong foot.

While this list may also apply to folks other than programmers, this post was created specifically with the good folks over on reddit/r/programming in mind. By the way, before posting there, I highly recommend reading "The Unspoken Truth About Managing Geeks." If you are a programming subreddit regular, thanks for visiting. I look forward to your feedback and hope that folks in management reading this take your feedback to heart. Without further ado, I give you the Top Ten Programmer Gripes with Agile!

#10 Agile is Snake Oil sold by Snake Oil Salespeople
Let the buyer beware! There is definitely an element of truth to this. “Agile” can mean almost anything and there are definitely folks out there taking advantage of the fact that many people are new to Agile. Not only are they just looking to make a buck, they often either have no idea what they are doing or have simply repackaged whatever they used to sell with the “Agile” label. The good news here is that these folks are pretty easy to uncover. Just keep in mind that there are Agile snakeoil salesfolks out there, do your normal due diligence when evaluating Agile advocates and you should do fine.

#9 Agile Has Too Much Jargon
Say, how many story points did we burnup during this sprint? It seems like our velocity went up after the product owner started using user stories in the backlog. What’s up with all of this jargon? Is it really necessary?

Rather than answering that question directly, let me ask you this: doesn’t every subject have its own subject matter jargon? Isn’t OO programming “jargon rich?” Isn’t .Net “jargon rich?” Heck, programming in general is “jargon rich.” I’ll bet you didn’t even bat an eye when I used the jargon terms “object-oriented” and “.Net” ! Oh wait, I didn’t even say “object-oriented,” I only said “OO.”

#8 Agile Means The End of Freedom
This is an interesting one because when Agile is implemented well, there is actually much more freedom for all involved. I think the message here is that management needs to keep in mind that programmers are understandably wary of new initiatives that smack of yet another way to impose control. When talking to programmers about Agile, be sure to emphasize that Agile will affect the whole organization, management included. After all, “waterfall” is primarily a management method and does not include specific technical practices.

It is true though that Agile means focusing on customer value and frowns on major re-architecting projects. Agile promotes incremental architecture where the architecture is built or improved over time and directly tied to implementing customer value.

#7 Management is Only Doing This Because They Think We’re Not Very Good
If you want Agile to succeed, you need to point out, and be sincere about it, that Agile will affect the whole organization, management included. Otherwise it can come across as an “us vs them” situation where management isn’t the problem, the programmers are the problem. And lets face it, waterfall is a management method, not a programming language.

See also: "Better, Faster, Stronger Programmers" .

#6 The Agile Consultant Will be A Pain in The Neck
From experience, programmers know that if you are going to introduce some new thing there’s probably going to be a know-it-all consultant coming in to point out all of their flaws and give generic advice without even getting to know what they do.

Let the buyer beware! See #10 - "Snake Oil"

#5 Gross! Pair Programming!
You probably don’t have to worry about this one because you probably won’t introduce it anyway. On the surface it seems like paying double. While there is definitely value in this, it is far enough off the beaten path of Agile that nobody reading this needs to worry about it.

#4 Stand Up Meetings == The Inquisition
Stand up meetings are not a question and answer period. . If somebody turns them into an inquest, you have a problem and it isn’t Agile. Agile can’t help you with this, but it can make it more obvious. If you are a manager and feel tempted to turn the stand-up meeting into an interrogation, you are going to end up with a room full of unhappy and uncooperative programmers. Time to re-read "The Unspoken Truth About Managing Geeks."

#3 Drive-by Agile Transition
Some managers (not you of course) read an article about Agile, declare “we’re going Agile” and then ask at all future meetings “how are we doing on this Agile thing” and then don’t react well when they see lots of blank stares.

If you do this, Agile will fail.

One other point on this, not only will an Agile Drive-by fail, your programmers will be pretty upset that they weren’t consulted in the decision. Going Agile is a big change and you are much more likely to succeed if you involve everybody in the decision.

#2 Management Isn’t Fully Committed to the Transition
Going Agile takes sustained commitment. Not only will you need to see the transition through the tough parts, you’ll also need to provide budgetary support. That is, you need to fully fund the transition.

Depending on the size of your organization, it is likely that you will need to hire at least one Agile consultant to see you through the transition. You probably don’t have a Continuous Integration system, enough virtual machines, enough real machines, an Agile Project Management system, etc, etc. Whereas before you probably only needed one of everything, for instance an Oracle test environment, you will now probably need at least double. That said, see #10 - "Snake Oil."

#1 This is Really Just Another Way to Demand More with Less
Especially now, people are pretty sensitive to the call to “do more with less.” Agile can be perceived as a way to do more with less. But of all of the benefits that Agile provides, “doing more with less” is not one of them.

So, what is your pet peeve with Agile? Let management know!

Better, Stronger, Faster Programmers - We Have The Technology

2009-09-20T15:02:00.003-04:00

I recently got the following comment/question via reddit:

"Can we put this effort into making programmers better faster and stronger? Agile purveyors act as if they have an army of superhuman programmers already at their disposal and the only goal is utilizing them correctly."

Interesting observation. It raises all sorts of interesting questions, such as “what is better?” Really though, what is a “better programmer?” How do I know which programmers are better than others? How can I measure that? Is it the ones that produce the most lines of code? Is it the fewest defects? What if a programmer produces no defects, but very few lines of code and the code that they do produce isn’t used by anybody? How do I really know that it has few defects if it isn’t used by anybody? Perhaps the tests that would show that it is defective weren’t written because, again, nobody cares about that code?

programmer.upgrade(infinitely);
Let’s ignore the problem of comparing programmers and just assume that I have found a training regimen which creates better, faster, stronger programmers. If the programming effort of each individual was the only thing that determined the success of a project, we’d be all set, wouldn’t we? But most projects consist of at least two or more people interacting according to some sort of process. Many projects have nebulous goals and/or requirements. Quite a few projects get cancelled or shelved, either because they were clearly headed for failure or something more interesting came along. And let’s be honest about those so-called “shelved” projects. Do they ever come off the shelf? Don’t we just start over when we get back to that project?

All of this adds up to a system which has complex and unpredictable behavioral characteristics which are different than the sum of its parts. It isn’t enough to invest in better, faster, stronger programmers if the result never sees the light of day. Before investing a penny on any improvement project, wouldn’t it be better to have a way to very quickly find out what problems exist and very quickly see whether or not a solution to a problem actually makes a difference?

Let’s face it, traditional development methods aren't very good at providing visibility into the true state of the situation, nor at providing feedback on whether anybody is actually headed in the right direction. Agile is not a silver bullet and won't actually improve anything all by itself. Simply put, Agile is an improvement framework. It will expose quite painfully where the problems are, it is up to you to recognize them and solve them.

Programmer Planet
Any time I talk to folks about Agile, I point out that there is no magical place to get an army of superhuman programmers. According to IDC, there are ~12 million programmers out there. Those programmers include you, the folks you work with, and millions of other programmers just like the ones you’ve met.

Whoever you have now is pretty much the team that you are going to have to go forward with. Sure, you can probably switch who you have now with other folks, but that's no guarantee that you will get "better" programmers, only that you will get different ones. Better figure out how to make the most of the resources you have. And while I am a big fan of training and self-improvement at the individual level, why not leverage that investment by improving the framework within which those individuals work?

The Only Person Keeping You From Going Agile is You

2009-09-20T01:15:00.004-04:00

If you are interested in going Agile, but you are surrounded by folks that are not, don’t despair. You can still experience significant benefit from Agile on your own. Also, after reading this post you won't be able to say that somebody is preventing you from going Agile. This post is primarily directed at developers, but other folks may benefit from a similar approach.

The basic idea is to adapt the various Agile practices for a team of one. Let’s start by taking a look at scrum master, user stories, one piece flow, product owner, backlog, daily standup, iteration reviews, retrospectives, unit tests, and refactoring.

Scrum Master of One
Since there is only one person in your team, you are automatically the Scrum Master. Congratulations, but try not to abuse your new found authority over your team. Remember, the Scrum Master is a facilitator, not a boss! The key thing that the Scrum Master does is to facilitate Agile. The Scrum Master understands how Agile works, makes sure that all of the meetings are happening, and also looks for and removes impediments.

If you are going to be an effective Scrum Master, you need to understand Agile development. If you can get your company to sponsor you to take a Certified Scrum Master course, that’s a great way to get started. If not, consider sponsoring yourself. It is expensive and will probably require you to expend two vacation days, but I assure you it will pay off in the end. Short of taking a CSM course, go and get a copy of “Agile Software Development with Scrum” by Ken Schwaber. You may also be interested in checking out my freely downloadable book “Do It Yourself Agile.”

Reverse Engineered User Stories
The next step is to start working with User Stories. A user story is a simple statement in plain English (or whatever language you are primarily using) that describes the work that needs to be done from the perspective of the end user. They are generally of the form “As a user, I want X” where the “X” describes a goal that can be achieved with the software to provide value to the user. For instance, “As a user I want to be able to purchase anything that I am looking at with a single click.” The user story doesn’t describe the nitty-gritty details.

I realize that since you are not the product manager (or business analyst), you don’t get to determine what you are working on or how it is described. The way that you can benefit from user stories is to reverse-engineer whatever description you have been given of the work you need to do into user stories. The benefit to you is that you will either have a better high-level view of what you are looking to accomplish, or you will immediately start to see holes in the description of the work you have been given. User stories give you a framework for thinking about your work.

Once you start using user stories, you will start having more questions for your manager or the product manager or business analyst. At first, folks may see your questions as annoying, but over time I believe they will see your questions as reflecting your interest in their success and the success of the company.

Divide and Conquer
Now that all of your work is described in terms of user stories, think about the size of those stories. Can any of the stories be further broken down? For instance, if you had a story that was “As a user I want a simple calculator function” you could break that down into “As a user I want to be able to add a number to an existing value,” “As a user I want to be able to subtract a number from an existing value,” etc. Breaking your work down into user stories has many benefits from an Agile perspective. We’ll look at just a few of them here.

One Thing at a Time
Most Agile teams use iterations for organizing their work. As a team of one, that probably doesn’t make much sense. Instead, as a team of one, try to do One Piece Flow. The idea of One Piece Flow is that you do all of the work required for a single user story before going on to the next one. Try to resist working on two stories at once.

Product Owner For a Day – Creating the Backlog
Ok, now you have a bunch of user stories that describe everything you’ve been assigned to do and they are as small as you can make them. It is time to put on your product owner hat. As product owner, you need to rank each of these stories relative to all the rest in terms of value. If you are not sure what the value of the stories is, ask around to see if you can gather enough information to make a reasonable guess. Perhaps you know one of the actual end users. Ask them what they think about the stories you have.

Over time, you’ll get better and better at ranking your user stories. No matter what you do, having some sort of ranking is much better than none. Rank all of your stories from most value to least value and start working on the top story. This ranking is called a backlog.

The benefit of having a backlog of small user stories and working on just one story at a time comes into play when somebody taps you on the shoulder and asks you to do something other than what you are currently doing. When you are interrupted, you now have some options. You can point to your backlog and say “where do you think this new request fits in this backlog? Is it really more important than this task that I am currently working on?” If you are lucky, the person interrupting you will decide that their request has a lower priority than whatever you are currently working on. At the very least, the backlog provides a framework for discussing the relative priority of the new request.

If the new request does seem to be higher priority than what you are working on, you have one more tactic you can try. Since your stories are small, you can say that you only have one task in progress and how much work remains on the current story and suggest that you finish what you are doing prior to starting the new task. Over time, this tactic should work more and more often as you get a reputation for delivering higher quality work and being able to quickly change gears. The backlog examination exercise should also slowly train other folks on the value of user stories and the backlog. Your goal should be to have folks give you new requests and for them to say “let’s see where this belongs in your backlog.”

Even if you have to stop whatever you are currently working on and start working on a new task, it is much easier to change course if you only ever have one story in progress at a time and it is small. Don’t forget to treat this new request the same as any other task. Create one or more user stories for the request and order them relative to each other at the top of the backlog.

Once you get the hang of maintaining a backlog, consider making your backlog visible to others. One way to do this would be to print out your backlog in a large font along with an indication of which stories you are currently working on and post it near you in a place that passersby can clearly see. You may even find that the number of interruptions goes down as people can see for themselves that you are clearly working on the most important thing and leave you alone to get it done.

Talking to Yourself
It may seem silly to do a “daily standup meeting” with yourself every morning. So don’t stand up. But do go through the exercise of going through the three questions:

What have you accomplished since the last meeting?
What are you working on next?
What impediments do you have?

This will help to keep you focused on your current story. If you find that you just aren’t making the kind of progress you’d like to make, it is time to get some help! Let’s face it, if you go work on something else in the meantime, you’ll eventually have to come back to face whatever it is that is blocking you now. Yes, it is possible that you will solve the problem on your own, but why not at least find somebody to describe the problem to? Maybe the simple act of explaining the problem will provide a new insight. In any case, if you do end up working on something else, try to keep the total number of stories you have in progress to a minimum.

The Moment of Truth - Did it Work For You?
When you finish a story, now it is time to do an iteration review. Normally this would be done at the end of an iteration, but since you aren’t doing iterations, you might as well do the iteration review now. It is simple. Find somebody to demo the story to. The idea here is to have an audience and get feedback. If you feel like you aren’t ready to have somebody else see a demo of your work… then your story isn’t really done, is it? Keep working on it until you feel ready for an audience.

In Retrospect, That Was a Good Idea
Once you have finished your story and done the “iteration review,” it is time for the retrospective. Find a quiet place that you can spend 30 minutes or so to reflect back on how things went, from the time that you had the story ready to the time that you finished the iteration review. What went well? What didn’t go so well? What could you do better in the future? Write down your thoughts for future reference.

Unit Tests
If you aren’t already doing unit tests, now is the time to consider it. Unit tests are something that you can decide to do on your own for your own sake. At first it may seem that it is slowing you down, but consider all of the rework you have to do when QA finds problems with your work. Why not invest that effort into writing unit tests? If you are using Java, you can use JUnit. If you are using .Net, you can use NUnit.

Refactoring
Unit tests go hand in hand with refactoring. Refactoring is simply the practice of rewriting code to make it simpler to understand and change. The more unit tests you write, the more you will refactor the code and the easier it will be to understand the code and add new functionality.

Growing Your Team
See if you can get somebody in QA interested in what you are doing. You might look for their feedback on the ranking of your backlog or the quality of your unit tests. A QA person would also be a good person to invite to an iteration review. The more that QA sees the work you are doing as helping them, the more they will want to help make you successful. A close relationship with QA is an essential ingredient to Agile success.

As you practice these Agile techniques, I think you will find that you are producing higher quality results in less time as well as keeping focused on the work that the business cares the most about. You may also find that what you are doing gets noticed and appreciated. If you can get other folks to follow your example, you may soon find yourself part of a real Agile team. Good luck, and let me know how it works out for you!

If you enjoyed this post, you may also enjoy the freely downloadable book "Do it Yourself Agile."

Scrum and Kanban - Like Chocolate and Peanutbutter

2009-09-15T15:14:00.005-04:00

This post is primarily geared toward folks who have been doing Scrum for at least 6-9 months and are curious about what options are out there for further improving their process. One reason for looking beyond Scrum may be that you are bumping your head on the iteration boundaries. A common place to look is Extreme Programming (XP). There are many great practices in XP that complement Scrum. Even if you personally don’t want to do pair programming, XP has much more to offer than just pair programming, take a look!

Whether you’ve looked at XP or not, another great place to take a look is Kanban. Kanban is a process into itself and can be adopted whether you are doing Scrum or not, but I will be looking at Kanban as a source of good ideas to apply to an existing Scrum process.

One of the reasons that I am assuming 6-9 months of Scrum as the starting point is that Kanban requires a certain level of experience with breaking work down into small user stories and doing One Piece Flow which I believe is easier to attain via Scrum than starting with Kanban from scratch.

One of the biggest and most noticeable differences between Scrum and Kanban is that Kanban doesn’t have iterations. At first this seems outlandish, but consider that most of the impact of having iterations is at the beginning and end of the iteration. In the midst of an iteration, the iteration boundaries are immaterial; everything you are doing is story-oriented. In that respect, Scrum and Kanban are the same.

Our Old Friend: The Decoupling Principle
Now imagine that you are in the midst of a Scrum iteration… and the end of the iteration never comes. Whenever the team finishes a story, they move on to the next story, just as in Scrum. The difference here is that the “todo hopper” is being constantly filled by the product owner.

One question at this point is “but what about the per-iteration activities such as retrospectives?” And that is one of the first things about Kanban that can be applied to Scrum, with no need to remove the iterations themselves. There is an underlying principle in Kanban which is a well known and widely used principle in programming: the decoupling principle. When doing Kanban, you still need to do the equivalent of planning, assignment, estimation, retrospectives, delivery, etc. In Kanban, all of these activities are decoupled from each other whereas in Scrum they are all coupled to the iteration boundary.

How can this be applied to Scrum? Consider retrospectives. If you are just starting with Scrum, you probably have an iteration length of 1 month (or four weeks). From that it follows that you will have a retrospective once per month. If you eventually end up with an iteration length of 1 week, then it follows that you will have a retrospective every week. But this actually seems like the wrong way to set the cadence of retrospectives. Wouldn’t it be better to have the cadence of retrospectives meet the need for them? If it eventually makes sense to do a retrospective every week, doesn’t it make sense to get the benefit of them on a weekly basis when you are just starting Scrum?

Go deeper: “Applying the Decoupling Principle to Scrum”

Do One Thing and Do it Well
Kanban has something called “work in progress limits.” Limiting work in progress is a concept that comes from Lean. The basic idea is that the less you have in progress the simpler everything becomes and the more likely you are to actually get work finished. If you believe in Scrum then by extension you pretty much also believe in work in progress limits. An iteration is in effect a work in progress limit. If you have an iteration length of two weeks and a velocity of 60 story points then you are saying that you will take on no more than 60 points of work per two week period. Kanban takes this further and says that in general the smaller you can make your work in progress limits, the better.

One of the most straightforward ways to create a WIP limit is to do a headcount of the # of developers on the team. If you have three developers, that gives a WIP limit of 3. That means that you never have more than 3 stories in progress at any given time. If you have been doing Scrum for a while, are good at breaking work down into small stories, and are doing One Piece Flow well, then what you are doing now is very similar to having WIP limits.

Take it to The Limit
Once you are comfortable with having a WIP limit, the next step is to remove iterations all together. Hopefully, this doesn’t seem quite so outlandish any more. After all, you are just exchanging one constraint for another. There is a hidden danger here though, which is one of the reasons that I strongly recommend building up the discipline of One Piece Flow first.

Let’s say you have an iteration length of two weeks. That also puts a two-week limit on your stories. In Kanban, if you only have a WIP limit on the number of stories you can take on, you could end up doing things like working on a task for months and months without finishing it. Luckily, there is a simple cure for this problem: impose either a maximum elapsed time rule, a per-story story point limit, or both.

One more thing to consider here is that if you had an iteration length of 4 weeks, there's a good chance that most of the stories were no more than a week from start to finish, and probably on the order of days. You may want to consider having a per-story elapsed time limit of a week, possibly 2 weeks to start.

Kanban in Action
Now that we’ve taken a look at the various aspects of Kanban, let’s see what this would look like in practice. First of all, we’ve started with Scrum so we still have a backlog, a Scrum Master, a Product Owner, user stories, and the like. As soon as a story is completed, the team takes another story from the “todo” list and starts working on it. That then triggers the product owner to add another story from the backlog to the todo list. Stories move from backlog, to todo, to coded, to tested, to done and then on to "shipped" on a regular basis. If you have been doing Scrum well, this is essentially what you have been doing anyway, but now you no longer have the awkward iteration boundaries. Instead, you have a continual flow of stories with no need to stop artificially.

Yes You Kanban
In summary, a good approach to take when adopting some or all of Kanban is:

Do Scrum for 6-9 months
Get good at breaking work into small user stories
Get good at doing One Piece Flow
Apply the decoupling principle to your Scrum implementation
Impose work in progress limits
Drop iterations and impose a story point limit

If you liked this post, you may also be interested in my new book “Do it Yourself Agile” which is freely downloadable.

Interesting links:
Limited Work in Progress Society (Kanban resources)
Kanban vs. Scrum (Friendly comparison)
One day in Kanban Land (Kanban Cartoon!)
Scrumban (Fully mixed!)

If you are in the process of moving to Kanban from Scrum or adopting some of Kanban in your Scrum implementation, let us know how it is going! What worked well for you and what are some of your lessons learned?

One Piece Flow -- Transitioning from Scrum to Kanban Part II

2009-09-13T00:15:00.004-04:00

If you have been doing Scrum for a while, and doing it well, it is 99% certain that you are frequently practicing something called “one piece flow”. One piece flow is a concept from Lean and is the key to succeeding with Scrum. Doing it well also happens to be one of the requirements for successfully transitioning from Scrum to Kanban.

There are four main concepts in “one piece flow.” Each story is done as if it was the only thing in the release, each aspect of developing a user story happens in rapid succession, there is as much done in parallel as possible, and each team member focuses on a single user story at a time. The result of one piece flow is that the time between when the team first starts working on a user story and when they can ship it fully developed, tested, and documented is very short. The timeframe is generally on the order of a week at most and usually days.

In one piece flow, when development starts on a story, QA should be creating test cases for that story (story testing) at the same time. When development is done, QA should then automate the test cases and make sure they all pass. At the same time, the user documentation is written. One last step that must be completed prior to considering the story done is that all of the artifacts connected to that story such as source code changes, documentation updates, new test cases, etc must be integrated into the source code mainline as part of continuous integration.

In the following diagram, there are 20 stories done over the course of three iterations. There are three developers on the team and each takes on one story at a time. In the diagram, the whole team is doing one piece flow well and you can see how the specifying (S), designing (D), coding (C), writing of tests (W), integrating (I), and testing (T) is evenly spread out but also clumped into stories.

So, from the perspective of a story, it is started, and completely finished in a short period of time. QA should not be waiting until all stories are done. Conversely, developers should not have lots of stories in progress that all finish together which keeps QA from getting involved.

You will know that you are succeeding at one piece flow when stories are being started and then are completely ready to go on an individual basis within days with all documentation written and all unit and story tests written, automated, and passing.

Once every QA person is actively engaged in a story that is part of an iteration, you are “in the flow.” Everybody is now flowing smoothly from story to story. Unfortunately, Scrum has a bad habit of disrupting one piece flow on a regular basis. As soon as one of the developers finishes their work for the last story assigned to them for the iteration you are no longer “in the flow.”

For more about how Scrum disrupts one piece flow, read about Bumping Into Scrum’s Iteration Boundaries.

Next: Scrum and Kanban -- Like Chocolate and Peanut Butter

Transitioning From Scrum to Kanban, Part 1 of 3

2009-09-12T21:39:00.012-04:00

Bumping Into Scrum’s Boundaries
Scrum is the most popular Agile process in use today. But if you have been doing Scrum for a while, you know (or will soon experience) that it has some structural problems. Many of Scrum’s activities are tightly coupled to the cadence of the iterations but have a natural tendency towards their own cadences. In order for a story to be done-done-done, all of the work to make sure that it is done-done-done must be done within the iteration. A prerequisite of doing that work is that the code must be complete. That causes problems.

At the beginning of an iteration, there are no stories for testers to be testing until the developers get the first stories finished. At the end of an iteration, while the story validation work is being done on the last stories, the developers (and some of the testers) have nothing to do. A common response to this is that “there is always something to do.” Obviously there is always something to do, but that’s not the point. A key point of Agile is to focus on the highest value work. By definition, the highest value work is whatever is next in the backlog and not anything else. So, if you are doing something other than work required to move that iteration’s stories from todo to done you are working on the wrong things. One approach is to have the developers start on the next items in the backlog, but now you are moving outside the framework of Scrum.

Another problem is that the last stories rarely all transition neatly to done right at the end of the iteration. You may have some stories in progress at the end of the iteration which were hoped to be finished by the end of the iteration. For these folks, there is pressure to fit the work into the time remaining. Other folks may finish up early and not have any stories left to work on. While it is true that the team should be working and thinking as a team and helping each other out, it isn’t always possible to have two people work on the same story.

Just What the Doctor Ordered
There’s a commonly prescribed remedy for all of this: change the iteration length. Sometimes the recommendation is to make it longer, sometimes the recommendation is to make it shorter. In my experience, neither of these recommendations works very well. The reason is that the size of the stories that you work on tends to vary over time. The smaller, the better of course, but some stories are just naturally larger than others. No matter what you do with the iteration length, all you are really doing is changing the size of the problem, you aren’t really removing it.

The Source of the Friction
There are two simple reasons for this friction. It isn’t any easier to predict the future using Scrum. Software development is a creative endeavor. It is unpredictable. There is no way to get stories to all end smoothly at the end of the iteration, regardless of the iteration length or how often it is varied. The structure of the work itself just makes it worse.

Within each story there are two kinds of work: work that is mostly done during the first half of the story and work that is mostly done at the end of the story. The end work depends on the beginning work. As a result, it is impossible to “get the air out of the pipes.” It will always be there. This is illustrated in the following diagram.

The diagram shows two iterations of work split into three "swim lanes." I've artificially put one developer and one QA person into each swim lane to simplify the explanation. The numbers in each swim lane correspond to individual stories, the green areas represent developers and the yellow areas represent testers. The structure of the work means that there will always be areas that are either empty or filled with “other work.” You may have heard that it is good to have slack time, and I agree. However, I believe it is better for slack time to be under the control of the people in the system, not as the unpredictable result of a side-effect inherent in the process.

Long Live Scrum!
So where am I headed with this? Am I anti-Scrum? Am I about to describe a way to fix these problems with Scrum? Neither. I don’t believe these problems can be solved within Scrum. I do think that Scrum can be improved, see “Applying the Decoupling Principle to Scrum,” but I see the real solution as transitioning to Kanban. However, I also believe that it is better to do Scrum for a year or two first in order to build up the discipline of doing One Piece Flow that is necessary for succeeding with Kanban.

Next: One Piece Flow -- Transitioning from Scrum to Kanban Part II

Agile, Pizza, and Beer! Boston, Sept 24th, 5-7pm

2009-09-10T11:44:00.001-04:00

If you are curious about Agile, you may enjoy my presentation "Is Agile Any Better?"

Download "Do It Yourself Agile" For Free!

2009-09-08T15:38:00.009-04:00

In my copious spare time I've been working on a book about Agile development. I had originally intended to publish it as an offline paper book, but over time I realized that I was generating a bunch of content that was just getting dusty on the shelf and wasn't providing any value or getting the benefit of rapid feedback. So now I've switched gears and am moving to publishing entirely on the web.

Translating a book to the web takes time, so I’ve also decided to make the book available via pdf in the meantime. As the material has made its way to the web, it has often been rewritten. Some of those changes have made it into the book, and some haven’t.

While I believe that an Agile coach, whether recruited internally or externally, is still the best and fastest path to Agile success, not everyone can do that. "Do it Yourself Agile" is intended to be a resource you can lean on as you transition to Agile on your own.

In any case, let me know what you think. I look forward producing new versions incorporating your feedback.

[Sep 29, '09: Second version now available, link is the same.]
[Sep 14, '09: Based on the feedback so far I will be adding recent blog entries and putting out the first update to the book in a couple of weeks.]

"Do it Yourself Agile" (pdf)

For $65/pp, Send Your Whole Team to Agile Training in Boston Oct 5th

2009-09-08T13:09:00.003-04:00

This is truly a killer deal. Alan Shalloway will be in town on October 5th doing a special 1-day Agile event for the Agile Bazaar titled "Team Technical Agility". At $65pp including lunch, I have a hunch this will fill up fast. Why not sign up your whole team? Even if you aren't in the Boston area, the airfare and hotel will probably still add up to less than you would pay for any other 1 day Agile course.

Applying the Decoupling Principle to Scrum

2009-08-31T20:27:00.028-04:00

While working on a post about Kanban, I realized that there is a general software engineering principle at work in Kanban that can be applied to Scrum without needing to mention Kanban at all. The principle is: decoupling. That is, separating two or more things which are currently coupled together but don't need to be.

In Scrum, there are many activities which are often tightly coupled to the iteration cadence: iteration planning, having a shippable increment of work, the size of the largest story, iteration reviews, retrospectives, and releases. If you have been practicing Scrum for a while, it is likely that you have already started to decouple some activities from the iterations.

Decoupling Iteration Meetings and Story Point Estimation

Many Scrum teams have a weekly meeting to estimate story points for stories that have made it near the top of the backlog and don't yet have story point estimates. This decouples story point estimation from both the iterations and the iteration planning meeting itself. How far down into the backlog you go depends on what goal you are trying to achieve. If you just want to simplify iteration planning, you only need enough stories to cover one iteration worth of stories and perhaps a bit more just in case some stories aren't chosen for that iteration.

Decoupling Retrospectives

Let's see how decoupling might be applied to more practices. When just starting with Agile, it is much harder to use 1 week iterations than 4 week iterations. If you are doing 4 week iterations, you would naturally do a retrospective once every four weeks. If you move to 2 week iterations, you would naturally do a retrospective every two weeks.

But why not do retrospectives every week or at least every two weeks, regardless of whether you are doing 4 week iterations or two week iterations? Shouldn't the cadence of retrospectives match the cadence of their usefulness? In my experience, there is always something worth talking about every week of any project. There is always something to reflect on and improve. In any case, instead of just scheduling a retrospective at the same cadence as your iterations, schedule the retrospectives at a cadence that makes sense for your needs.

Decoupling Iteration Reviews

Another practice to consider decoupling from your iteration cadence is iteration reviews. Let's say you have 1 week iteration reviews, but it is logistically difficult to schedule iteration reviews involving multiple customers more often than once per month. Perhaps you normally have both internal and external stakeholders at the reviews. One solution would be to continue to have weekly reviews with just the internal stakeholders that are interested in being there every week, and have monthly reviews for external stakeholders. That may actually require a bit more work on your part, but if the value is there, why not consider it?

Simplification and Delegation

Another benefit of decoupling is that by breaking things apart it makes it simpler to tackle or delegate process improvement efforts because you can concentrate on smaller parts which are running at their own cadence. In the example of the iteration reviews, by separating out the external stakeholder meeting, you can now focus on ways to simplify or delegate that part of the process. Perhaps the only problem is getting a web meeting solution in place. Now you can solve that problem without impacting the internal stakeholders.

Consider how the decoupling principle will work for you. I would be interested to hear what you've already decoupled and your thoughts on applying the decoupling principle to Scrum. If you want to read about how to take decoupling to the next level, click on the next post below.

Next: Transitioning from Scrum to Kanban

Back to Work After Agile 2009

2009-08-31T18:51:00.003-04:00

It is my first day at work after Agile 2009, and I must say I am both exhausted and re-invigorated at the same time. For me, one of the most exciting parts of Agile 2009 was that there were so many people buzzing about Kanban, a concept that I believe should be getting much more exposure and use than it currently is. By the way, if you are a fan of Kanban, check out the "Limited Work in Progress Society" web site to find all sorts of Kanban resources. Another good resource on Kanban is Henrik Kniberg's comparison of "Kanban vs Scrum". I'll write more on my thoughts about Kanban soon.

Software Development Bloggers Unite!

2009-08-29T10:25:00.003-04:00

Do you love reading blog entries about software development? Social news sites are a great way to find posts that you are interested in. The more people know about these sites, the more people will use them and submit great posts to them. Please help to spread the word about these sites. If you know of some that aren't on the following list, let us know!!

Here are the ones I regularly visit:

Agile sub-reddit

Software development sub-reddit

Joel on Software sub-reddit (not just Joel)

Dzone Agile tag

Dzone methodology tag

What else is good??

Why Bother Going to Agile 2009 or Agile 2010??

2009-08-28T20:20:00.000-04:00

Agile 2009 was a blast again this year. I learned a lot and met lots of incredibly interesting people. While talking to a friend about MVP for GUIs I realized that somebody that had just presented on the topic was sitting at a nearby table and we asked him about it and got a great answer.

I believe there are lots of folks out there that would get a lot out of some of the conferences that are out there, like Agile 2009, if only they went once and experienced it. I've decided to dedicate some portion of my time to getting the word out on that. As a first step I created an impromptu video which includes many great answers to the question "why go to a conference vs just surfing the web and reading books?" Next time I'll be better prepared and do an even better version.

The video was taken while the conference was in full swing, but it was the last day and some folks were tired of going to sessions and were just hanging out in the open area. Esther Derby was doing an "Open Jam" session, Alistair Cockburn was just hangin' out, Dave Anderson was having a spirited discussion on something interesting (I know not what), and others were networking, just catching up on e-mail or waiting for the next session that they wanted to attend.

I'll post more impressions of the conference soon.

Continuous Integration Sessions at Agile 2009

2009-08-12T13:49:00.002-04:00

If you are interested in Continuous Integration, and you are going to Agile 2009, please consider attending one of my CI sessions:

Removing Integration Delays with Collocated Whole Teams and Multi-stage CI
As individuals we work in transient isolation to reduce the impact of work in progress on each other. Organizations isolate WIP by using only official versions of 3pty sources and by producing official releases for customers.

Multi-stage continuous integration (MSCI) scales CI to large distributed environments by isolating work in progress at the team level. Changes move from individual to team to mainline as fast as CI allows, but stop on failure.

MSCI is particularly important in a distributed environment where fixes to problems exposed by CI can be delayed by a full day. [Full Details]

Agile Source Code Management using Stories, Agile Workflow, and CI
You create iterations from a backlog of user stories managed via a taskboard with a simple “workflow” from “todo” to “done.” You use Continuous Integration. But in your source control system you’ve just got files and branches. You could create a branch for every story, but that’s a lot of branches to manage! How can you ask the source control system which versions/files correspond to the stories that are done in order to build the “done” version and do exploratory testing? This session will show how to manage changes using stories and how to use branches to represent your workflow. [Full Details]

Guilty Confessions of a Software Developer

2009-07-12T17:17:00.036-04:00

I tried to look like I was still mulling over the question when Mr. J continued his interrogation, "The description of this feature is incomplete. What should happen when the -t option is combined with -V and there are no other options given?" I was sweating and nervous, my chest felt tight and I knew I was right on the edge of panic.

I couldn't remember the answer to Mr J's question. I'm not sure I ever knew it and the customer that had asked for this new feature had gone out of business 2 months after we implemented it, which was 3 months ago. I was helpless and trapped. The harsh light and the look in his eyes wasn't helping. He knew that I didn't know. I wouldn't be able to hold out for much longer, it felt like the room was starting to spin.

Finally I couldn't take it any more. I cracked under the strain and confessed what he already suspected. "I don't know. I wrote that code so long ago, I just can't remember. I'm sorry." Mr. J shook his head and walked out of my office saying, "I'll figure it out."

Ok, so this is a bit of a dramatization loosely based on a real event, but being in the situation where you are asking about or needing information about something that happened months ago in the development process is fairly typical in a traditional development project.

Perhaps you are the developer in this situation or the person trying to write the tests for the functionality. Or perhaps you were trying to write the documentation, do the demo for the user, market it, sell it, or... use it? As an industry, we've become numb to this problem. Sure, we blame specific situations or individuals and we often try to take corrective action... but it seems to keep happening again and again.

As an industry, we try to write better requirements, but it doesn't seem to help. The tough questions keep coming. We try to take a bit more time on the design, but things just get worse. The questions get more complicated. We try to do more testing and hire "better" people, but we keep running into situations where we just don't have the answer, there's no supporting documentation, and we promise we'll try harder next time.

Why does this keep happening? What can be done to prevent it? It turns out that there is a simple explanation as well as a practical solution.

Next: Traditional Development == The Game of Telephone?

Leveraging Human Strengths in the Development Process

2009-07-12T15:14:00.033-04:00

In the previous post we saw two ways that traditional development is like the game of telephone. First, communication suffers from translation errors at each stage of the development cycle. Second, the frequent attempts to detect, prevent, and recover from translation errors require people to remember details about work they did weeks or months in the past. This is unfortunate because people are much better at remembering something they did within the past couple of days than they are at remembering something they did weeks or months ago.

One of the advantages of Agile development is that it leverages human strengths and minimizes the need for humans to do things that humans do poorly. In the case of communicating important concepts throughout the development cycle, Agile relies on four tools: user stories, one piece flow, conversations, and reduced documentation.

A user story is a simple statement in plain English of a value proposition that the end user of the software is interested in. For instance, "As a user, I want to purchase an item I am looking at with a single click." The main value of user stories for communication is that they are used by everybody involved in the development of that story, from user to tester. While the story may be translated into a design document or test cases, the original intent of the user is always available as a double-check via the user stories.

An important part of increasing communication effectiveness is a concept called "one piece flow" which comes from Lean. The idea of one piece flow is that each aspect of developing a user story happens in rapid succession, and that each team member focuses on a single user story at a time. The result of one piece flow is that the time between when the team first commits to doing a user story and they can ship it fully developed, tested, and documented is very short. The timeframe is generally on the order of a week at most and usually days.

Because user stories go from start to finish in such a short period of time, the bulk of the communication that occurs on behalf of a user story is done via conversations. Of course, Agile teams do document their work, but it is for the purpose of creating an audit trail, not for the purpose of short-term communication between team members. Conversations are a much better way to communicate technical concepts than documentation.

Conversations between Agile teams and their customers

From the perspective of a customer, the time between when they provide detailed information about a feature and when they can see a demo of the result is often as short as a month. Also, the scope of functionality is going to be much less. The reduction in timeframe and scope means that it is much more likely that a customer will remember exactly why they wanted something and that they will be much more able to provide useful feedback on the result.

That doesn't mean that all customers have to be involved every month, only that there is an opportunity to go full circle with customers over a much shorter period of time than with traditional development. Consequently, miscommunication can be caught earlier and corrected faster.

Conversations in an Agile team

From the perspective of individual team members, interaction is now focused on a specific user story at any given time and the timeframe of its development is on the order of days rather than months. Work can be initiated, rapidly completed, and then confidently considered done instead of having an ever growing list of work in progress. With Agile development, the amount of work in progress is very small and is constantly changing. There is much less of a need to rely on piles of documentation or on the long term memory of oneself or others.

Are you an Agile Do It Yourselfer? Check out the web-only book

"Do It Yourself Agile" .

Traditional Development == The Game of Telephone?

2009-07-12T12:05:00.060-04:00

In theory, traditional development is a good idea. The work of development is broken up into stages in order to reduce risk and increase efficiency. But in reality a significant proportion of projects are shelved, cancelled, or delivered with low quality. In part, traditional develelopment has problems for the same reasons that it is fun to play the game of telephone.

I remember fondly playing a game called telephone in elementary school. In the game of telephone, a message is whispered from person to person. The fun part is that the last person says what he heard to the whole group and everybody gets a laugh out of how a phrase such as "Send reinforcements, we're going to advance" gets inadvertently translated into "Send three and fourpence, we're going to a dance". This is very similar to what happens in traditional software development projects.

Figure: The Traditional Development Telephone Game

The figure above represents a typical traditional development process and some typical times between handoffs. At each stage, instead of whispering the information along, the participants create and use a series of documents, each of which serves a different purpose. As the information moves from stage to stage it needs to be translated into an appropriate form for each stage of development. The documentation also serves as a record of information that was discovered along the way and the decisions and justification for decisions that were made along the way.

The product manager talks to customers and those conversations are translated into Marketing Requirements Documents (MRDs) which are translated into engineering requirements which are translated into specifications which are translated into designs which are eventually translated into code. The end result is then given to the user who often says something pithy like "that's not what I wanted."

At each stage of development there is a risk that the intent from the previous stage is mistranslated. Another way to think of the mistranslation is data corruption. The larger the amount of work that is flowing from stage to stage, and the longer it takes for work to pass from stage to stage, the more likely it is for information to become "corrupted" and the greater the extent of the corruption. Of course, people don't just blindly translate information without interacting with people from earlier stages, they ask for clarification when they think they need it.

When clarification is needed, such as when somebody is writing a test and has a question about how a feature is supposed to work, it requires a conversation with the author of the documention. There is no guarantee that the author is available when needed, knows the answer themselves, or can find the answer in the supporting documentation that they themselves used. That then requires another conversation with the architect, and then the architect with the product manager and then the product manager with customers. The customers may have given their feedback based on ideas they had during the discussion with the product manager but have long since forgotten. The further back in the chain you go, the harder it will be for people to remember the details required to answer clarifying questions.

People aren't very good at remembering the exact details of the work that they did weeks or months in the past. That’s one of the main reasons for creating the documents in the first place. When people are routinely put in the position of having to do something they are not good at, it leads to stress, lower morale, and mistakes. [See "Guilty Confessions of a Software Developer"]

Ok, so now we have a good idea of how customer intent can get lost in translation and how the people developing the software can get stressed out. While that's all very interesting, it isn't very useful unless we can figure out how to act on this information.

Next: Leveraging Human Strengths in the Development Process

Do It Yourself Agile

2009-07-07T11:20:00.002-04:00

Originally, my blog was focused on Software Configuration Management. Then I described in a blog post how I realized that I'm not really an SCM person, I'm a software development process person that had been focused on SCM. So, I changed the name of the blog to "Agile Development Thoughts." Today I realized that this blog has changed its focus from random thoughts about Agile Development to a blog about Agile for the Agile DIY'er. So as of today, this blog is now "Do It Yourself Agile."

I still believe that the quickest route to Agile success is to get people involved who have done Agile before. But, that option is not always available and even when it is, it may not be possible to retain an Agile coach for the multiple years that it will take to reach your full Agile potential.

So, the idea of this site is to give the Agile DIY'er a resource to lean on. A good place to start is the web-only book "Do It Yourself Agile." This is an evolving book which is expanding and changing every week in response to questions and discussion from readers like you. If there is a topic that you don't see covered or that you would like to explore more fully, let me know! Just post a comment wherever you see fit or use the Q&A page. I'll answer your question or create a new blog post on the topic as soon as I can.

Check out the web-only book "Do it Yourself Agile" .

Agile Exposes Scaling Problems

2009-07-06T12:42:00.002-04:00

One of the first questions that seems to come up in any discussion of Agile is "How Well Does Agile Scale?" Sometimes this is asked explicitly, but more often there is an underlying assumption that Agile does not or can not scale very well. When I was first exposed to Agile, my first impression was that Agile didn’t scale beyond a small team, say 1-12 people.

I used to try to tackle the question head on, but then I realized that there's something else going on here. Let’s take a step back. What do we currently assume about traditional development? I think we come to the discussion thinking that just because there are teams of 100 engineers, 500 engineers, and even 2,000 engineers doing traditional development, that traditional development is a proven quantity when it comes to scaling. Let’s first ask the question: “How well does traditional development scale?”

To answer that question we first have to define "scaling." I think a good working definition is that as you add new resources, there is a linear increase in effective output with little or no decrease in efficiency. For software, output and efficiency translate to new functionality and productivity. That would mean that if you have a 50 person engineering team and you add 50 more people you get twice the output. But when was the last time you saw that? Have you ever seen it? In my experience, after talking to hundreds of development organizations doing traditional development, productivity falls and thus output does not increase linearly with additional resources. In many cases, output actually decreases as more resources are added.

What do you actually do to "scale" your development organization? Do you have meticulously updated Gannt charts and estimates? Do you schedule lots of meetings? Do you spend a lot of time making sure that requirements and designs are just right? Do you reserve 30% of the development schedule for testing (and fixing)?

In my experience, after talking to hundreds of development organizations, the answer to the question “How well does traditional development scale” is “not very well.” We've just suffered along with this problem, in large part because while the pain was there, the root causes were difficult to trace, let alone address.

The main point here is that if you are in the process of rolling out Agile to a large organization, don't be discouraged when you run into scaling problems. The problems were always there, they just weren't as obvious. Now, instead of wondering why things aren't coming together as expected at the end of the project, you may find out right away that your organization doesn't really have a good way to coordinate API changes for APIs that are used by multiple teams. As Agile exposes problems like this, you can take steps to solve the problems and thus create a more scaleable development organization that just happens to be doing Agile development.

Is Agile Really Any Better? Free Talk with Pizza and Beer

2009-06-02T20:49:00.002-04:00

If you will be in the Boston area on June 16th, I'll be talking about how and why Agile works, pitfalls to watch out for, and why it goes well with beer and pizza. There will also be an opportunity to learn how AccuRev helps in an Agile environment. You can read more details here.

Ken Schwaber to Speak on "Flaccid Scrum"

2009-05-23T01:28:00.001-04:00

Come hear Ken Schwaber's talk on "Flaccid Scrum" at the Agile Bazaar in Burlington, MA on June 18th. Visit our web site for more details.

How Continuous Integration Promotes Healthy Agile Practices

2009-05-14T17:26:00.002-04:00

Come learn "How Continuous Integration Promotes Healthy Agile Practices" in a webinar I'm doing for Version One at noon EDT on June 17th . See you there!

Abstract: Healthy, successful Agile teams rely on a robust Continuous Integration implementation as the hub of their Agile infrastructure. Many of the Agile practices depend on Continuous Integration (CI) in order to provide their full value. For instance, unit tests are the most valuable when all unit tests are run automatically on a frequent basis. Scaling Agile to large and/or distributed teams also depends on CI. Conversely, CI can be used as a leading indicator of the health of your Agile teams. This session will cover best practices for transitioning to Continuous Integration as well as show the strong link between CI and one-piece-flow, unit tests, short iterations, whole teams, refactoring, user stories, shared code ownership, distributed Agile, and large scale Agile.

Product Managers and Product Owners - Newton MA, Thu Feb 26th, 6pm

2009-02-22T09:38:00.002-05:00

Rich Mironov of Enthiosys will be speaking on "Product Managers and Product Owners: Which Do We Need, and What Do They Do". Hosted by the Agile Bazaar.

For more details, please check out the full meeting announcement.

Poll - Short Iterations, The Benefits

2009-02-10T12:51:00.006-05:00

Short iterations are the practice of breaking up a release into 1-4 week development cycles that each produce a shippable increment of work. Some of the benefits of short iterations include working software that you can show to people in order to get their feedback and having nothing in progress at the end of the iteration so you can change your plans to take into account market changes, feedback from customers, or information you discovered while implementing that iteration. Also, all of the tests for the functionality implemented during the iteration have been written and run and you have a well known starting point for the next iteration.

There are some assumptions hiding in this practice. It assumes that you believe that incremental design is possible and that you believe that any refactoring you have to do in subsequent iterations is worth the benefits as described above.

So, if you were starting a company with your own hard-earned cash, would you use short iterations or wouldn't you?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Whole Teams, The Benefits

Poll - Product Backlog, The Benefits

2009-02-10T12:46:00.005-05:00

A product backlog is a list of high-level product requirements (or user stories), prioritized by business value (typically determined by a product manager or product owner). It contains entries for any and all work that is performed, including bug fixes. As new requirements are gathered or submitted, they are inserted into the backlog at the position that best represents their business value with respect to requirements that have already been entered. As requirements are met, they are removed from the product backlog.

A backlog gives a clear context to discussions of priority. Anybody can go and look at the backlog to see the position of something they have a stake in. If it is lower than they think it should be, they can examine the backlog to see what is higher in order to help frame a discussion around either moving it higher or moving other things lower.

So, if you were starting a software company with your own hard-earned cash, would you use a product backlog or wouldn't you?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Your Preferred Methodology

Poll - User Stories, The Benefits

2009-02-10T12:37:00.004-05:00

User stories are a description of what is needed from the user’s perspective. User stories help to separate business value from implementation and focus all parties on the desired outcome.

User stories are different than requirements. When using requirements, it is likely that the developer implementing the requirement will be presented with an implementation task or a design document and be constrained to implementing as specified or as designed. A user story removes invisible constraints by focusing on the outcome desired by the user. The developer doing the work will see the user story, will be able to better understand what the user needs, and will be able to participate in or even own the specification and design of that story. User stories provide engineers more freedom to utilize their creativity and ability to innovate without the risk of implementing something that the user doesn’t want.

So, if you were starting a software company with your own hard-earned cash, would you use user stories or not?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Product Backlog, The Benefits

Poll - Collocation, The Benefits

2009-02-10T12:34:00.004-05:00

Collocation is simply having everybody on a whole team in close proximity to each other. This compounds the coordination benefit of whole teams. This is not related to using or not using outsourcing. Whether you are outsourcing or not, collocation only refers to whether your whole teams are sitting near each other.

So, if you were starting your own software company with your own hard-earned cash, would you use collocation or wouldn't you?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - User Stories, The Benefits

Poll - Whole Teams, The Benefits

2009-02-10T12:21:00.002-05:00

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Collocation, The Benefits

Poll - Unit Tests, The Benefits

2009-02-10T12:12:00.005-05:00

Unit tests are simply tests that exercise small amounts of isolated functionality. That is, if you have a function that adds two numbers, instead of depending on running a user function that eventually calls the function, exercise the function directly. This often requires the use of mock objects that pretend to be things that the function needs in order to test the function in isolation from other functions that it depends on. Unit testing and refactoring are often done hand in hand.

The cost of unit tests is in writing the tests themselves and refactoring code as new functionality is introduced to keep the unit tests testing at the right level. The benefit is that you can easily test changes quickly to find simple problems before doing more thorough and slower testing. It also provides a good safety net for refactoring, gets developers more involved in testing, and usually improves the design of the software.

So, if you were starting a new software company with your own hard-earned cash, would you use unit tests or wouldn't you?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Stand-up Meetings, The Benefits

Poll - Incremental Design, Possible?

2009-02-10T12:08:00.004-05:00

Incremental design is the practice of designing as you implement. You may do some upfront work to sketch out the basic ideas and groundwork, but don't have a fully fleshed out design up front. The basic idea is that you only implement what you absolutely need to satisfy the user-centered work that you have committed to for the current milestone/iteration. That way, you don't spend time designing for work that you never do.

If you aren't doing incremental design, don’t you end up having to make changes to your software during the next release to take into account all of the feedback and requests for new features that you get from customers? Don’t you have to rework the software (including the design) anyway? Aren't you already doing incremental design on a macro scale anyway?

One of the points of incremental design is that design changes are inevitable and that you are better off writing your code in ways that make it more amenable to design changes. For instance, unit tests encourage the creation of mock objects and the use of mock objects encourage the creation of abstraction layers which as we all know are a good thing.

And let’s not forget that dirty little “major overhaul” secret. You show me some software developed without incremental design that hasn’t had a “major overhaul” and I’ll buy you dinner. So, if you agree that you are going to have to make changes to your design anyway, sometimes major changes, why wouldn’t you want to do incremental design?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Short Iterations, The Benefits

Poll - Your Preferred Software Development Methodology

2009-02-10T11:21:00.005-05:00

Last question. With the exception of change reviews, all of the practices listed are the core of Agile development. Sure, there's that whole philosophy and manifesto stuff. But IMHO, if you are doing short iterations (for real) and many or most of the other practices, you are doing Agile development. No 3x5 cards, pair programming, certification, or manifesto required. Just don't like the idea of a new word or "joining a movement?" No problem, improving your dev process is the goal, not "doing Agile right" or anything else for that matter.

For any of the practices that you are not currently doing but that you believe have a significant benefit, think about what the combined effect would be if you implemented all of them. It may take time to achieve, but it will definitely be worth the investment.

So, you are starting a new software company with your own hard-earned cash. What software development methodology will you use?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Suggest additional practices to add to survey

Poll - Retrospectives, The Benefits

2009-02-10T11:17:00.003-05:00

A retrospective provides a forum for the team to talk on a regular basis about what worked and what didn’t work as a team, to discuss ideas for improvement and to work out problems. Instead of a manager having to unearth all problems and come up with all solutions, a retrospective accomplishes much of this, providing the manager with more bandwidth to deal with sensitive issues and to focus on people management. Retrospectives are particularly effective when combined with short iterations because problems have less time to fester and there are more opportunities to take corrective action.

So, if you were starting a software company with your own money, would you use retrospectives or wouldn't you?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Change Reviews, The Benefits

Poll- Milestone/Iteration Reviews, The Benefits

2009-02-10T11:15:00.004-05:00

Milestone, or iteration reviews, are done at the end of each milestone or iteration. It is a demonstration to key stakeholders including one or more customers that you have completed the work for that milestone. It is also an opportunity for the stakeholders to provide feedback on the work. Reviews provide a test of the functionality, show progress towards goals, and build confidence in the work. The demonstration is a visible, trustworthy, and confidence building way of showing progress. If you can’t demo the work, how do you know how much real progress you are making?

So, if you were starting a software company with your own money, would you do milestone reviews or wouldn't you?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Retrospectives, The Benefits

Poll - Change Reviews aka Code Reviews, The Benefits

2009-02-10T11:10:00.005-05:00

A change review, aka a code review, is a review of the changes (code or otherwise) that were done in order to fix a bug or produce an enhancement. This can be as simple as having an engineer other than the author review the changes and provide comments to the author to something as sophisticated as using a specific code review methodology and/or a code review tool such as Smart Bear. So, if you were starting a software company with your own money, would you do code reviews or wouldn't you?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Incremental Design: Possible?

Poll - Stand-up Meetings, The Benefits

2009-02-10T11:04:00.005-05:00

A stand-up meeting is a quick daily meeting of the whole team where you answer three questions: what did you do since the last meeting, what will you do until the next meeting, and what impediments do you have. Everybody gets a quick sense of how things are going and who needs help. The benefits are that problems don’t fester for long and it removes the need for most other meetings.

So, if you were starting a new software company with your own hard-earned cash, would you use stand-up meetings or wouldn't you?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Milestone/Iteration Reviews

Poll: Refactoring, the Benefits

2009-02-10T10:54:00.008-05:00

Refactoring is the practice of continuously improving the usability, maintainability, and adaptability of code without changing its behavior. That makes it much easier to add new and unanticipated functionality. Refactoring has the disadvantage that it takes extra effort and requires changing the code. Any change has the potential to reduce the maturity and stability of the product, especially if you don't have adequate testing in place.

If you were starting a software company with your own hard-earned cash, would you use refactoring or wouldn't you?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Unit Tests, The Benefits

Poll: Continuous Integration Benefits

2009-02-10T10:47:00.007-05:00

With Continuous Integration, all work from all teams is integrated into a single codeline as frequently as possible. Every check-in automatically triggers a build and usually a subsequent run of the test suite. This provides instant feedback about problems to all interested parties and helps to keep the code base free of build and test failures. It also reduces the integration headaches just prior to release.

There are two benefits of Continuous Integration. The first is that it forces you to break work down into small, manageable pieces. The second is that it spreads the integration work out over the entire development timeframe instead of just a short period at the end. Thus, you have more time to find and fix issues instead of a compressed high-stress window at the end of the release.

So, if you were starting a software company with your own money, would you use continuous integration or wouldn’t you?

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll - Refactoring, the Benefits

Suggestions For Practices to Add or Subtract

2009-02-10T10:37:00.005-05:00

Thanks for participating in the practices survey.

If there are other practices you would like me to add to this survey, or other general comments you'd like to make, please comment below. If you think there are practices that just don't belong on here, I'd be interested to hear which ones and why.

If You Were Starting a Software Company With Your Own Money, Would You... ?

2009-02-10T10:18:00.014-05:00

If you were bootstrapping a software company with your own hard-earned cash, what software development practices do you believe in enough that you would use them right off the bat?

You have to vote to see the results. There is a link below each poll which links to the next. There are 15 in all. The first is on your role, the rest are about which dev practices you value.

[Note to the jaded cynics: the polls really do only collect votes, your privacy is assured]

Click Here for Poll Survey Software
| Website Polls
| Email Marketing

| Innovation Management
View MicroPoll

Next: Poll: Continuous Integration Benefits

Agile Transformation Seminar

2009-02-05T20:09:00.002-05:00

If you are in the Boston area, consider going to the Agile Transformation Seminar on Thursday February 26th in Lexington. The seminar is a 1/2 day in-depth look at the business benefits of Agile and includes a case study on Avid's Agile transformation. My session will be about the effects of Agile on the needs of your tool infrastructure.

What I Believe About Agile

2009-02-05T13:18:00.004-05:00

Before I get too far into what I believe about Agile, let me give a little background. In 2005, I was a staunch Waterfallian. Stuff happened (blogged elsewhere) and then I turned into an Agile advocate.

There are two important things that I believe about Agile. First, while I do believe that being Agile is more than just a set of practices, I also believe that there are a set of practices which are so good that for me they define what I currently think of as Agile and believe everybody can do them and would benefit from them.

Here are the practices which currently define “being Agile” for me. These are roughly in order of benefit: short iterations (preferably 1-2 weeks), Kanban (as an alternative to short iterations), backlog, one piece flow, continuous integration, whole teams, collocated whole teams, test first/test early, use of a product owner, incremental design, refactoring, unit tests, multi-stage continuous integration, user stories, sustainable pace, task assignment and estimation by team members, stand-up meetings, burn-down and burn-up charts, iteration review, retrospective, and frequent releases. There are other good practices out there, but these are the ones that I believe will become mainstream.

Second, and most importantly:

I truly believe the benefits of Agile are significant enough to focus your resources to get to the point of saying this sentence.

This is based on the belief that the algorithm of Agile is just plain better than traditional development combined with direct observation of the significant benefits experienced by both external and internal teams.

To put it another way, if you are not yet investing in Agile, I believe it is the #1 investment you can make to increase your productivity, quality, revenues, customer satisfaction, profitability, and employee satisfaction. It is possible to invest in Agile but not get these benefits. If that is happening to you, I urge you to stick to your guns and look for help. Getting to Agile is worth the investment. It is worth redirecting resources, changing priorities, and doing what it takes to get there.

The web book “Zero to Hyper Agile in 90 Days or Less” is a good place to start.

You may also be interested in “The Top Ten Reasons I May be Wrong That Agile is Better” .

The Top Ten Reasons I May be Wrong That Agile is Better

2009-02-04T22:12:00.010-05:00

First of all, it is important to note that "Agile" is just a word and means different things to different people. There is a good chance that we have different definitions of what “Agile” means. If you want to know what I mean by Agile, please read “How Agile Works in a Nutshell”.

10. “Customers don’t want frequent releases.”

That’s often true, but it is not actually an impediment. While frequent releases are an important benefit, it takes time to get to that point, isn’t the right option for all situations, is just one of the benefits, and is not required to get the other benefits. Read more in a series of posts on this subject.

9. “The advocates are overzealous and arrogant.”

Yes, some of them are. But not all of them. Many people are overzealous and arrogant, that’s not unique to Agile. It is the concepts of Agile that make Agile worthwhile, not the advocates.

8. “It is all just a marketing ploy. You don’t really believe Agile is any better than traditional development and in fact it is just more of the same stuff in a shiny new package. You just want to hop on the buzzwagon.”

This one is hard to refute. The only way I can dispel that belief is if you get to know me better.

7. “I haven’t tried it, and I believe Agile Development is a good development method, but it is just another flavor. Whatever works for you.”

You may be of the opinion that traditional vs Agile is just a preference/philosophy/ideology thing, like chocolate vs strawberry, Aristotle vs Plato, Democrat vs Republican. But then in that case, why would anybody go to all of the effort of investing in Agile? Tech folks don't generally make massive switch-overs from one way of doing things to another just for preference or just to be 2% more productive, they generally only switch in order to get an order of magnitude improvement. So, if it is just a preference thing, there's not much to talk about (from my perspective).

6. “Yeah, I tried that Agile thing. What a disaster!”

There are any number of reasons for failure. I've seen plenty of cases of Agile failure. But in every case that I've seen "Agile failure" it was either because it wasn't really Agile that was being done, or there was a major impediment. An example of a major impediment would be that the attempt was initiated by executive decree followed by no investment in success and a goal of 100% Agile within an impossibly short amount of time. I realize that saying “it probably wasn’t really Agile” sounds pretty convenient. But, I have yet to hear anybody say "I feel like we were really doing it right, but it just didn't work." If that's something that you would say, I'd love to hear from you.

5. “Yeah, I know a group doing Agile and they are really successful. But they would succeed anyway because they’ve got all of the best people on that team.”

All I can say to that is that for me Agile is only worthwhile if it is something that has mainstream applicability. If it won’t work for the majority of teams, it isn’t interesting to me.

4. “I believe that you believe that it is better, but I can clearly see that it is not. I don’t know what your reasoning is, but it is clearly wrong.”

This is exactly what I was saying in 2005. Well, that works both ways and I’ve been on both sides of it. All I can say is that I believe it is worth your time to consider that if so many people that have devoted so much energy to the software development process believe that Agile is a good thing, then maybe there really is something worthwhile about Agile and it is worth your time to learn more about it.

3. “You don’t really understand how successful traditional development can be, and there are probably a bunch of techniques for success you are not aware of.”

My career for the past twenty years has been to contribute to the software process improvement efforts of countless companies. So it isn’t that. By the way, if you haven’t heard of the Niagra method (the best traditional development method), you should take a look.

2. “You have blind faith in Agile. You want it to be better and you will ignore any evidence to the contrary.”

Well, you'll have to take my word for it that I'm a very pragmatic person. If something isn't working, and it doesn't look like it is going to, I don't have a lot of patience to continue doing it just because I think it should work. I originally believed in Agile when I realized that it is algorithmically better than traditional development. But now I believe in Agile because I've experienced significant benefits from practicing it.

1. And the number one reason I may be wrong is the one I don't know about yet. If you've got a good one that I haven't covered, please let me know. I look forward to your thoughts and feedback.

Next: What I Believe About Agile

Agile Twittering

2009-01-31T18:29:00.002-05:00

In case you haven't already been bitten by the twitter virus (what else to call it?) the whole Agile world seems to be on twitter. Now, I'm there too! What's next? The hive mind? :-)

Getting Management and Engineers Out of Each Other's Hair

2009-01-20T16:49:00.006-05:00

This is the fifth in a series of posts on self-management that began with "The Use of Self-Managing Teams Outside the Context of Software Development."

Short Iterations Increase Management Leverage
One of the hardest jobs of a manager is discovering and resolving problems. If you don’t know about a problem, you can’t resolve it. If you discover a problem late, you can’t do much about the negative effects it has already had. If you don’t get feedback on the results of the actions you take for a long time, it is difficult if not impossible to judge the effectiveness of your actions. Short iterations, the practice of breaking up a release into 1-4 week development cycles that each produce a shippable increment of work, vastly increase the amount of feedback available to management for making good decisions. It also provides more opportunities to make adjustments and judge their effectiveness.

Burndown Charts are a Management Dashboard That Everybody Loves
A burndown chart is a useful tool for measuring the amount of progress, positive or negative, during an iteration which clearly illustrates trends which may indicate problems. It is simply a day-by-day chart of how much effort remains. At the beginning of an iteration, 100% of the work for that iteration remains. At the end of the iteration, there should be 0% remaining.

An important multiplier of the effectiveness of a burndown chart is its visibility. A clearly visible burndown chart makes it obvious to everybody how the project is progressing. Nobody wants to see a burndown chart that isn’t making progress, and nobody wants other people to see lack of progress for a significant period of time. The burndown chart helps to motivate and inspire. If your own work is going well, but the chart isn’t going down in general, you’ll be more engaged in helping to solve problems that the team is having overall. When used effectively, the burndown chart reduces the effort required for a manager to determine that there is a problem and also reduces the need for the manager to cajole the team into taking corrective action.

Retrospectives Emphasize That You Are All In This Together
A retrospective provides a forum for the team to talk on a regular basis about what worked and what didn’t work as a team, to discuss ideas for improvement and to work out problems. Instead of a manager having to unearth all problems and come up with all solutions, a retrospective accomplishes much of this, providing the manager with more bandwidth to deal with sensitive issues and to focus on people management. Retrospectives are particularly effective when combined with short iterations because problems have less time to fester and there are more opportunities to take corrective action.

User Stories Reduce Micro-Management
Agile prefers user stories over requirements. User stories are simply stories that describe what is needed from the user’s perspective. User stories help to separate business value from implementation and focus all parties on the desired outcome. The separation removes the invisible constraints which can appear when the engineer assigned to a task is given a specification which completely constrains the problem such that only one solution is available. By focusing on the outcome desired by the user, the engineer working on the task has more freedom to utilize their creativity and ability to innovate without the risk of implementing something that the user doesn’t want.

There are two reasons that user stories support self-managing teams. First, they reduce the temptation of technically minded managers from becoming an implementation bottleneck. Second, they encourage team members to innovate and take on more responsibility which increases their self-sufficiency. This is much easier to do within an Agile environment because there are more practices in place such as short iterations, burndown charts, and retrospectives which encourage learning while reducing the risk of making mistakes.

I hope you have enjoyed this series on how Agile practices reduce and redistribute the job of management and thereby increase your capacity for self-management. If you missed the beginning, the first posting is "The Use of Self-Management Outside the Context of Software Development" .

You may also enjoy reading the the online mini-version of the book that I am working on called "Zero to Agile in 90 Days or Less."

Three Cures for the Management Bottleneck

2009-01-20T13:19:00.006-05:00

This is the fourth post in a series on self-management that began with "The Use of Self-Managing Teams Outside the Context of Software Development"

Divide and Conquer
One of the ways that Agile, especially when using Scrum, reduces the management bottleneck and enables self-managing teams is by having clearly identified roles and responsibilities. Two very useful roles are those of Scrum Master and Product Owner. The Scrum Master role clearly frames the project management function. The Product Owner role clearly frames business management related to the product that a team produces. Both of these roles clearly delineate an area of responsibility that would normally be done by a traditional manager and makes it possible to delegate those responsibilities.

The Scrum Master
In Scrum, the Scrum Master is the person making sure that the (Scrum) process is working, looking for and removing impediments, and acting as a liaison between management and the team. Regardless of what you call the role of Scrum Master, the role itself does not require you to be doing Scrum.

The Product Owner
The role of Product Owner also comes from Scrum. As with a Scrum Master, the use of Scrum is not required in order to have a Product Owner. However, the effectiveness of the Product Owner is definitely leveraged by Scrum. The Product Owner is the person who is solely responsible for the content of a release. They are responsible for all decisions related to what changes are made to the product, including all defects and enhancements. Of course, the Product Owner is most effective if they gather information from a wide variety of sources and work closely with all stakeholders. A Product Owner that regularly makes decisions without consulting with stakeholders or acts on a whim won’t be a Product Owner for long.

The Product Owner Needs a Product Backlog
The role of Product Owner goes hand in hand with the use of a product backlog. A product backlog is simply a list of high-level product requirements (preferably in the form of user stories), prioritized by business value as determined by the product owner.

A backlog gives a clear context to discussions of priority. Anybody can go and look at the backlog to see the position of something they have a stake in. If it is lower than they think it should be, they can examine the backlog to see what is higher in order to help frame a discussion around either moving it higher or moving other things lower. Thus, many people can be leveraged to collectively groom the backlog. This distributes the work of grooming the backlog and helps to ensure that the team is focused on the highest value to the organization.

Reducing the Management Bottleneck
Even if the roles of manager, Scrum Master, and Product Owner were all handled by a single person, the use of these roles and their artifacts simplifies the job of management while at the same time amplifying its effectiveness. And of course, that makes it even easier to delegate these roles. The more that management is delegated, the less chance that management will become a bottleneck.

Next: Getting Management and Engineers Out of Each Other's Hair

No-Brainer Self-Management Practices in Agile Part 2

2009-01-19T14:48:00.003-05:00

This is a continuation of "No-Brainer Self-Management Practices in Agile Part 1"

Whole Teams Improve Coordination and Communication

A whole team is a small group of people (no more than 12) that work together towards a common purpose, primarily spend their time as part of the team, and as a team have all of the skills they need in order to be self-sufficient. These skill sets may include server side programmer, web designer, tester, technical writer, scrum master, product owner, etc. The whole team practice contributes to self-management by reducing the amount of coordination required in order to achieve a goal. For instance, instead of having to coordinate with a QA group in order to find somebody that can test a piece of software, having a QA person as part of the team means that there is always somebody familiar with the software available to test it.

Another benefit of having a whole team is that it makes it more obvious if you have resource imbalances. If the QA folks in the team can’t keep up with the developers, then you either have too many developers on the team or too few QA folks. If your front end developers can’t keep up with the folks working on the back end, then you have a resource imbalance. You either need to add more folks with the right skill set, or you need to reduce the amount of work being done on the back end.

When you have a whole team, you spend less time waiting for other groups and bringing part-time participants up to speed, you lose less time due to communication delays, and individuals spend less time multi-tasking between multiple projects. Taken together, the benefits of whole teams can have a significant impact on reducing management complexity which makes it easy for teams to do more self-management.

Collocation Further Improves Coordination and Communication
Collocation is closely related to whole teams. Collocation is simply having everybody in close proximity to each other. This compounds the coordination benefit of whole teams.

If you have a large group of people that are not all near each other but they are all working towards a single deliverable, collocation may seem impractical on the surface. However, collocation at the whole team level is more important than collocation of multiple teams. Focus on creating whole teams that can collocate rather than collocating the whole group of people.

The challenge with both whole teams and collocation is that they often require either relocating people, changes to management structures, or both. However, the benefits of having a tightly coordinated team working towards a common goal far outweighs the difficulties associated with creating collocated whole teams.

Next: Three Cures for the Management Bottleneck

“No-Brainer” Self-Management Practices in Agile

2009-01-16T16:22:00.005-05:00

While there may be some confusion over the meaning of self-management in Agile, not to mention some contention as to whether or not it is a good idea, many of the Agile practices contribute to self-management without requiring the wholesale adoption of self-management. For simplicity, I will define “self-management” as anything that is done by a team member that would traditionally be done by a manager. The practices contribute to self-management either by reducing the amount of management required or by encouraging the delegation of management tasks to team members. Some practices do both. As a result, managers have more bandwidth for dealing with things which are a more leveraged use of their time.

The Agile practices related to self-management include:

whole teams
collocation
short iterations
using a scrum master
using a product owner
user stories
maintaining a product backlog
assignment of tasks by team members
estimation of tasks by team members
producing a burn down chart
retrospectives

Simplifying and Redistributing the Management Load
Let’s start with a simple example: task assignment. Traditionally, tasks are assigned to engineers by their manager. In Agile, tasks are selected by the engineers themselves. It may take some time for everybody to get used to this and to work out the kinks, but the practice itself is not particularly earth-shattering.

Certainly, somebody may find themselves struggling with a task they don’t fully understand, or one engineer may accomplish a task more slowly than whoever the manager might have chosen to do the task. But is that really all that different than what typically happens when the work is assigned by a manager? How likely is it that a manager has enough time to devote to the minutia required to optimize task assignment? Wouldn’t it be better for the manager to spend their time mentoring team members on how to pick tasks that are appropriate for them while at the same time encouraging engineers to stretch their capabilities?

Next: "No-Brainer Self-Management Practices in Agile Part 2"

The Use of Self-Managing Teams Outside the Context of Software Development

2009-01-05T14:21:00.008-05:00

[Note: this post was heavily edited on Jan 18th to take into account comments on the earlier version.]

Most of what you need to do to become Agile requires only re-arrangement of what you are already doing. For instance, managing a backlog is still a process of deciding which work to do and in which order to do it, it isn't something that boggles the mind and makes people say "some people might be able to do that, but I don't see how we could." Managing what work to do with a backlog is like tuning an engine. It is straight-forward to explain, learn, and implement. Whereas, moving from conventional management to self-managing teams sounds like moving from steam to internal combustion. It is completely different and not well understood. But wait, what exactly are self-managing teams?

The terms “self-organizing,” “self-managing,” and “self-directing” have been used to mean many things, but are generally thought of as being in the same ballpark. In this post I will use the term “self-managing.” While looking for other writing on the topic prior to writing this post, I found that it is a poorly defined topic within the Agile literature.

What the Heck is a Self-Managing Team Anyway?
What sort of management does self-management apply to? What is the scope? Does it just apply to how I manage myself on my pre-assigned tasks? Does it only apply to project management? Does it mean dealing with hiring and firing? What about budgets? Some folks have taken it to mean that there is no manager, that the Scrum Master is the manager, etc. Sure, there are lots of short sections on self-managing teams in Agile books and articles, but it is not much more than a vague description of the concept repeated in different ways by different authors. This is not a criticism, only an observation that self-management it is an underdeveloped area of Agile.

In the Poppendieck's "Implementing Lean Software Development" there are four pages on "Self-Directing Work", most of which talks about project management. Kent Beck's "Extreme Programming Explained" takes self-management for granted without specifically talking about it. Ken Schwaber's book "Agile Software Development With Scrum" contains a couple of paragraphs on self-organization but mostly takes it for granted. In general, most of the material on the topic assumes that you will be doing it and there are practices which are meant to support it, but there is no in-depth discussion of exactly what it is and how to do it.

In searching the web for a more in-depth treatment, I did find one really good article by Esther Derby in the December 2008 issue of Better Software Magazine entitled "What's a Manager to Do?" This article gives a good overview of the concepts as well as some pitfalls and how to avoid them.

I had better luck finding reference material on self-management by looking outside of software development all together. At some point during my searching, I accidentally left off the keyword "Agile" and discovered that there are whole books on self-managing teams, and they have nothing to do with software development at all. I ordered three of them and am currently in the process of reading them. I'll post new entries as I go.

Here are the books on my reading list related to self-managing teams:

Business Without Bosses
Leading Self-Directed Work Teams
The Wisdom of Teams

Apparently, people have been promoting and practicing self-managing teams completely outside the realm of software development for quite a while. While it holds promise, it does not yet seem to have become a widespread practice. If it is still in its infancy in general, is it really a good idea to promote it in conjunction with all of the other parts of Agile which, IMHO, are much more straight-forward and are closer to our core-competencies as an industry?

Even though self-managing teams are not well defined or described in Agile, it doesn’t really matter. There are actually many no-brainer “self-management” oriented Agile practices that are straight-forward and beneficial in their own right. The fact that they may get us closer to creating self-managing teams is a side benefit.

Next: "No Brainer Self-Management Practices in Agile"

Jim Coplien's New Book on Agile Architecture

2008-10-31T14:23:00.004-04:00

Did you know that Jim Coplien will be publishing a new book on Lean Architecture called "Practical Agile Production?" Find out more by attending Deep Agile 2008 at MIT on Nov 8-9 2008 and receive an autographed manuscript of the book, a once-in-a-lifetime opportunity! Find out more at the Agile Bazaar's Deep Agile 2008 web page.

SD East - Oct 30th - Multi-stage Continuous Integration Presentation

2008-10-28T00:22:00.002-04:00

Interested in Continuous Integration? Come see my presentation at SD Best Practices this Thursday at 8:30am in room 102 to see how it scales to teams from 20 to 500 with Multi-stage Continuous Integration. See the presentation conference page for more details.

Oct 23rd - Presentation on Multi-Stage Continuous Integration at MIT

2008-10-19T13:57:00.004-04:00

Come hear my presentation on Multi-Stage Continuous Integration!

I'll be speaking at this month's Agile Bazaar meeting.

MIT Tang Center
Building E51-Third Floor
(Room number TBD - We will meet outside Room 335)

6:30 - 7:30 Presentation
7:30 - 8:00 Q & A, Networking

We usually walk over to the CBC or similar venue afterwards to continue the networking.

Continuous Integration is an increasingly popular technique for discovering and fixing problems early. As larger groups start to adopt it, a problem arises. A full integration build and test cycle can easily take 30 minutes. The chances rise exponentially with project size that during that time, other people's check-ins will silently invalidate your results. Multi-Stage CI is an extension of the common practice of shielding others from our changes by only checking-in when we have tested our changes and only updating our workspace when we are ready to absorb other people's changes. With Multi-Stage CI, each team does a team-based CI first and then cross-integrates the team's changes with the mainline on success. This limits project-wide churn and allows CI to scale to large projects. This session will introduce Multi-Stage CI, cover the pros and cons, and give examples of how to
implement it.

Scaling Agile to Large Teams

2008-10-08T22:16:00.002-04:00

Read what Scott Ambler and I have to say about scaling Agile for large teams in "Agile Development: It isn't just for small projects" from SearchSoftwareQuality. Scott has an interesting point about bureaucracy versus discipline which I haven't heard anybody make before.

Deep Agile 2008 - Reminder

2008-10-08T20:40:00.003-04:00

In case you haven't heard about it, or forgot to register for this great event, you need to register for the Deep Agile 2008 event before the price goes up! If you register before October 14th, you'll get the early bird price of $500, which is $100 off the standard price. And for readers of my blog, when you register, enter the discount code of DA20087UPAN for a further $50 off. You’ll pay only $550 – a full $150 savings off the standard price!

What: Deep Agile 2008: Not as Easy as you Thought!
When: November 8-9, 2008, 9:00am to 5:00pm (Registration opens at 8:30am)
Where: MIT, Cambridge, MA - Room E51-345 Hosted by Jay Conne

Space is limited to 90 in this MIT seminar room and it has been selling very well. So register early! Details are on the Agile Bazaar website.

Deep Agile 2008 - Not as Easy as You Thought!

2008-09-11T09:24:00.002-04:00

A 2-Day Intensive Conversation With James Coplien and Bob Martin

The sparks will fly when two passionate professionals - Jim Coplien and Bob Martin - square off to make the world safe for software development.

In the corner of architecture, patterns and agile is Jim Coplien.
Driving the necessity of test based design is "Uncle Bob" Martin.

Each will use their long track records, numerous case studies, and success stories to argue that they have the answers you need to deliver successful projects and products. The difference here is that we are presenting both sides of the story, and working with Jim and Bob to show how both approaches meet in the arena of professional software development.

Come prepared to be surprised and have your assumptions questioned! Our goal is to get well beyond the buzzwords and introductory agile ideas, and to get you thinking.

What: Deep Agile 2008: Not as Easy as you Thought!
When: November 8-9, 2008, 9:00am to 5:00pm (Registration opens at 8:30am)
Where: MIT, Cambridge, MA - Room E51-345 Hosted by Jay Conne

Only $495 at the Super early bird rate. That expires on Sept 23 so don’t put off registering! And for readers of my blog, when you register, enter the discount code of DA20087UPAN for a further $50 off. You’ll pay only $445 – a full $205 savings off the standard price!

Space is limited to 90 in this MIT seminar room and we are expecting to sell out early. So register early! Details are on the Agile Bazaar website.

Software is Indistinguishable From Magic

2008-07-30T16:41:00.010-04:00

Plush red curtains withdraw to the sides of the wide stage, only the dark of midnight can be seen beyond them. A tall figure with a top hat strides into the spotlight at the center of the stage. The weight of anticipation presses us into our seats like breathless astronauts at take-off as we wait for something “never before seen by any audience.”

The magician raises both hands into the air as though preparing to pull himself up on an invisible bar. After a pause just long enough to hear your heart beat loudly twice, the magician pulls down hard on nothing until his knuckles rap the floor. Lightning flashes and thunder cracks. The stage is filled with a pyramid of elephants, maidens juggling bowling pins, and a flock of doves darting over our heads and towards the back of the theater.

Software is indistinguishable from magic. It gives us the power to conjure new capabilities at the click of a mouse. Magic springs from our fingertips. We move electronic mountains of information, we uncover patterns that were previously invisible. New revenue streams well up out of the internet. Users eagerly try new software, hoping for a magical experience. Developers demonstrate new software reveling in the reaction of the users, journalists, and bloggers.

Timing is everything. The only magic of poor timing is a disappearing audience. When the search engine Cuil was launched, users of Google everywhere wondered what sort of new trick Cuil had up their sleeves. 120 billion indexed pages seemed unfathomable. Unfortunately, Cuil’s timing was off and the effect fell flat. Just as it would be if you saw a wire during a levitation act, Cuil was revealed to be ordinary software with bugs and scalability problems.

When the timing is right, as it was with the introduction of the first iPhone and its amazing user interface, software can send a tingle up and down your spine. I still remember my first experience with software. It was a simple Tic-Tac-Toe program at the Boston Museum of Science in the early 70’s. There were just a few stations and you had to wait your turn in line. It was amazing to me that a computer could play Tic-Tac-Toe against a person and win.

I kept going back for more until Dad asked me if maybe I’d like to visit some of the other exhibits. I suggested we go to the gift shop because I knew they had a plastic mechanical computer kit for sale. It was called a DigiComp and it could be programmed to do simple computation including playing the game of Nim. It was pure magic.

What was your most magical software experience? Post your comment below!

Part 2: The Magic of Demos

The Magic of Demos

2008-07-30T16:32:00.004-04:00

When I started writing software, I enjoyed the thrill of showing people something they hadn’t seen before. Even today, one of the main reasons I enjoy working in the software industry is the thrill of demoing new software. When you demonstrate new software, you become a magician, conjuring feats of computation that dazzle the imagination. The audience starts out skeptical, wondering if you are just a two-bit side-show act. You slowly build up to the main event and then, when you’re lucky, they gasp in amazement as you show them something that they’ll no longer be able to live without.

One of my favorite demos was many years ago when I was showing an early version of a product to some folks for feedback. As part of the demonstration I interrupted the power to the laptop (with the battery already removed), and showed them that the software continued as though nothing had happened when the power was restored.

Even though we had told them that the software wouldn't ship for at least six more months, they called us the next day to place an order anyway. For them, the value outweighed the risk. We decided to accept the order. That early exposure to a real customer changed the way we thought about things. Even though we considered the product to be pre-release, we made sure that every new feature worked as it was developed instead of waiting until the end game of the official release. As a result, the end game was much smoother than we had expected.

The Magic of Agile Development
From a business perspective, the main reasons I appreciate Agile development are an increase in quality and ROI, more options, and higher visibility into progress and status compared to traditional development. But from a purely personal perspective, the reason I enjoy Agile development is because it made my job more fun.

Today, thanks to Agile development, I interact with customers more than ever before. As a product owner, I do more demos and am able to provide new features that hit closer to the mark faster and more frequently than ever before. This in turn means more oohs and ahs from customers which is more fun for me and more profitable for the business.

Related:

The Simplest Thing That Could Possibly Work

2008-07-02T23:05:00.006-04:00

One of the principles of Agile, mostly related to design and architecture, is “The Simplest Thing That Could Possibly Work.” This is sometimes taken as a license for cowboy coding. But that is not the intention. A better way to express it would probably be something like “The Simplest Solution That Could Possibly Satisfy Your Requirements.” For instance, if you have a requirement to create the back end for a web site like amazon.com, then while a perl/cgi solution on a single core machine could possibly “work,” it doesn’t work from the point of view of high availability, fast response time, or reliability.

From Oversimplification To Rube Goldberg
On the one hand, there is a wide spectrum of complexity of construction ranging from doing nothing to Rube Goldberg level complexity. On the other hand, there is the set of solutions that work, meaning that they meet all of the requirements. TSTTCPW refers to the solution which works and which is lowest in complexity.

Part of being simple means simple to read, maintain, use, design, understand, and implement balanced against the time it takes to get the job done. Spending too much time to create the ultimate in simplicity starts to get you into a different kind of trouble.

As somebody that struggles to apply this principle on a regular basis, I was happy to stumble upon an example of this principle which can be captured in a picture and kept in mind as I am working on a new design. Perhaps you will find it to be useful food for thought as well.

A Bridge Too Far
There’s a construction project that you’ve probably heard of which is affectionately called the “Big Dig.” Part of this project was the construction of the “Leonard P. Zakim Bunker Hill Bridge” aka the “Zakim bridge.” This part suspension bridge, part cantilever bridge is an enormous one of a kind architectural marvel. It supports five lanes of traffic in either direction for a total of ten lanes. It was built at a cost of approximately $11M per lane.

Running parallel to the Zakim (to the left in the photo) is the Leverett Circle Connector Bridge. It serves a total of four lanes of traffic. It was built at a cost of approximately $5M per lane.

Part of the requirements for the Zakim bridge were clearly “create a stunning new Boston landmark.” On the other hand, the Leverett Bridge is a very simple but also very strong bridge. It could have been made even more simply, but not without a safety risk and/or a shorter lifespan. In other words, it is “The Simplest Thing That Could Possibly Work.”

Next: The Faberge Egg

TOC: Zero to Hyper Agile in 90 Days or Less

[Note: revised 7/3/08 to reflect comments on reddit. Clearly the original post didn't work. :) ]

Many Hands Make Light Work, But I’ve Only Got Two

2008-06-25T17:06:00.008-04:00

When choosing how to allocate resources, it can be difficult to do an effective cost benefit analysis in a short period of time. A technique that I stumbled upon is to look at things from the perspective of “what would I do if I was the only person on the team?”

In 1992, I convinced my manager at the Open Software Foundation (OSF) to create the position of tool smith and allow me to work on the OSF Development Environment (ODE) full time. As I now had much more time for doing development and made many more changes to ODE than I had ever made before, there was much more testing to be done.

Initially, I didn’t realize just how much testing was fully required for ODE. Not only were there no automated tests for ODE, there were no documented test cases either. The test plan consisted entirely of “round up as many volunteers as you can and have each volunteer test on one of the nine platforms.” The testing that each volunteer did was entirely up to their discretion. It was different volunteers every time, and there was no record of what they did, only a “seems fine to me” or “I found these problems.” Once the number of problems got down to an acceptable level, we’d ship the new version.

After a couple of releases it started to sink in that I had to do something differently. My first attempt was to document a standard set of test cases. At first this seemed to work really well. Testers commented that it was much easier and took much less time. I felt like I had gotten a consistent set of results from a consistent set of tests. But a test/fix cycle requires running the same tests over and over again until things stabilize. Following the same steps over and over again can become pretty mind-numbing. Pretty soon I couldn’t get the volunteers I needed and I was starting to suspect that people were skipping steps.

I also discovered another problem. As bug reports came in from production use, and as I thought of test cases that were missing, the list of test cases mushroomed. I was very careful not to include overlapping test cases, but still the list grew and grew and grew.

Since the QA of the tool was ultimately my responsibility, I had to pick up the testing slack. The combination of the ever increasing list of test cases and the diminishing pool of volunteers soon made the QA of a new release pretty much impractical. I would end up spending much more of my time testing than coding.

But then I remembered how I had gotten the job of tool smith in the first place, by automating myself out of my previous job of manually kicking off builds on all platforms. Test cases are basically a set of steps to take and the expected results. Automating test cases is basically coding, and that was much more fun than manual testing. A couple of inspired weekends later I had automated all of the test cases and added many more new ones as well.

That was the last time I ever relied on manual tests.

Read more from: "Zero to Hyper Agile in 90 Days or Less"

Reinvest in Your Engine by Improving The Work Environment

2008-06-25T16:03:00.003-04:00

There are really only five ways to increase the profitability of a business based on software development: reduce costs via outsourcing, reduce headcount, reduce other expenses, increase productivity or increase revenues. Reducing expenses can only go so far. The most expensive part of software development is the people. Thus, one of the most successful ways to increase profits is to increase the productivity of the software development team.

The Agile Workplace
At Litle & Co., developers like the fact that Agile provides the additional challenge of solving business problems instead of just technical problems which requires thinking at a higher level. Developers at Litle report that they have a higher level of job satisfaction than in previous companies that were not using Agile because they see the results of their software development efforts installed into production every month. Also, they like the fact that there is much less work which amounts to “implement this spec.”

Your development infrastructure is really no different than the general company infrastructure which includes your cube or office, the carpet, the artwork on the walls, the company cafeteria, your phone, your computer, and the company location. These are all part of your work environment. If you have a computer that is 5 years old, your work environment is not as good as if you have a computer that is only 2 years old. If you are writing in C rather than C++, C# or Java, your work environment is sub-optimal.

The closer that your development infrastructure is to the ideal environment for your circumstances, the more productive your team will be. This principal extends to all aspects of the development environment, from development language, to build system, to build farm, to issue tracking system, to the process that you follow.

Next: Your Development Process is Part of Your Work Environment

Your Development Process is Part of Your Work Environment

2008-06-25T15:55:00.003-04:00

Your development process (regardless of how it is implemented), is also part of your work environment. If as a result of your development process you regularly end up redoing work because problems weren’t discovered until just before the release, or projects get cancelled or shelved, then this is also likely to reduce productivity and job satisfaction. As this process improves, so does your work environment. The smoother it operates, the more pleasant your working environment will be.

There are many problems which you may think of as being unrelated to your development process. For instance, broken builds. Broken builds are simply the result of somebody making an idiotic mistake, right? Perhaps that’s true some of the time, but most of the time it is due to the complexity of integrating many changes made by many people for software that has many interdependencies.

To be sure, a “perfect” process does not guarantee happiness, success, or the absence of problems. You still have to debug complicated problems, port to new platforms, deal with unforeseen circumstances, etc. However, the state of your process impacts the efficiency with which your effort is applied.

If your process is perfect and completely frictionless, then 100% of your effort will be applied to the work that creates value. If it is rife with problems, it may mean that only 50% (or less!) of your effort will be applied to work that creates value. If there are problems with the process, then you are already expending effort which is essentially wasted. You would be better off investing some of that effort in removing the problems permanently instead of losing it to friction on a regular basis.

Next: Quick Summary of The Benefits of Adopting Agile

The Benefits of Adopting Agile

2008-06-25T15:45:00.004-04:00

The benefits of moving to Agile development can be split into two categories: benefits to the organization, and benefits to you personally. As a result of increased productivity, higher quality, and responding more rapidly to market demands, Agile can provide the following benefits to the organization:

Increased revenues
Reduced costs
Increased market share
Higher customer satisfaction

Each of these benefits lead to a stronger organization which is then in a better position to reward you for your efforts both directly and indirectly. Some of these benefits include:

Getting a raise and/or bonus
Having more discretionary income to buy cool stuff
Improving your working conditions
Actually using all of your vacation time
The opportunity to spend more time working on cool stuff

In addition, Agile can provide the following direct benefits:

A less stressful environment
Less canceled or shelved work
Career advancement due to learning new skills
Having the resume that gets you your dream job

Many of these same claims have been made in the past about tweaks to traditional development, but nothing ever came of it. Why will this be any different with Agile? Let's take a look at why attempts to "fix" traditional development haven't panned out.

Next: Having Dev Team Performance Problems Using Traditional Development? Try Niagra!

Agile Development Scenario

2008-06-25T12:21:00.005-04:00

Here's an Agile development scenario. For simplicity, let’s use two month iterations. Marketing says the three features with the highest ROI are a Facebook plug-in, a Second Life plug-in, and an RSS feed plug-in.

You start with the Facebook plug-in. You plan. You design. You create a test plan while the code is being written. You discover potential problems and deal with them. You automate those tests while the code is being written. As the code is written, it is integrated, built and tested continuously; problems are found and fixed immediately. At the end of development, the only problems that remain are the ones that could only be found at the end of development. If you wanted to, you could cut a new release with very little overhead.

Marketing announces that the Second Life plug-in is not as marketable as they had hoped and iPhone support is showing signs of becoming very lucrative. So, you start on the RSS feed support. Now marketing says the Second Life plug-in is worthless but iPhone support is hot, so you add iPhone support. During the whole process, you kept your options open. At the end you have produced more business value than using the traditional process, and you were in a position to start realizing that value much earlier.

To get the primary benefits of Agile, as described above, you don't have to use 3x5 cards or pair programming, and you don’t have to do frequent releases. These things are optional and you can use them or not according to your preference and circumstances.

Next: Achieving the Primary Benefits of Agile

Traditional Development Scenario

2008-06-25T12:12:00.004-04:00

A useful way to describe Agile is by contrasting it with traditional development. To do that, let's consider a typical development scenario: adding new features to stay competitive. First we'll look at the scenario using traditional development and then using Agile development.

In a traditional project, you have a known timeframe for major releases. Too short and you’ll spend too much time on overhead, too long and you’ll miss opportunities. For the sake of argument, let’s pick a timeframe of six months and say that allows you to provide three “big features.” Marketing says the three features with the highest ROI are a Facebook plug-in, a Second Life plug-in, and an RSS feed plug-in.

Midway through the design process, marketing announces that the Second Life plug-in is not as marketable as they had hoped and iPhone support is showing signs of becoming very lucrative. You think, oh well, nothing we can do about that now. Just after you finish coding marketing declares that the Second Life plug-in is going to be a complete flop and when can they get iPhone support?

Now that the functionality has settled down, QA starts writing tests and running them. Planning and development took longer than expected and the release deadline is looming. Testing time is compressed, QA concentrates on the most critical stuff and gives the rest a spot check. Here’s a mystery. If your full test cycle takes two days, why does testing take a month? The answer is that you aren’t really doing a month of testing. Problems have been creeping in all along the way that you are just now finding out about and it takes many test/fix cycles to expose them and fix them. Once the find/fix rate gets down to an acceptable level, you declare victory and deliver the new release.

Next: Now Let's Try Agile

Agile Bazaar Mtg at MIT, Thursday June 26th

2008-06-24T16:43:00.005-04:00

If you are interested in Agile and live in the Boston area, come check out the Agile Bazaar. It is a great bunch of folks that love to talk about Agile and have a lot to offer for all experience levels. For more details, please see the Agile Bazaar website.

The "Faberge Egg" Widget

2008-06-24T00:22:00.007-04:00

There was a developer that worked for me once, I’ll call him George. He wrote a lot of really good code. But one day he decided he wanted to make his mark on things. George believed that the way to do it was to create a beautiful widget. He wanted it to be so beautiful and so useful that it could be used generically and would be something that we could sell separately on its own merits.

The widget was part of a Java/Swing user interface for an issue tracking system. It was responsible for the data model used by a form, a query editor, and a handful of other objects. So, it did need to be fairly general purpose, but it didn't need to be so generic and so functional that it would be something that people would want to buy separate from the application.

George referred to this widget as his “Faberge Egg.” It was an apt name for the widget. It was overly ornate, intricately detailed, and supported a very wide range of functionality that we had no immediate use for and still don’t to this day.

I was very clear with George that I only wanted a “Cadbury Egg” and tried hard to convince him that he could provide much more value to the company in other ways and the company would reward him for that, not the Faberge Egg. I like folks to have the freedom to grow and explore. Even though we were under deadline pressure, I gave George some wiggle room while at the same time trying to guide him towards the simplest design that would work. Unfortunately, after weeks of work and many conversations, George’s desire to produce a masterpiece of a widget prevailed.

Over time, that widget has been the source of more than its share of problems, both in bugs and in making it more complicated for other developers to maintain it and extend it. It has been partially refactored several times, but it seems there has never been the time to really do the full job required.

The same functionality that the widget provides was needed in our new web interface. Instead of reusing the code as we have done for other functionality, we decided to start from scratch. Doing just what was needed for the job at hand took only two days and that code has been very reliable right from the start.

Next: Frequent Releases Improve Code Quality Faster

Related: The Iterative Design of a Transforming Lego Sports Car

Do You Need a Standup Meeting?

2008-06-23T23:56:00.006-04:00

Stand-up meetings are a great way to reduce delays in communicating important information. Another benefit of stand up meetings is the elimination of time-wasting status and progress meetings.

Stand up meetings are most closely associated with Scrum and are called “Daily Scrum Meetings” within Scrum, but have become populare independent of any particular methodology which is a good indicator of suitability for mainstream use.

A stand-up meeting is simple to implement. There are just a handful of guidelines:

Limit the time to fifteen minutes.
Pick a regular time for the team to meet, preferably in the morning.
Start on-time regardless of who is absent.
Each person answers these three questions:
What have you accomplished since the last meeting?
What are you working on next?
What impediments do you have?
All discussion and problem solving is deferred until the end of the stand-up meeting.
Follow-up as needed in smaller groups

Although it is called a stand-up meeting and standing is encouraged, the time limit is the most important part and standing is optional.

The point of a stand-up meeting is to improve communication and to discover and resolve impediments, not to have a meeting just for the sake of having a meeting. If the team feels that other practices make the stand-up meeting redundant, then by all means reduce their frequency or even discontinue use until such time as it appears to be necessary again.

To help make this decision, let’s take a look at the expense side of stand-up meetings. First, people have to get to it. And then they have to get back to their computers. Scrum discusses how to minimize this time, but practically speaking, there is more overhead than just the ideal 15 minute meeting. If you are at a larger company, somebody has to book the room and let people know where it is. Let’s call the cost of the meeting 20 minutes per person. If you have 12 people in a stand-up meeting, that’s 4 person hours per day. That’s the equivalent of half of a person. Those meetings had really better be worth it!

Now let’s take a hard look at the stand-up meeting itself. One of the basic ideas of Agile (and Lean) is continual self improvement. If the value of the meeting exceeds the cost, then there’s no problem with the meetings, especially if they are eliminating other meetings. If the stand-up meeting is the only remaining meeting, that seems like a good thing. However, continuous improvement means we’re never satisfied. Now that you are down to just the one meeting, you should still ask the question: “is it providing more value than the cost? Is there a better way?”

What is the purpose of a stand-up meeting? To quickly find out if people are getting their assigned work done and if not why not. If it is more efficient to do that via e-mail, IM, an issue tracking system, or other means, then use those means. Someone might say “but seeing folks face to face is worthwhile.” Ok, so why not just do that then? Go out to lunch together or something like that.

Or perhaps the stand-up meeting is needed because otherwise folks wouldn’t complete their work, or people wouldn’t speak up when they run into an impediment. In that case the stand-up meeting isn’t a solution at all, it is a crutch. For instance, perhaps somebody isn’t completing their work because they don’t like it, but the constant peer pressure of the standup meeting is goading them into completing their work anyway. So then the real problem is lack of job satisfaction or low moral or something along those lines. Until you fix that problem, the stand-up meeting is just acting as a band-aid.

The real measure of project status and health is having an increment of shippable value at the end of every iteration. A standup will only expose problems that are on people’s minds, but the forcing function of the increment of shippable value is where you will get the true picture of how things are going. A one month iteration interval is good, but if you can get it down to 2 weeks or even 1 week, that may do far more to expose real problems than a standup will.

Next: How Agile Helped Litle & Co. Get to #1 on the Inc. 500

Sustainable Pace: Supply vs Demand

2008-06-11T10:59:00.011-04:00

In a traditional project, the demand for resources from the four major aspects of software development see-saws dramatically over the course of a project. These aspects are project management and planning, architecture and design, development, and QA. You need resources on hand to serve the peak demand level, but during periods of low demand those resources will either be idle or used for activities which have a lower ROI.

A common circumstance is that there are insufficient resources on hand for the peak demand level and so people end up working in “crunch mode.” During crunch time, people tend to make more mistakes. Agile levels demand out over time and removes this see-saw effect which simplifies resource planning and removes the need for crunch time.

In the figure above, the straight green lines represent the resource load in an Agile project and the zig-zagging purple lines represent the resource load in a traditional project.

With traditional development, delays during development compress most of the testing to the end of the process which then requires taking shortcuts due to schedule pressure. I used to think that one way of compensating for insufficient QA resources was to delay the release until QA finishes. On the surface it seems to make sense. But only if the folks writing code sit on their hands while QA does their work. Ok, so you have multiple projects and the developers work on another project. But then they finish that. Now QA starts on the second project and the developers move to the third. The problem is still there.

On the other hand, as a result of the need for increased QA resources during testing, you may have two other problems. If you have enough QA resources to handle the pressure of the endgame, you may have too many QA resources during the rest of your development cycle. Alternatively, you may bring on additional QA resources on a short-term basis to compensate. Both of these options are obviously undesirable.

There’s a natural balance between the amount of effort required for developing something and the amount of effort required to QA it. No matter what you do, if you have the wrong ratio of development resources to QA resources, it will cause problems. If development creates more than QA can absorb, you will create a backlog of QA work that will always grow.

There are six options for dealing with a QA backlog: do less QA than needed and thus shift the burden of finding problems that you could have found onto your customers, increase your QA capacity, decrease your development capacity, have development idle, have development help with QA or allow the backlog to grow. The larger your testing backlog, the longer it will take to ship it and the greater your opportunity cost.

The imbalance may be in either direction. After you transition to Agile development, you may find that you have more QA resources than are needed. In that case, you have the option of having QA take on some of the work currently done by developers. See The Role of QA in an Agile Project for more on that topic.

This natural balance holds between all four aspects of software development. Depending on your organization, there may be an imbalance between supply and demand at any stage in the pipeline. Wherever there is an imbalance you have the same six options as described above. For example, you may end up with project plans that are never used, developers idle because the design isn’t ready yet, etc. To the extent that some of the resources are actually the same people you can use that fact to manage this problem.

When using short iterations, resource imbalances are easier to detect and correct. Having balanced resources means that all development activities are done consistently and on a regular basis and there is no need to take the shortcuts that are typical of traditional development.

Next: The Usability of Short Iterations

TOC: Zero to Hyper Agile in 90 Days or Less

Agile Adoption Stage 2: Establishing a Natural Rhythm

2008-06-09T16:43:00.004-04:00

The first stage in adopting Agile is Preparing for the Transition to Agile.

The next stage is to make some early and measurable progress to encourage adoption. As it is, you have actually already experienced some success. You’ve planted the idea of process improvement, you’ve got a map of the areas of general consensus and contention, and you’ve provided a forum for people to learn more about the current process, uncover misunderstandings and correct them.

One of the areas of consensus you’ve uncovered is very likely a problem which everybody agrees should change, there is a consensus or near consensus on how to improve it, enough pain to easily instigate a change, and a sufficiently small scope that the change can be made in a relatively short amount of time. That should be your first area of attack. It doesn’t really matter what it is or if it has anything at all to do with Agile. At this point, you just want to establish the idea that process improvement is worthwhile, doable, and worth doing on a regular basis.

Short Iterations
The main point of this stage of adoption is to discover the natural rhythm of your team and the project that you are working on. It may be a month, it may be a week, or it may be some other timeframe between 1-6 weeks.

For your first iteration, start with a goal of a one month iteration time. It is better to make it a goal rather than a hard requirement. Making it a hard requirement will encourage rushing to finish or taking shortcuts at the end which will defeat the purpose of establishing a rhythm. It will take time to get used to short iterations.

Don’t plan to do a release from the first iteration. Whichever iteration you do plan to do a release from, keep your release process the same for now. The purpose of the first couple of iterations using Agile is to identify obstacles in order to start removing them.

The Backlog
To determine what to work on for the first iteration, make a conservative estimate of how much work your team can take on. Create a backlog of the highest priority items, adding estimates to them as you go in order to determine the cutoff point. You don’t need to have a product owner or ScrumMaster or make any changes to how you plan a release at this point. Just do whatever you would normally do to plan the contents for a release, but keep the timeframe to a single iteration.

If there are one or more work items which are too large to fit into a one month iteration, you have a variety of options at this point. First, you can postpone going to short iterations until they are done and concentrate on the other aspects of the first stage of adoption. I don’t recommend this. If you’ve gotten to this point, this is your chance, don’t postpone it, find a way to move forward.

Second, if the work items would fit into a slightly longer iteration, use that as your iteration time. It isn’t ideal, but it is a better alternative than giving up!

A third option is to delay the oversized work items until after you’ve got 1-2 iterations under your belt. If they are in your backlog, somebody thought they were important, but if your release isn’t for many iterations anyway, perhaps it won’t matter to the overall release.

Done Means Done, But What Does "Done" Mean?
You should have a clear definition of what “done” means for a work item. At a high level, any definition of done should include whatever you believe is necessary for that work item in isolation to be considered shippable. Think about your entire release cycle, not just what it takes to get something ready to check-in. Do you eventually do a code review? Then that should be a pre-requisite for “done.” Other candidates are: documented, tests written and all pass, demoed by a QA person to a third party (that is, not just the developer and the QA person themselves).

Iteration Retrospective
Before you get too giddy with the success of your first iteration, be sure to schedule a retrospective. This is exactly same as the traditional “post-mortem” but doesn’t have the same air of death about it. The purpose of the retrospective is to take a look back at the iteration and look for opportunities to improve your next iteration. What went wrong? Why did it go wrong? What went well? What ideas do people have for improving the next iteration?

Most likely, the duration of your first iteration overshot your goal. This should be the focus of the retrospective. If in fact you did make the goal of 1 month, what could you do to reduce the time to 2 weeks? What could you do to have a higher ratio of value-producing activities such as adding more new functionality? You should not change your goal of 1 month to 2 weeks, even if you made the goal. At this point, it is more important to establish a rhythm rather than reduce the time of the interval.

At the end of the retrospective, decide as a team which changes you want to make in the next iteration. Don’t try to change everything at once, concentrate on the top 3-5 ideas. Make sure that somebody is taking notes. The notes will be invaluable in your next retrospective for detecting trends.

Repeat
Breaking down entrenched habits and ingrained beliefs while simultaneously establishing new habits and beliefs takes time and patience. The most important things at this stage are to establish a rhythm via the short intervals of the iterations and to find and remove obstacles. Because you should already be experiencing positive results, there is no need to rush to the next stage and endanger the progress that you’ve made. Remember that there are lots of people watching the success or failure of this process and support for a new way of doing things is still shaky at best. That includes you and everybody else on the team! I recommend you simply repeat the process with another 2-3 iterations, finding and removing obstacles as you go.

Considering a transition to Agile? Read more of "Zero to Agile in 90 Days or Less."

Preparing for the Transition to Agile

2008-06-06T11:37:00.008-04:00

Once, when I was just starting to snowboard, I was at Sugarloaf for the weekend and they had very little cover and very few trails open. But then Saturday night, they got 33” of powder. A friend and I came to a trail that was closed. It looked like a great trail; endless powder with no tracks. The problem was that it had been rocks and grass the day before and there was no base underneath, so it was just the same as riding on rocks and grass. It was not a pleasant experience. Adopting Agile without understanding it and without creating a proper ecosystem for it is destined for a similar fate.

Adopting Agile development requires breaking down mental barriers and building up new skill sets. There is nothing particularly hard about actually doing any of the Agile practices, but many of them are counterintuitive and do take a bit of practice to get used to. That said, don’t underestimate the amount of effort required. The effort required is at least on the order of taking a team which is very used to writing in C++ and moving to Java. There’s nothing particularly difficult about such a transition, but there are many subtleties which must be learned and it takes a while to build up the same base of experience.

Self Education
Before getting too far along, make sure that you have done your homework. Read other books on Agile, find other folks in your organization that have done Agile development before. Go to conferences, join the local Agile user group, become a certified Scrum Master, do whatever you do to find people that you can lean on when you need it.

Scope
Determine the best scope of the adoption. As with most things, it is best to think big, but start small. Is there a small project with no more than 12 people that is amenable to piloting something new? There are two advantages in starting small: minimizing disruption and leveraging what the pilot group learns about doing Agile in your environment when transitioning other groups.

Scouting
Agile development has certain perceptions related to it. One of the most prevalent perceptions is that it is “for small groups.” That was certainly my perception when I first started hearing about it. Another perception is that small iterations aren’t a good thing because customers don’t want frequent releases, there’s more overhead involved, the quality will be lower that way, and it makes marketing’s life more difficult.

If you just advocate Agile without knowing the landscape, you run the risk of alienating the people whose support you need in order to go Agile. Find out how receptive your organization is to going Agile. Think about who is in a position to help or hinder its adoption. Those are the key stakeholders. You will need to find out where they stand, what they like about the idea of going Agile, and what their objections are. This information will come in handy later in the adoption process.

Prepare Your Organization
Once you have a basic lay of the land, see what you can do to raise people’s awareness and understanding of the advantages and potential pitfalls of Agile. Do a presentation for folks that are interested, invite in somebody from the Agile community to do a presentation or workshop. Recommend books and websites that you found helpful.

Transition Development and QA Together
The most important component of reducing the rework period that comes from long iterations is improving your testing process. For many if not most organizations, this is the hardest goal to achieve in practice.

If you don’t have any automation at all, it is a good bet that there is an ingrained belief that automated testing is either a bad idea, doesn’t work as well as manual testing, is too expensive, or that the tools are too expensive. As a result, it may be that there are no QA automation experts in the building and possibly nobody with scripting skills in the QA group. The best course of action in this case is to concentrate on introducing the idea of QA automation.

If it is clear that there is a bias against automated testing that is too strong to overcome any time soon, another tactic is to have the development organization champion automation with an eye towards handing it over to QA once the idea catches on. A good place to start is with unit tests. It should be clear from the start that your goal is to have QA own test automation. Developers will write good tests, but they are too optimistic by nature. Developers start from the assumption that “it will work.” QA people start from the assumption that “it doesn’t work and I’ll prove it to you.” Pessimism is a good trait for a person creating tests.

Keep Your Healthy Skepticism
Think carefully about the value of each practice that you plan to adopt and make absolutely sure that it is appropriate for your exact circumstances before you adopt it. You shouldn’t be adopting practices simply for the sake of saying that you are adopting Agile practices. Every practice you adopt should be because now that you know about it you simply can’t imagine getting by without it.

Don’t Throw Out the Baby With the Bath Water
I’ve seen many first-time Agile projects fail because they threw out everything they already knew about developing software. Most of the individual steps of developing software with an Agile methodology are the same as traditional software development. You still have to talk to customers, decide what you are going to do, write code, write tests, do testing, etc. Agile development is simply a different framework for those steps. You have a business to run, and you don’t really need to introduce large and sudden risk factors. Before you decide to chuck everything that you know and start from scratch, spend some time developing your knowledge and understanding of Agile. Look closely at your existing practices and see which ones will fit well within Agile, and which ones may cause problems. Create a game plan and start out gradually.

Next: Agile Adoption Stage 2: Establishing a Natural Rhythm

How Does Choice of Methodology Influence Strategy and Tactics?

2008-06-05T13:05:00.012-04:00

Once More Unto the Breach, Dear Friends
As Helmuth von Moltke (the Elder) said, “No battle plan survives contact with the enemy.” In the case of software development, the “enemy” is reality. What we think we need to do to satisfy customers and how we think we need to implement it generally changes as we implement and as we get customer feedback.

Similarly, Dwight D. Eisenhower said, “In preparing for battle I have always found that plans are useless, but planning is indispensable”. Planning (short and long, tactical and strategic) is still important in the Agile world. The difference is that Agile is specifically designed to take advantage of the fact that planning, design, and development have a learning component and to accommodate and embrace the resulting need to change plans on a regular basis. As you discover new information during an iteration, you can easily adjust your plans to take what you have learned into account in future iterations.

Too Busy to Plan
Short iterations and other Agile practices such as “You Aren’t Going to Need It” (YAGNI), may seem to indicate that Agile is tactical in nature. However, I would say that it is when looked at through a Waterfall lens. For instance, with Waterfall the planning horizon and iteration length (the full start to finish time) are the same. So if you look at Agile with this perspective you might think that the longest that Agile folks look ahead is 30 days. You might then conclude from this that Agile can only be used tactically. But in fact there is no connection between the planning horizon and the iteration length.

Many teams use very short iterations; two weeks or one week. What happens within an iteration is whatever has been planned for that iteration. Different Agile methodologies use different approaches for planning iterations. In any case, you can plan as far out as you would like. As an Agile product owner I can tell you that some of my products have plans going out multiple years.

Sometimes there is work that can’t be fit into a single iteration (whatever iteration length you have chosen). In that case, you can use multiple overlapping iterations. For instance, a one-week iteration for most stuff and an overlapping one month iteration for the exceptions. Over time though, teams generally get better at avoiding the need for exceptions.

Wherever You Go, There You Are
In my experience, whether the software development of a company is done tactically or strategically is connected to the planning culture of an organization, not the methodology. If an organization thinks and plans strategically, that will be reflected in their software development whether they are using Agile or Waterfall.

I’ve seen plenty of companies that use Waterfall but think tactically, packing mostly tactical features into a large release. Feature creep is a good example of how tactical thinking is fairly common in Waterfall projects. While of course it is true that “that shouldn’t happen” the facts on the ground are that it does.

I’ve also seen plenty of Agile projects that were strategic in nature. For instance, an Agile company that thinks strategically will have a strategic plan which they carry out using multiple iterations. As they discover that they need to do something tactical in the middle of their strategy, or they need to change their tactics in support of their strategy, it is simple to do with short iterations. They just change their plans for future iterations. As soon as their current iteration is done, they begin the next iteration which has now changed to take into account the new information.

It may well be that folks that think and plan strategically are less inclined to look deeply at Agile because it appears on the surface to be tactical in nature. I know that was true in my own case. I believe that people and organizations that think strategically will bring that thinking with them into the Agile world. Having crossed over I can report that I have been able to very happily and successfully bring strategic thinking with me to the other side.

Read more about Agile in "Zero to Agile in 90 Days or Less" .

[Note: this post was inspired by Jordan Bortz’ post, Agile and Waterfall are Two Sides of the Same Coin where he states that ‘ “Waterfall” is basically a Strategic method of attaining goals, and “Agile” is basically a Tactical method for achieving goals.’ ]

An Agile Case Study: Litle & Co.

2008-05-22T08:38:00.008-04:00

As part of the book that I am working on, "Zero to Agile in 90 Days or Less" I decided to include some case studies to illustrate how Agile is different, the problems that people run into and the solutions that they find, as well as the results. One of the companies that I discovered in my search for case studies was Litle & Co. I was immediately struck by their high level of expertise and success with Agile. I couldn't wait to hear how they did it, their trials and tribulations, and I am very grateful to them for allowing me to use them as a case study. The case study speaks for itself, so without further ado...

Case Study: Litle & Co.
Litle & Co. is a leading provider of card-not-present transaction processing, payment management and merchant services for businesses that sell directly to consumers through Internet and Multichannel Retail, Direct Response (including print, radio and television) and Online Services. They interface with the world’s major card and alternative payment networks, such as Visa and Master Card, and PayPal, and must be available 24/7 with no downtime.

During the founding of the company in 2001, the initial team of six developers wanted to use Extreme Programming (XP), an Agile development methodology. Some of the developers had used it before and the rest had read about it and liked the ideas. When they talked to Tim Litle, the founder and Chairman, about using Extreme Programming (XP) they told him “you’ll have to accept that we won’t be able to tell you exactly what we will deliver in 9 months.” Tim said, “That’s fine with me, I’ve never gotten that anyway!” He agreed not only to use it, but to insist on it as the company grew and brought in additional management. He liked the idea that he could change his mind about prioritizing feature development for merchants without cancelling or shelving work in progress.

In 2006, Litle & Co. landed at No. 1 on the Inc. 500 list with $34.8 million in 2005 revenue and three-year growth of 5,629.1 percent. In 2007 Inc. magazine cited Litle & Co.’s 2006 revenue as $60.2 million, representing a three-year growth rate of 897.6% over its 2003 revenue of $6.0 million. How has Litle achieved these impressive results? One factor that they cite is their use of Agile development.

Litle uses many of the XP practices including pair programming. The director of software development, David Tarbox, said “at first I thought I would hate pair programming, but I quickly came to enjoy it and depend on it.” Some of the side benefits that they see in pair programming are better on-boarding of new developers due to the mentoring and training that is inherent in pairing. They also like the fact that all code was developed by two people which provides both redundancy for knowledge of all code and real time code review. It may be that one of the developers for a piece of code is on vacation or otherwise unavailable when somebody needs to talk to them, but it is rarely the case that both are.

They started with weekly iterations then moved to 2 weeks then 3 and are now monthly. They gravitated to a month because it was the easiest cadence for everybody to work with and it meant that the whole company was able to adopt that same cadence. Agile development is part of the psyche of the whole organization. It permeates every aspect of the organization.

One thing that other areas of the organization had to get used to was that they had to be very clear about what they wanted from the development organization because there is no slack time. If you want something and you expect to have it done in the next iteration (monthly cycle), you have to be able to state it very clearly or it won’t even get started. It also means that development is very closely connected to the corporate strategy. For every task, developers ask “is this aligned with our overall direction?”

Agile has had a very positive effect on their hiring. Developers like the fact that Agile provides the additional challenge of solving business problems instead of just technical problems which requires thinking at a higher level. Developers at Litle report that they have a higher level of job satisfaction than in previous companies that were not using Agile because they see the results of their software development efforts installed into production every month. Also, they like the fact that there is much less work which amounts to “implement this spec.”

Litle’s software is hosted at multiple off-site datacenters and they upgrade their software every month like clockwork. In reality, they are doing ~7 week iterations which overlap such that they are releasing monthly. The first week of the development cycle is planning, the next four are development, and the final two are production acceptance testing (PAT) and deployment. Each iteration starts on the first Monday of the month. Some iterations are 4 weeks and others are 5 weeks, but there are a total of 12 iterations a year.

Although updating such a mission-critical application might seem risky to do every month, a lot of the perceived risk comes from the typical lack of fully-automated testing. Litle has a very large body of automated tests (over 30,000) and also has a test system which simulates the usage patterns and load of organizations like Visa and Master Card. Every change is subjected to the same sort of testing that typically takes traditional development organizations multiple person years of effort to achieve. As a result, the quality of their monthly upgrades is unusually high even compared to traditional product release cycles of six months to a year or more.

Their development teams are entirely collocated with no offshore or offsite development teams. For each project they assign a team consisting of multiple pair programming teams. Over time they have grown to a development team of 35 including QA and a codebase that contains 50,000 files including test cases and have had to add practices in order to scale XP. The biggest problem they ran into as they have grown was integration of newly developed code into the existing codebase. To address this, in 2007 they added the practice of Multi-Stage Frequent Integration to their implementation of XP. They do frequent integration instead of continuous integration because a full build/test cycle takes 5 hours.

Prior to implementing Multi-Stage Frequent Integration, they would have to manually pore over all build and test failures to determine which change caused which failures. This was done by very senior developers that were familiar with all aspects of the system to be able to understand complex interactions between unrelated components of the system.

Using Multi-Stage Frequent Integration, each project works against their own team branch, merging changes from the mainline into their team branch every day or two, and doing their own build/test/cycle with just that team’s changes.

Thus, any failure must be a result of one of their changes. When the build/test cycle is successful, the team then merges their changes into the mainline. As a result, any team-specific problems are identified and removed prior to mainline integration and the only problems that arise in the mainline are those that are due to interactions between changes from multiple teams. The isolation also simplifies the job of figuring out what changes may have caused a problem because they only have to look at changes made by a single team, not the changes made by all developers.

Tests and code are written in parallel for each work item as that work item is started. They do not distinguish between enhancement requests and defects. All assigned work is tracked in their issue tracking system and all work that developers do is associated with a work item in the issue tracking system. This is done to automate compliance as they need to pass biannual SAS70 Type 2 audits and annual Payment Card Industry (PCI) audits.

The PCI audits go smoothly. According to Dave: “You have to do them yearly, we’ve been doing them for 5 years. We’ve never had a problem. If you’re doing development the right way, you’re not going to have a problem with audits, because all they’re checking is that you follow the best practices of software development. You must use source code control, track why changes are made, ensure that the work done aligns with the company’s goals, and that your app meets data security requirements. On the auditors’ first visit, they said passing it was going to be a big problem because of our size (they figured, ‘small’, meant insufficient), and that we would likely have to make a lot of changes to pass. When the audit was complete--the auditors were surprised. They said they had never seen anyone do this good of a job the first time out.”

Dave also sees their long track record of successful use of Agile as a competitive advantage: “Because we have our monthly releases, our sales people are able to work with our product people to adjust the company priorities for what makes the most sense for the business. Rather than having to pick everything we need to fit in for the year now and then get together again in a year, it becomes a regular monthly meeting where they look at the priorities together. So if there is a vertical we’re not in yet or there’s a vertical that we are trying to expand in or there’s something that makes sense for the business, if we decide as a company we should go after that business, we bid on it.”

More Information
I look forward to any questions about the case study. By the way, if you are an AccuRev user, keep a look out for your invitation to the local user group meeting. David Tarbox will be speaking about his experiences using AccuRev for Agile Development.

In the meantime, you may be interested in learning more about Litle & Co., Multi-Stage Continuous Integration, or AccuRev.

If you would be interested in participating in an Agile case study for the book, please contact me.

TOC: Zero to Hyper Agile in 90 Days or Less

Upcoming Conference Sessions

2008-05-15T07:35:00.004-04:00

Agile 2008 - August 4th - 8th
I will be doing three sessions at Agile 2008 in Toronto. If you are going, I hope you will consider attending one of my sessions. In any case, I look forward to meeting you there. Agile 2008 promises to be a truly great conference. The planning process was amazing with lots of great discussion of potential proposals.

Here are the three sessions that I will be doing:

Questioning the Perception of Agile (Full Listing)
There are many misperceptions of Agile. This session is mainly for Agile advocates. It is a facilitated discussion about common misperceptions/misconceptions and ways to alleviate them.

Calling All Agile Skeptics, The Curious, and Die-Hard Non-Agile (Full Listing)
If you are looking for a place to either air your skepticism/questions or respond (helpfully) to skeptics/questions, this will be a facilitated forum for both. Please bring an open mind, your sense of humor, and at least one anecdote to share. Participants will be expected to share the floor and help to keep the atmosphere fun and relaxed.

See Large Scale Multi-Stage Continuous Integration in Real Time (Full Listing)
Multi-Stage Continuous Integration is a way to scale CI to larger teams as well as a way to make CI more effective for teams of any size. For a preview, I have a series of posts on the topic. This session will show a real-time environment using AccuRev, Cruise Control, and Jira with hundreds of scripted developers resolving issues, making changes, causing builds to fail and pass, etc.

SD Best Practices, October 27th - 30th

Introduction to Multi-Stage Continuous Integration (Full Listing)
Multi-Stage Continuous Integration is a way to scale CI to larger teams as well as a way to make CI more effective for teams of any size. For a preview, I have a series of posts on the topic.

Agile Development Practices, November 10th - 14th
A full list of sessions is TBD soon, but all other details are available now and registration is open.

Introduction to Multi-Stage Continuous Integration (Details TBD)
Multi-Stage Continuous Integration is a way to scale CI to larger teams as well as a way to make CI more effective for teams of any size. For a preview, I have a series of posts on the topic.

Calling All Agile Skeptics, The Curious, and Die-Hard Non-Agile (Details TBD)
If you are looking for a place to either air your skepticism/questions or respond (helpfully) to skeptics/questions, this will be a facilitated forum for both. Please bring an open mind, your sense of humor, and at least one anecdote to share. Participants will be expected to share the floor and help to keep the atmosphere fun and relaxed.

The Iterative Design of a Lego Sports Car that Transforms Into a Robot, Part II

2008-05-09T14:17:00.018-04:00

In Part I, I talked about how the design of a Lego car that can transform into a robot illustrates the process of iterative design. At the beginning of the project, I had the exact requirements and they didn't change at all from the start of the project to the end. However, just having the right requirements did not mean that I could immediately design something that would meet the requirements. At the end of Part I, I had a Lego car that met many of the basic requirements, but still didn't transform. I was stuck because I didn't have the technology that would meet the rest of the requirements.

Part of iterative design is keeping your eyes open, always looking for ways to intelligently expand the solution space. If the information you need is not already at hand, trying variations of things you already know may not lead to a good solution.

Thinking Outside The Box
I had always eschewed the Bionicle Legos as “not really Lego,” so I hadn't originally considerd those. But then two things happened that breathed new life into my Lego project. My son’s third birthday was coming up, and my wife suggested that we make it a Lego birthday. I was responsible for filling the gift boxes for the guests, and I decided to start by visiting the Lego store to see what they might have.

The Lego store had a wide variety of gifts for $5 or less including Duplo dinosaurs, a pony and princess set, knights on horses, and most importantly (for the project) small Bionicle sets. I had never bought a Bionicle set before, so I had never noticed that many of them had articulated joints. The joints were ball-and-socket parts which were small and had enough friction to make them poseable. They were perfect for the project.

There is another kind of Lego set called Exo-Force which has a completely different technology for doing articulated joints and also has a part which when combined with a Bionicle part provides a third technology for articulated joints.

Creating limbs which can fully fold up is a difficult proposition at best. But, I managed to create a couple of variations that allowed me to create the next version of the project. It didn’t really meet the goals of looking like a sports car or hiding the fact that there was more to it than met the eye. It was quite boxy, had odd proportions, and some of the joints were exposed. It was also fairly fragile. But, it could transform!

Sometimes Flexibility is Constraining
Sometimes software that is too generic or two flexible creates another set of problems such as taking too many resources or being too difficult to configure for the task at hand. The knee and elbow joints of the robot form had the same problem. Because they were fully articulated they were just too darn big and difficult to hide. I knew that these joints didn’t need to have the same degrees of freedom as the shoulder, hip, and ankle joints, but had not previously been able to create a hinge that could fold flat and have the full range of motion required. I repeated the exercise of determining all of the available hinge options and found a couple of new ones. The most promising one was a dual Technic hinge which was perfect for the knees.

For the arms, this hinge was too large. But then I realized that less resistance was required for the arms and a single hinge would do. Perfect! Now I had the robot skeleton, I just needed the car exterior.

Inspired by Elegant Design
Many months later, when looking for a new Lego set for my son, I saw a new car set (#4939) that was perfect for the project. I’m not an artist by any means, so one of the challenges has been finding Lego car designs to emulate. Previously they had always been either too big or too small. At one point I had been very excited about the Ferrari that comes with both red and yellow exterior pieces, but it was double the size I needed. Set #4939 seemed to be the perfect size. It didn’t hurt that it was very cool looking and reasonably priced. I had the feeling that this was it, I was finally going to achieve the goal.

Using set #4939 as inspiration, I was able to layer a car exterior onto the robot skeleton without too much trouble. As a bonus, the head, which had previously never really worked out well, was just there. I was within reach of the goal. But still one problem remained. I couldn’t cover the shoulder and hip joints without preventing transforming. I needed something that would allow the top and bottom parts to slide out and then lock in place. I had created parts that slid from place to place before for other Lego projects, but never something that could also lock in place.

Sliding Into Home
After trying many variations using small hinges to act as the locks, I had something that worked, but it was just too big. It was also very clumsy. Luckily, I was in denial. I forged ahead anyway. I wasn’t sure what to do about it being clumsy, but I was determined to create more room in the model. The most obvious step was to do what they do to create stretch limousines: take a regular limo and stretch it out. I made the whole model a bit longer, wider, and taller while keeping the same overall proportions. I also removed all unnecessary internal parts or replaced them with parts that would still do the job but took up less space.

It wasn’t enough, and I couldn’t stay in denial about the sliding part being clumsy forever. But while I was stretching things out I observed that some of the technic parts would slide around a little bit when I exerted too much pressure on them. Of course, the part that would give depended on the amount of friction. Whichever had less friction was what moved and the other one did not. From that came the solution: two bricks together to anchor the axle, and one brick that moved freely but with enough friction that it would stay wherever it was slid to.

The slider was the last major challenge. Once I had this part, it was just a matter of trial-and-error shifting things around until it all worked. There are two sliders; one for the lower body, and one for the upper body, the following picture shows the car with the upper and lower body pulled out as the first steps of transforming.

After the car has been extended, the legs are unfolded.

Lastly, the arms are extended. The head is also fully articulated.

There is still room for improvement, but at this point I think it is more a matter of aesthetics than mechanical design. I’m happy with the current design and hope that perhaps a Lego fan with more art skills than I will take the design to the next step. My other hope is that perhaps the amazing folks at Lego will start producing transforming Lego sets. And of course, if they were Transformer branded, that would be a wonderful (but not necessary) bonus.

[Get the LDraw model for the car here . ]

Motivation, skill, and experience all play important roles in the design process, but they are no guarantee that you are going to be able to design, let alone build what you have set out to do. It almost always requires trial-and-error, serendipity, creative inspiration, research, and an open mind.

Next: Frequent Releases Improve Code Quality Faster

The Iterative Design of a Lego Sports Car that Transforms Into a Robot, Part I

2008-05-01T14:24:00.020-04:00

In the lead-up to the debut of the movie “Transformers,” there was already plenty of Transformers merchandise available. My son, who was 2 ½ years old at the time was instantly entranced. While my wife was strolling him through the bookstore he grabbed a Transformers calendar without her noticing. When she got to the register he then used his charms to convince her to buy it.

The Challenge
This got me to thinking. Wouldn’t it be fun to build my son a car out of Legos that could transform into a robot? After all, how hard could it be? I decided to give it a shot and I set myself the following design goals and constraints:

Build a Lego model that has two forms: a sports car and a robot
Folks looking at it agree that each form is clearly recognizable as that form
On casual inspection, it doesn’t look like it transforms
100% Lego, no other parts, and no glue
Easily held in one hand
Strong (parts don’t fall off easily during use)
Robot form can stand on its own
The model can easily transform from one form to the other
Model is completely connected and transformation does not require the addition or removal of any parts
Fully articulated

Now, a year and several generations of Lego models later, I've finally created a Lego model which meets these deceptively simple goals.

As the project progressed, I wasn’t thinking of it as an exercise in iterative design or a formal project, it was always just something I did for fun in my spare time. I didn’t realize the link to iterative design until I started talking to people about the history of the project.

Transforming Requirements Into Working Software
When working with software, even if you have clear requirements that you have validated with users, it is not always clear how you will implement them. The search for a solution has two possible components: thinking about how to do something and implementing something. When thinking about how to do something, that’s really just fantasizing. Of course, it is fantasizing that is focused on producing something real, and it is more likely to produce something real than fantasizing about winning the lottery, but it is still fantasizing.

Requirements, analysis, specifications, and design are all examples of fantasizing. Yes, it is useful to do it, and you will often hit on solutions or eliminate problems much more quickly and much more economically than creating something real without giving it the same amount of thought, but fantasizing is only an approximation of reality, an interlocking web of educated guesses. Not until you actually attempt to create something real will you find out how all of the individual requirements interact and the real costs and difficulty involved. You don’t always find a clear match between requirements and implementation that is cost-effective to implement. In this case you have four paths forward: implement with the closest match and hope for the best, search for new possibilities, invent new possibilities, or shelve the project.

Transforming Lego Car, Iteration 1
The first attempt to build the transforming Lego car didn’t go very well. I was able to create a reasonable looking car which had lots of interior space for the mechanics needed to transform, but creating the sliding parts and the joints proved to be more difficult than I had imagined. I had lots of different hinges, flat parts, special joints, and Technic parts, but all of the possibilities I came up with were either too big, too fragile, too loose, or didn’t support articulation well enough. I was stumped. After a quick trip to the nearby Lego store to see if there was something I had missed, I reluctantly decided to shelve the project.

From a requirements perspective, I had satisfied requirements 2-6, which is half of the requirements. Of course, it was easy to satisfy "doesn't look like it transforms" because it didn't transform at all. :-) From an Agile perspective, the car was fully functional as a car in its own right, and I had discovered lots of valuable information that would help with later iterations of the car.

Little did I know that a hidden bias had kept me from seeing what was right in front of me. It wasn't until a couple of months later that serendipity broke through that bias and brought together the necessary ingredients for the first transforming version of the car.

Next: Part II (of 2)

Applying Usability to the Software Development Process

2008-04-24T10:14:00.003-04:00

In order for something to become widely adopted, it has to provide an advantage that is significantly greater than whatever it replaces, and the advantage has to be easily accessible to a mainstream audience. Another way to describe “easily accessible” is to use the term “usability.” That which has high usability is more likely to become mainstream.

There’s a lot of literature on the usability of both software and everyday objects. But what does usability mean when applied to the process of software development itself? If we think of the myriad steps we use to produce software as the implementation of a software development process, then we can think of that implementation as software and consider its usability.

To consider the usability of a particular process, we first have to have a measuring stick for usability. Here’s a set of categories loosely based on some of the chapters from Joel Spolsky’s excellent book “User Interface Design for Programmers” and also adding in “visibility.”

Feedback
We get feedback our whole lives. When we first come into the world, we need constant feedback in order to learn how to survive. Touching something hot is painful, going without food is painful, falling down stairs is painful. In school we get feedback as to which subjects we are good at and which subjects we are not so good at. We can use this feedback to decide what to do more of, what to avoid and as an aid to improve, if we are able to.

The software development process, along with its rules, is just one process that you are a part of. Process, rules, and laws are all around you. You don’t have to consciously think about them but you are constantly reacting to them and they are second nature to you. On your way to work you drive on the correct side of the road, you stop at stop signs, and you stay in your own lane. When you go to lunch, you pay for your food. After work you go to your house instead of going to somebody else’s house.

It is easy to remember the laws, rules, and processes because you interact with them every day and you get feedback in the form of reminders and possibly even unpleasant consequences when you go astray. A great example of this is the rumble strip that has become commonplace on the side of highways. If for some reason you start to drift off the road, you’ll hear a loud warning noise as your tires hit the rumble strip. If you are really violating the law, you’ll hear an even louder sound accompanied by flashing lights. This is also known as feedback.

With traditional development, the development timeframe is much longer and thus the feedback loop is much longer. With Agile, the feedback loop is usually no longer than a month and often on a weekly basis.

Consistency
An important part of absorbing all of these laws, rules, and processes is that they are clearly defined, easy to understand, and easy to remember. For instance, once you have mastered basic physical laws such as the fact that running at full speed into a door is painful, it is easy to understand the reasoning behind having an upper limit on the speed of vehicles on public roads. You may disagree with that limit, but you can generally guess what the limit is for a given stretch of road and it is posted at regular intervals to help you score your guesses. Sometimes there is even a referee.

In traditional development, the long timeframe makes it difficult to absorb what the actual process is and usually everybody has a different mental model of that process. When people have different mental models for the same process, bad things happen. In Agile development, the process is far simpler and much easier to absorb. You have a chance to see the full process from start to finish on a very regular basis.

Expected Behavior
When something happens unexpectedly, it interrupts our train of thought and slows us down. It is a little bit like taking a wrong turn and becoming lost. We have to take time to figure out where we are and how to get back to familiar territory. Most of the time, when working on a software project we are concentrating on a single task that we alone are responsible for.

Consider software development from your own perspective. I’ll bet that you identify with most or all of the following statements:

When working on a software project on your own, you make changes and use the new version right away.
One of the reasons that you got into software development is the fact that you can make a change and see the result right away.
When you have something new working for the first time, you want to show it to somebody.
It isn’t easy to create something that the customer thinks is exactly right the first time.
Even when creating something for yourself, it isn’t easy to create what you want the first time.

Most software projects are part of a larger scope, even if we are the sole developer. Somebody has to test it, and there are usually organizational standards that must be adhered to. In any case, you will spend a fair amount of time developing according to the rules of the organization (“the process”) rather than the way you would do it naturally if it was an individual project just for yourself. Whenever the way that you would develop software naturally conflicts with “the process,” the more you have to slow down and think about the correct next step and the greater the usability problem.

With traditional development, there is generally much more process to cope with the greater complexity of trying to develop a much larger increment of functionality over a much longer time period. Because of the large gap between how one would develop software on one’s own and how one develops software as part of a much larger process, the steps in the process are often counter-intuitive and thus it is much harder to establish a natural rhythm.

Visibility
It is hard to know what to do next if you aren’t sure where you are. Visibility is akin to project status. Do you feel like you know exactly where you are and what to do next at every step of the process? Or do you feel pulled in a million different directions at once and lose track of what you’ve already done and what you need to do next? If you do feel like you’ve finally “got” how software is developed in your organization, how long did it take to get to that point?

In a traditional project, it is almost impossible to see at a glance what the real project status and progress are. You know that you won’t really know until sometime after code freeze. In an Agile project, the next milestone is much closer, so it is much simpler to see where you are. Granted, if your real goal is six iterations out, you don’t have the same sort of visibility into the next five iterations, but the chances of risks hiding out until code freeze is substantially reduced.

The Usability of the Software Development Process
Based on just these few criteria, it is clear that traditional software development suffers from poor usability and hinders the ability of otherwise competent people to get their work done. While it is true that people can succeed on projects that have very long timeframes and involve a high degree of stress at the end, that mode of working does not align well with our strengths as human beings.

People are much more productive and much less prone to error when they work at a constant and sustainable pace. This also means that when there are unforeseen circumstances, people are more likely to be able to respond well.

Agile Development provides frequent feedback via short iterations, a simple process which is more in keeping with natural human rhythms and capabilities, and significantly more visibility into project status and progress. This creates a highly usable and people-oriented software development environment.

Next: It is Not the Bug That Bends, it is Ourselves

How Agile Works in a Nutshell

2008-04-23T17:07:00.007-04:00

Yet Another Fad
We’ve heard it all before. The Segway, UML, Virtual Reality, TQM, object-oriented databases, Artificial Intelligence. Agile’s no different, right? Just the latest fad? That’s what I thought.

Back in 2005 at the SD Best Practices conference, I had my first run-in with Agile development. At a speaker’s reception, my blissful ignorance of Agile came to an end. I was surrounded by Agilists. They looked perfectly normal to me, but then they started to talk about the advantages of keeping track of requirements on 3x5 cards, pair programming, and lots of other crazy sounding stuff.

3x5 cards! Are you kidding me? What is this, the seventies? How can people who make a living writing software advocate the use of 3x5 cards? And what about pair programming? Two people sitting together all day sharing the same keyboard and mouse switching between coding and sitting on their hands, having to hear phone calls about the results of medical tests. Yuck!

I wasn’t just skeptical, I was outraged that all these people were trying to push this nonsense. Keep in mind that I work at a company that provides software tools to companies that are looking to improve their process. On the surface, Agile seems to be diametrically opposed to that. I felt that it was time for somebody to step forward and do something about it. I nominated myself. To gather evidence I read the books, talked to the experts, and attended the seminars. But then one day a funny thing happened. I had an aha moment and I realized there might actually be something of value buried under all of the rhetoric.

To illustrate what I realized, let's compare traditional development and Agile development using a typical development scenario.

Next: Traditional Development Scenario

New Reddit for Agile Development

2008-03-29T11:44:00.002-04:00

Reddit is a great way to surf the net for things that you are interested in, and you can participate in determining what's good and what's not. Now you can surf using reddit for all things Agile! Check out the new Agile reddit!

Is Your Dev Team Having Performance Problems? Try Niagra!

2008-03-26T15:03:00.003-04:00

Have you ever thought “why do the companies I work at have so many problems with developing software?” Well, I can tell you from interacting with literally thousands of software development organizations that most development organizations have problems with reliably and predictably producing high quality software. What I hear over and over again is “if only we could hire the right people” or “if only people could be more disciplined.” The other thing I hear over and over again are various tweaks to traditional development that people believe will fix things but that “other people just don't get it.”

Well, there are only so many people out there to hire so unless we find some magical place to hire more of “the right people” from, we are going to have to figure out how to use the people we have. And I hate to break it to you but you just aren’t going to get more discipline out of people. Whatever discipline that exists today, that’s all you are going to get. So what’s left? Changes to the way we do things.

Let’s say there was some combination of techniques which would mean that most development groups could use traditional development to reliably and predictably produce high quality results. Let’s call it the “Niagra” method. Furthermore, you (or whoever wants to make this claim) get to define the Niagra method. It works every time. Just use Niagra and performance improves overnight (when used as directed).

If the Niagra method existed, it would have become widespread by now. It would have become the #1 prescribed solution to project performance problems. If it existed and produced the claimed results, people would start referring to it, and it would spread rapidly. That’s how C++, Java, and HTML became mainstream. People rapidly adopted them because they provided benefits that anybody could realize. In fact, there are many proposed Niagra methods, but none of them have become mainstream because they only work under special conditions.

This raises an obvious point. If traditional development is so bad and is such a big failure, then why has the software industry done so incredibly well? The software industry is an incredible success, that’s absolutely true. It is incredible how much of our daily lives runs on software and how much software has positively influenced our quality of life. The benefits of software, when it does finally ship and it does finally have a high enough level of quality are worth waiting for. Otherwise, of course, there would no longer be a software industry.

There have been incremental improvements to the software development process which have produced incremental improvements and have become widespread. Examples are: the use of software development tools such as source control and issue tracking, breaking down large tasks into small tasks, nightly builds, one-step build process, moving from procedural programming to object-oriented programming and many others. The important point here is that each of these improvements have become widely adopted and made things somewhat better, but still didn’t solve the root cause. The process is still unpredictable and unreliable.

This is akin to the O(n) approach to problem solving. If you have an algorithm which takes 100*n^2 operations, then getting that down to 50*n^2 is a good thing, but changing the algorithm to (for instance) be 200*n is much better. To date, changes to the software process have been of the first variety, shrinking the constant.

Perhaps Agile development is the answer. But isn’t Agile “yet another fad” which will blow over any day now? After all, we have had CMM, 9001, TQM, Critical Chain, and many other “next big thing” ideas in the past. What makes Agile any different? Previous ideas have all been centered around the idea that the framework underlying traditional development is fine, it is the implementation, discipline, people, or management of the people that is the problem. No, it is the underlying framework that is the root problem.

Certainly, people issues are important. Getting a team gelled and working well together with the right combination of skills is essential. But the success rates of these teams isn’t exactly stellar either! Yes, it is better, but they have similar problems with predictability and quality. A root problem remains. I do not claim that solving this problem means that suddenly teams that have insufficient skills and hate each other will start pumping out high quality software on a regular basis. But I do claim that Agile development has the ability to dramatically increase the potential of any team. And to truly succeed you have to do both: create a good team, and use Agile development.

Let's take a deeper look at the fundamental problems with traditional development.

Next: Traditional Development is a Chinese Finger Trap

Apply Elegant Architecture to Your Dev Team: Part II

2008-03-25T23:44:00.004-04:00

Previous: "Apply Elegant Architecture to Your Dev Team: Part I"

When you are creating new software, what do you think about? Don’t you think about how it will scale to meet your needs as they grow with high availability? Even if you don’t achieve that on the first try, you are still thinking about it and striving to achieve it. You know that to do it, you will need to make the right technology and architectural choices. Over time, you may need to change some of those choices to keep pace with competitors. Even if your current needs are modest, software development architecture has evolved technologies and patterns to allow software to scale from a single user to hundreds or even thousands of users on multiple platforms at multiple locations with 99.999% uptime.

If you consider your development organization in the same way, how would you apply the same thinking? What technologies are you using? What is the architecture of your organization? Will it scale from its current size to double its size? Will it scale seamlessly to include new teams in new locations? What will happen if you acquire a company? When creating software, you want to design it so that it is flexible and adapts to new circumstances. The same should be true for your development organization.

Another way to look at it is how a particular process would fare if each of the resources available were available at seemingly random times for random periods of time. This exactly describes the world of Open Source Software (OSS). At any given time you have no idea who will be contributing on what or how valuable the contribution will be. You don’t know where the contribution will come from and in many cases you don’t even know who the contributor actually is. This is an extreme example of a development situation. Even though your situation is probably not as extreme as this, by using techniques from the OSS world you will be better positioned to handle unexpected events when they inevitably occur.

Problems are an inevitable and regular part of life. Examples include illness, job change, human error, flight delays, system failure, unanticipated market changes, and natural disasters. The ability to cope with these problems is one measure of the robustness of the organization.

Part of robustness is that as things scale up or down, the impact is minimal. Tools and processes should be selected and designed to work well together whether they are used by 1 person or 10,000 . At all times, it should always feel like each individual is part of a team which is no bigger than 12 people. If practices are not scaleable from 1 to 10,000 then people can develop bad habits that resist scaling. If you develop habits that exist in a scaleable framework, then it is more likely that scaling can and will happen as and when needed.

Next: How Agile Solves Problems