Everything You Need to Know About Scrum

If you're interested in learning more about Scrum, but just don't have the time to read a book on the subject, you can quickly pick up the basics from this video produced by High Moon Studios.

Hyper Agile Project Now a Jolt Finalist!

AccuWorkflow, a new product from AccuRev that allows the use of a single workflow for both issue tracking and code development, has been announced as a Jolt Finalist! I think AccuWorkflow is pretty cool in its own right, but it never would have been nominated if it hadn't been for Hyper Agile. By following this methodology, we were able to do much more with less and produce a high quality product at the same time.

A Preview of the Hyper Agile Methodology

In their February 2007 issue, Dr. Dobb's has just published an article that I wrote on best practices for global development titled "Think Globally, Code Locally". It also happens to overlap with the Hyper Agile methodology. So, for a preview of Hyper Agile, check it out!

SCM vs Software Development

I came to the realization a while back that I'm not really an SCM person. In fact, I'm a software development process person who has been focusing on SCM. That's probably why most of my posts lately can't be directly linked to SCM. I've been working on broadening my thinking. As a result, I've decided to change the focus and title of this blog from SCM to Agile Software Development.

I think there's a lot of value to be realized using top-notch SCM tools like AccuRev, ClearCase, Perforce, and others. But I've come to the conclusion lately that a lot of the value of these tools either sits on the shelf as potential value or goes unrecognized and unappreciated. I think there are two reasons for this. First, I think that most people think of SCM tools as a repository and nothing more. Second, most SCM tools are optimized for being repositories and nothing more.

To put it another way, I've decided to turn my emphasis upside down. I don't think that most developers show up for work thinking about how best to optimize their branching and merging. So, instead of starting there and ending up at the developer benefit only to find that the audience has surfed to another page, I've decided to always start from the developer's point of view and link the discussion back to SCM only when necessary.

This also gets to another point. SCM folks by and large already recognize most of the value of SCM tools. That's what they do all day! Whereas, developers prefer to just write code all day and keep their interaction with an SCM tool to a minimum. That is as it should be. As a result though, I don't think that developers are getting the full value out of their SCM tools. I'm not saying that developers are thick, just that their focus is on writing code, not playing with SCM tools on the off chance that they might stumble across something that will increase their productivity.

Thus, it is up to folks that do think all day about how to improve developer productivity through better SCM tools to make the extra effort to pitch things from a developer's point of view so that when a developer does come up for air they see right away that they are being offered a lifeline and not a lead weight.

First Hyper Agile Project is Now Complete

The project that I mentioned back in October, now called AccuWorkflow, has just been announced this week! You can also learn more about it here. While the connection to Agile Development may not be that clear yet, this is just the first release.

I believe that for Agile to succeed it must be supported by more tools and it must scale to larger teams. Further, short iterations will actually require much better defined and much more automated process. To that end, AccuWorkflow, a new add-on to AccuRev makes it much easier for teams to establish and automate their desired process across the entire software development life cycle regardless of methodology.

For this project we used an Agile methodology that I call "Hyper Agile." The key ingredients of Hyper are:

Short Iterations
Automate Everything
Virtual Pipelining
Requirements
Source Control
Issue Tracking
Development Hierarchy
Estimation
Transparency
Meritocracy
Ranking
Quality Quotient

For more information on the role of short iterations in Hyper see this article. In future updates I'll elaborate on other aspects of Hyper and the experience of using it in the development of AccuWorkflow.

Streams for Large Scale Development Projects

Joel Spolsky recently did a posting on large scale development and mentioned AccuRev!

Agile Adoption

A comment from "AMWOP" states: "Unfortunately while progressive developers may want to work in an Agile fashion the ecosystem around them is frequently fighting against being Agile. Executives, Customers, Testers, Business planners and many other stakeholders insist on detailed, committed plans up front which give them a (false) sense of confidence about viability of long range plans for software delivery. This in fact drives the developers back towards waterfall methodologies. It won't be until these functions change their mindset that Agile processes will make significant inroads in commercial environments."

I wholeheartedly agree. As with most things, adoption takes time. This is a classic example of crossing the chasm. Currently, Agile is in the visionary and early adopter phase. The software development habits that people have now took years to form. It will take many more years before they change.

In addition, the infrastructure for widespread adoption is not yet there. There are lots of books that you can buy, but they are mostly oriented towards the early adopter. On the software development tools side, most are still oriented towards waterfall methodologies. This is changing, but slowly.

This is no surprise. Change takes time. My prediction is that in 5 years time, we will drop the term "Agile Development" and just talk about software development because we'll no longer need to make the distinction between Agile and non-Agile. Instead, people will talk about how often you release because everybody will be doing short iterations. I also predict that Agile will evolve to the point that people stop saying that you might as well just throw up your hands on predictability because they will realize that Agile actually gives you the opportunity to be more predictable. More on that later.

Agile Development - It's Human Nature

The Agile Manifesto claims that the benefits of Agile Development come in part from "People over Process". That never sat quite right with me, but I could never put my finger on why. An obvious reason of course is that I work for AccuRev, a development tool vendor that specializes in process-centric SCM. So of course I must love lots and lots of complicated process that requires a tool to implement. Actually, I'm a big believer in simplicity and automation.

I believe that attitude, people, and culture have always been important to the success of any project. Therefore, I don’t agree that there is anything special about Agile when it comes to attitude, people, or culture. It may well be that current Agile projects are populated by risk takers and visionaries, but if that was what it required to succeed, then it would never become very popular.

In my opinion, all of the benefits of Agile Development can be directly attributed to the practice of short iterations. That said, simply changing your plans to release more often, say every month, won’t work. While it doesn’t matter how you get to short iterations, you do have to do things differently to get there and many of the practices associated with Agile Development are good candidates for doing it.

Over time, I’ve come to realize that there is in fact something special and people-oriented about Agile. Ironically, it is the process that makes it people-oriented. I believe that short iterations leverage human nature and thus allow us to fall into a more natural rythym in which we make fewer mistakes and we work more productively. While it is true that people can succeed on projects that have very long timeframes and involve a high degree of stress at the end, I believe that people are much more productive and much less prone to error when they work at a constant and sustainable pace. This also means that when there are unforeseen circumstances, people are more likely to be able to respond well.

Consider the following statements. I’ll bet that you’ll agree with most of them.

• When working on software for personal use, people make changes and then use the new version right away.

• One of the things that developers enjoy about software development is the fact that they can make a change and see the result right away.

• When working on a hotfix, people are more likely to cross all of the “t”s and dot all of the “i”s and take extra care to make sure that everything is done just right.

• It isn’t easy to create something that the customer thinks is exactly right the first time.

• People prefer to delay making important decisions as long as possible.

• Towards the end of a release when a new bug is found or a great idea for an enhancement comes in, it is tempting to just put it into the current release rather than the next release because the next release won’t happen for quite a while.

Each of these are a part of human nature. All of the positive parts are reinforced with short iterations, and all of the negative parts are reduced with short iterations. Conversely, long iterations work against the grain of human nature.

A Big Bang AccuRev Release

Some of my more recent thinking on software development, especially in regards to Agile Development, is in an article in the October issue of Queue Magazine, entitled "Breaking the Major Release Habit."

With that in mind I would like to mention that AccuRev 4.5 is now available. While it is called 4.5, it is actually one of our biggest releases ever. Here's a blog that mentions it. So why didn't we call it 5.0? Thankfully, that's not my department. :-) Anyway, my point is that while I believe very strongly in short iterations and more frequent releases, organizational change takes time and is hard to do.

To that end, I am currently working on a new project and piloting a new Agile methodology to develop it called Hyper Agile. The goal of the new project is to take AccuRev, which is already well suited to Agile projects, to the next level of Agile functionality. The aim of Hyper is to take a fresh look at Agile Development and scale it beyond small co-located teams. A preview of the Hyper methodology will appear in an upcoming issue of Dr. Dobb's, stay tuned!

Agile: Turning on a Dime

If you've read any of my previous posts, the title of this post might make you think my blog has been hacked and somebody else is now posting in my stead. But actually, it is just that I had an epiphany. I still have the same beliefs as before, but I realize now that my (let's be frank) negative reaction to Agile is not actually directed at Agile itself, but rather at specific individual practices.

I've done lots more reading and thinking about development methodologies and have come to the following conclusion: the key benefits of Agile development derive from the Agile practice of short iterations. In my opinion, everything else is simply an enabler. For instance, writing tests first is a good way to make sure that you keep the quality of the frequent releases high. But, as long as you are able to consistently do short iterations, it doesn't matter what you do to keep quality high as long as you do it.

So if like me you don't like the idea of using 3x5 cards to manage the contents of a release, fine. Use whatever you want. Use a text file, use Bugzilla, use RallyDev, it doesn't matter. The method you use is inconsequential as long as you are doing short iterations.

The vocabulary doesn't matter either. I don't happen to like the idea of creating new words when there are perfectly good words that mean the same thing. I prefer to talk about issues rather than stories. As long as I'm using short iterations, that's fine, the benefits remain the same.

As to the benefits, I believe that the key benefits of Agile development are: shorter time to value and accelerated maturation of product changes. I say shorter time to value rather than shorter time to market because realizing value (via revenues ) is much more important than availability of new features.

Product quality is an ongoing effort. No matter how much testing you do, users will find ways to use your product that you didn't think of. For any change, whether it is a bug fix or a new feature, you don't really know the impact until customers start using it. That means that the maturation of a change doesn't really start until the change ships. If you have a long release cycle, you will have changes that are made early that are basically "sitting on the shelf" until you ship. You can't really say that they are tested until you test the whole product as it will ship, so again, true maturation doesn't start until you ship.

An important aspect of maturation is customer feedback. By using short iterations, you get customer feedback sooner. Instead of releasing a major new feature all at once, if you break it up into logical pieces (assuming that it can be done for a particular feature), you can find out sooner which pieces are taking the feature in the right direction and which are not and adjust your plans accordingly. For example, let's say that the perfect implementation of a new feature has 12 sub-components. Of course, you can't know in advance what that perfect implementation is. Using a traditional approach, you implement the first version in a year and it includes 12 sub-components. It turns out that 6 are exactly what was desired and 6 are completely wrong (even though that's what was asked for). So, a year later you release the second version and now you've got 9 out of 12 correct... and so on.

Using small iterations, you could have released a component a month, getting feedback once a month instead of once a year. Logically, you will converge on the correct solution much faster using a tighter feedback loop.

Another way to look at short iterations is as the turning radius of an organization. The shorter the iterations, the smaller the turning radius and the greater the agility.

Of course, agility is not the only important measure of an organization. Small teams may feel that their agility gives them the ability to outperform a larger team. It seems that Agile is primarily used by small teams today and so that's probably true today up to a point. However, I see no reason to believe that Agile is just for small groups. I believe that as Agile sheds the fetters of small-scale practices such as using 3x5 cards, larger and larger projects will adopt it and enjoy its benefits.

For reference, here is the reading material that steered me in this direction (so to speak) in the order that I read them:

Integrating Agile Development in the Real World, Peter Schuh
An excellent survey of most of the Agile methods.

Extreme Programming Explained - 2nd Edition, Kent Beck
Whether you agree with the idea of 3x5 cards or not, Kent's book is a short and easy read that is well worth the effort. A seminal book on Agile development.

Lean Six Sigma, Michael L. George
I read this book because I was interested in learning more about Six Sigma. However, I selected this book because the idea of "Lean" was appealing. The way that the problem and solutions for finished product sitting on a factory floor is explained immediately suggested parallels to finished features sitting in the source repository waiting for release.

Lean Software Development, Mary Poppendieck and Tom Poppendieck
These folks have done an excellent job of taking the lessons of Lean manufacturing, pioneered by the Toyota Production System, and translating them to software development. I read "Lean Six Sigma" first and on my next trip to the book store "Lean Software Development" became an obvious choice. Reading this book was the tipping point for me.

Agile Project Management with Scrum, Ken Schwaber
If you are already familiar with Scrum, this is a treasure trove of examples of how it has been applied in the real world.