Sunday, December 17, 2006

SCM vs Software Development

I came to the realization a while back that I'm not really an SCM person. In fact, I'm a software development process person who has been focusing on SCM. That's probably why most of my posts lately can't be directly linked to SCM. I've been working on broadening my thinking. As a result, I've decided to change the focus and title of this blog from SCM to Agile Software Development.

I think there's a lot of value to be realized using top-notch SCM tools like AccuRev, ClearCase, Perforce, and others. But I've come to the conclusion lately that a lot of the value of these tools either sits on the shelf as potential value or goes unrecognized and unappreciated. I think there are two reasons for this. First, I think that most people think of SCM tools as a repository and nothing more. Second, most SCM tools are optimized for being repositories and nothing more.

To put it another way, I've decided to turn my emphasis upside down. I don't think that most developers show up for work thinking about how best to optimize their branching and merging. So, instead of starting there and ending up at the developer benefit only to find that the audience has surfed to another page, I've decided to always start from the developer's point of view and link the discussion back to SCM only when necessary.

This also gets to another point. SCM folks by and large already recognize most of the value of SCM tools. That's what they do all day! Whereas, developers prefer to just write code all day and keep their interaction with an SCM tool to a minimum. That is as it should be. As a result though, I don't think that developers are getting the full value out of their SCM tools. I'm not saying that developers are thick, just that their focus is on writing code, not playing with SCM tools on the off chance that they might stumble across something that will increase their productivity.

Thus, it is up to folks that do think all day about how to improve developer productivity through better SCM tools to make the extra effort to pitch things from a developer's point of view so that when a developer does come up for air they see right away that they are being offered a lifeline and not a lead weight.

Thursday, December 14, 2006

First Hyper Agile Project is Now Complete

The project that I mentioned back in October, now called AccuWorkflow, has just been announced this week! You can also learn more about it here. While the connection to Agile Development may not be that clear yet, this is just the first release.

I believe that for Agile to succeed it must be supported by more tools and it must scale to larger teams. Further, short iterations will actually require much better defined and much more automated process. To that end, AccuWorkflow, a new add-on to AccuRev makes it much easier for teams to establish and automate their desired process across the entire software development life cycle regardless of methodology.

For this project we used an Agile methodology that I call "Hyper Agile." The key ingredients of Hyper are:

Short Iterations
Automate Everything
Virtual Pipelining
Source Control
Issue Tracking
Development Hierarchy
Quality Quotient

For more information on the role of short iterations in Hyper see this article. In future updates I'll elaborate on other aspects of Hyper and the experience of using it in the development of AccuWorkflow.

Streams for Large Scale Development Projects

Joel Spolsky recently did a posting on large scale development and mentioned AccuRev!

Wednesday, October 25, 2006

Agile Adoption

A comment from "AMWOP" states: "Unfortunately while progressive developers may want to work in an Agile fashion the ecosystem around them is frequently fighting against being Agile. Executives, Customers, Testers, Business planners and many other stakeholders insist on detailed, committed plans up front which give them a (false) sense of confidence about viability of long range plans for software delivery. This in fact drives the developers back towards waterfall methodologies. It won't be until these functions change their mindset that Agile processes will make significant inroads in commercial environments."

I wholeheartedly agree. As with most things, adoption takes time. This is a classic example of crossing the chasm. Currently, Agile is in the visionary and early adopter phase. The software development habits that people have now took years to form. It will take many more years before they change.

In addition, the infrastructure for widespread adoption is not yet there. There are lots of books that you can buy, but they are mostly oriented towards the early adopter. On the software development tools side, most are still oriented towards waterfall methodologies. This is changing, but slowly.

This is no surprise. Change takes time. My prediction is that in 5 years time, we will drop the term "Agile Development" and just talk about software development because we'll no longer need to make the distinction between Agile and non-Agile. Instead, people will talk about how often you release because everybody will be doing short iterations. I also predict that Agile will evolve to the point that people stop saying that you might as well just throw up your hands on predictability because they will realize that Agile actually gives you the opportunity to be more predictable. More on that later.

Tuesday, October 17, 2006

Agile Development - It's Human Nature

The Agile Manifesto claims that the benefits of Agile Development come in part from "People over Process". That never sat quite right with me, but I could never put my finger on why. An obvious reason of course is that I work for AccuRev, a development tool vendor that specializes in process-centric SCM. So of course I must love lots and lots of complicated process that requires a tool to implement. Actually, I'm a big believer in simplicity and automation.

I believe that attitude, people, and culture have always been important to the success of any project. Therefore, I don’t agree that there is anything special about Agile when it comes to attitude, people, or culture. It may well be that current Agile projects are populated by risk takers and visionaries, but if that was what it required to succeed, then it would never become very popular.

In my opinion, all of the benefits of Agile Development can be directly attributed to the practice of short iterations. That said, simply changing your plans to release more often, say every month, won’t work. While it doesn’t matter how you get to short iterations, you do have to do things differently to get there and many of the practices associated with Agile Development are good candidates for doing it.

Over time, I’ve come to realize that there is in fact something special and people-oriented about Agile. Ironically, it is the process that makes it people-oriented. I believe that short iterations leverage human nature and thus allow us to fall into a more natural rythym in which we make fewer mistakes and we work more productively. While it is true that people can succeed on projects that have very long timeframes and involve a high degree of stress at the end, I believe that people are much more productive and much less prone to error when they work at a constant and sustainable pace. This also means that when there are unforeseen circumstances, people are more likely to be able to respond well.

Consider the following statements. I’ll bet that you’ll agree with most of them.

• When working on software for personal use, people make changes and then use the new version right away.

• One of the things that developers enjoy about software development is the fact that they can make a change and see the result right away.

• When working on a hotfix, people are more likely to cross all of the “t”s and dot all of the “i”s and take extra care to make sure that everything is done just right.

• It isn’t easy to create something that the customer thinks is exactly right the first time.

• People prefer to delay making important decisions as long as possible.

• Towards the end of a release when a new bug is found or a great idea for an enhancement comes in, it is tempting to just put it into the current release rather than the next release because the next release won’t happen for quite a while.

Each of these are a part of human nature. All of the positive parts are reinforced with short iterations, and all of the negative parts are reduced with short iterations. Conversely, long iterations work against the grain of human nature.

Wednesday, October 04, 2006

A Big Bang AccuRev Release

Some of my more recent thinking on software development, especially in regards to Agile Development, is in an article in the October issue of Queue Magazine, entitled "Breaking the Major Release Habit."

With that in mind I would like to mention that AccuRev 4.5 is now available. While it is called 4.5, it is actually one of our biggest releases ever. Here's a blog that mentions it. So why didn't we call it 5.0? Thankfully, that's not my department. :-) Anyway, my point is that while I believe very strongly in short iterations and more frequent releases, organizational change takes time and is hard to do.

To that end, I am currently working on a new project and piloting a new Agile methodology to develop it called Hyper Agile. The goal of the new project is to take AccuRev, which is already well suited to Agile projects, to the next level of Agile functionality. The aim of Hyper is to take a fresh look at Agile Development and scale it beyond small co-located teams. A preview of the Hyper methodology will appear in an upcoming issue of Dr. Dobb's, stay tuned!

Thursday, June 01, 2006

Agile: Turning on a Dime

If you've read any of my previous posts, the title of this post might make you think my blog has been hacked and somebody else is now posting in my stead. But actually, it is just that I had an epiphany. I still have the same beliefs as before, but I realize now that my (let's be frank) negative reaction to Agile is not actually directed at Agile itself, but rather at specific individual practices.

I've done lots more reading and thinking about development methodologies and have come to the following conclusion: the key benefits of Agile development derive from the Agile practice of short iterations. In my opinion, everything else is simply an enabler. For instance, writing tests first is a good way to make sure that you keep the quality of the frequent releases high. But, as long as you are able to consistently do short iterations, it doesn't matter what you do to keep quality high as long as you do it.

So if like me you don't like the idea of using 3x5 cards to manage the contents of a release, fine. Use whatever you want. Use a text file, use Bugzilla, use RallyDev, it doesn't matter. The method you use is inconsequential as long as you are doing short iterations.

The vocabulary doesn't matter either. I don't happen to like the idea of creating new words when there are perfectly good words that mean the same thing. I prefer to talk about issues rather than stories. As long as I'm using short iterations, that's fine, the benefits remain the same.

As to the benefits, I believe that the key benefits of Agile development are: shorter time to value and accelerated maturation of product changes. I say shorter time to value rather than shorter time to market because realizing value (via revenues ) is much more important than availability of new features.

Product quality is an ongoing effort. No matter how much testing you do, users will find ways to use your product that you didn't think of. For any change, whether it is a bug fix or a new feature, you don't really know the impact until customers start using it. That means that the maturation of a change doesn't really start until the change ships. If you have a long release cycle, you will have changes that are made early that are basically "sitting on the shelf" until you ship. You can't really say that they are tested until you test the whole product as it will ship, so again, true maturation doesn't start until you ship.

An important aspect of maturation is customer feedback. By using short iterations, you get customer feedback sooner. Instead of releasing a major new feature all at once, if you break it up into logical pieces (assuming that it can be done for a particular feature), you can find out sooner which pieces are taking the feature in the right direction and which are not and adjust your plans accordingly. For example, let's say that the perfect implementation of a new feature has 12 sub-components. Of course, you can't know in advance what that perfect implementation is. Using a traditional approach, you implement the first version in a year and it includes 12 sub-components. It turns out that 6 are exactly what was desired and 6 are completely wrong (even though that's what was asked for). So, a year later you release the second version and now you've got 9 out of 12 correct... and so on.

Using small iterations, you could have released a component a month, getting feedback once a month instead of once a year. Logically, you will converge on the correct solution much faster using a tighter feedback loop.

Another way to look at short iterations is as the turning radius of an organization. The shorter the iterations, the smaller the turning radius and the greater the agility.

Of course, agility is not the only important measure of an organization. Small teams may feel that their agility gives them the ability to outperform a larger team. It seems that Agile is primarily used by small teams today and so that's probably true today up to a point. However, I see no reason to believe that Agile is just for small groups. I believe that as Agile sheds the fetters of small-scale practices such as using 3x5 cards, larger and larger projects will adopt it and enjoy its benefits.

For reference, here is the reading material that steered me in this direction (so to speak) in the order that I read them:

Integrating Agile Development in the Real World, Peter Schuh
An excellent survey of most of the Agile methods.

Extreme Programming Explained - 2nd Edition, Kent Beck
Whether you agree with the idea of 3x5 cards or not, Kent's book is a short and easy read that is well worth the effort. A seminal book on Agile development.

Lean Six Sigma, Michael L. George
I read this book because I was interested in learning more about Six Sigma. However, I selected this book because the idea of "Lean" was appealing. The way that the problem and solutions for finished product sitting on a factory floor is explained immediately suggested parallels to finished features sitting in the source repository waiting for release.

Lean Software Development, Mary Poppendieck and Tom Poppendieck
These folks have done an excellent job of taking the lessons of Lean manufacturing, pioneered by the Toyota Production System, and translating them to software development. I read "Lean Six Sigma" first and on my next trip to the book store "Lean Software Development" became an obvious choice. Reading this book was the tipping point for me.

Agile Project Management with Scrum, Ken Schwaber
If you are already familiar with Scrum, this is a treasure trove of examples of how it has been applied in the real world.

Thursday, May 25, 2006

I Skipped 3x5 Cards and Went Straight To PostIt Notes

Previously, I wrote "After all, isn't the fundamental reason for software the automation of manual processes? Why in the world should software developers ever have to resort to keeping track of things with 3x5 cards or spreadsheets".

In response, in a comment, someone anonymously responded: "Software is used to replace repedative(sic) manual processes. In practice the use use(sic) of cards works well. To be blunt if you havn't(sic) tried XP with cards you are not qualified to comment on thier(sic) suitability."

Anonymous, your first sentence seems to be a restatement/validation of my first sentence. If you are trying to make a different point, I missed it. As for the use of cards working well, they work well in comparison to what? I will of course agree that using cards is better than nothing. I tried that prior to using issue tracking. I will also agree that using cards is better than trying to keep it all straight in my head. I also tried that. Ok, you've got me on the actual use of actual 3x5 cards.

I have to admit that I have never used 3x5 cards to keep track of stories related to software development. Though I did use them to keep track of role-playing characters at one point. Hmm. But the whole time I was wishing that they were on my computer. And I did actually try that, but storing the info on tape cassette and trying to find the exact place on tape was just too much of a manual process. These days, role-playing games are on-line, in 3D, and globally distributed. Things change fast.

But I digress. So, I admit I never used 3x5 cards in software development. I skipped that technology and went right to post-it notes. I had actually been hoping to gloss over that period in my career, but now you've forced it out of me. And though I'm smiling as I write this, it is in fact true that I used post-it notes.

That was actually a very short-lived adventure when I was working on an early prototype of what later became AccuRev. Within weeks we had moved on to storing everything in a text file. Here's a snippet:

O --- M -- - special case of merging a defuncted element 4h
C 502 E D3 - be forgiving when principal is included in ws name

I don't remember the format any more, but the terse description was the "story" and there was usually an estimate, such as "4h" for 4 hours. Many of them even had bug numbers. It was primitive, but it was a big relief getting off of the post-it notes. At least with the text file we could send it around, make changes, do diff/merge, search, create scripts to do reports, etc. We even had a script that created a web page for remote viewing.

Of course it wasn't long after that we realized that we were writing yet another issue tracking system, so we installed a free one in a couple of hours, breathed another sigh of relief, and went back to working on our real goals.

In short, I suppose I can't comment on my experience using 3x5 cards. However, based on the application of logical thought to my above experience, I think I'll take a pass. Call me crazy for liking the ability to use powerful tools to edit, search, categorize, re-organize, report, track, manage, distribute, share, and do backups. Say that I'm too demanding to want to be able to do these things from my office, from an office on the other side of the building, from home, or from a remote site. That's fine with me. Just don't expect me to go back to post-it notes (or 3x5 cards) any time soon.

Here's another way to think about it. What sort of reaction would you get if you suggested to a user of whatever software that you produce that one of the new features they wanted would be implemented using 3x5 cards? What if you also suggested that they pay for that feature?

I'm not saying you should toss the 3x5 cards and shell out dough (though if you feel like it let me know :-) ). But for goodness sake at least use a text file, or Excel (or Open Office), or Bugzilla.

Wednesday, February 22, 2006

Leverage Your Core Competency

SD Times recently published my article on leveraging your core competency (free registration required).

In a recent posting, CollabNet is apparently attempting to reframe the article - which is fundamentally about leveraging an organization’s core competency - into a discussion of commercial software versus open source software. This is surprising because both CollabNet and AccuRev have business models that are based on a hybrid approach that involves a mix of commercial and open source tools.

AccuRev is a member of the Eclipse foundation, and a contributing member of the Eclipse ALF project which aims to provide a framework for software development tool interoperability. Furthermore, as an ardent believer in the advantages of good SCM, AccuRev salutes CollabNet’s efforts to advance the state of the art. We have a great deal of respect for the work that they have done with CollabNet Subversion and believe that CollabNet Subversion is arguably the best free SCM tool available today.

The central point of the article is that businesses should do a careful ROI analysis of the costs associated with maintaining their own homegrown SCM system (open source-based or not) in comparison with purchasing similar off-the-shelf functionality from vendors such as AccuRev and CollabNet. The subtler decision points are in the gaps between any given tool’s actual implemented capabilities and your full set of technology requirements.

Friday, January 27, 2006

What Exactly is a Stream Anyway?

From time to time, people ask "what is a Stream?" At this point pretty much anybody associated with software development has heard of branches, but streams are a relatively new concept which is similar to but at a higher level than branches.

Here's my .02 on streams, coming from the AccuRev point of view. First, I think that at some level the various SCM systems which talk about "streams" are probably all trying to achieve something fairly similar. That is, a "stream of development."

My initial exposure to "streams" was from hearing folks talk about "streams of development" independent of the SCM system that they were using (which did not have such a concept). The idea was that you had work which was towards a particular purpose, such as new development, maintenance, a team working together on a sub-project, etc. Each of these was a "stream of development".

I have also heard the terms "codeline", "development effort", and "line of development" used in the same context. At the end of the day, the folks which initiate these things (managers, business people, etc) don't really care how they are implemented, they just want to ask questions like "how is 4.0 coming along" and "are all of the fixes from maintenance in the latest release?" Somebody else then translates that into the appropriate queries, which may be in terms of branches, scripts, streams, or something else.

Prior to AccuRev, I found the need to tranlate somewhat mystifying. Why should there be any difference between the mental model of "streams" and the implementation model?

Thus, in AccuRev, the mental model and the implementation model are the same. Streams are the basic building block of the architecture. There are no branches or labels, just streams. There are streams for releases, streams for active development, streams for end users, etc. Each stream except the root stream is defined in terms of a parent stream and inherits everything from the parent (recursively).

So, if you want to do maintenance on the 4.0 release, you would create a new stream based on 4.0 . Through inheritance, it is the same as 4.0. The definition itself is all that is required. The definition is simply "4.0_maint" is everything in "4.0" plus all of the changes in "4.0_maint".

Since streams are first class objects, you can act on them directly. You can assign security attributes, lock them, define other streams in terms of them, compare them directly, do queries on them without having to understand how the streams themselves are composed, etc.

For the curious, I've written a whitepaper which describes AccuRev's stream architecture at an even deeper level:

And if you want to go even deeper than that, the basis of AccuRev's streams is TimeSafe: