New Reddit for Agile Development

Reddit is a great way to surf the net for things that you are interested in, and you can participate in determining what's good and what's not. Now you can surf using reddit for all things Agile! Check out the new Agile reddit!

Is Your Dev Team Having Performance Problems? Try Niagra!

Have you ever thought “why do the companies I work at have so many problems with developing software?” Well, I can tell you from interacting with literally thousands of software development organizations that most development organizations have problems with reliably and predictably producing high quality software. What I hear over and over again is “if only we could hire the right people” or “if only people could be more disciplined.” The other thing I hear over and over again are various tweaks to traditional development that people believe will fix things but that “other people just don't get it.”

Well, there are only so many people out there to hire so unless we find some magical place to hire more of “the right people” from, we are going to have to figure out how to use the people we have. And I hate to break it to you but you just aren’t going to get more discipline out of people. Whatever discipline that exists today, that’s all you are going to get. So what’s left? Changes to the way we do things.

Let’s say there was some combination of techniques which would mean that most development groups could use traditional development to reliably and predictably produce high quality results. Let’s call it the “Niagra” method. Furthermore, you (or whoever wants to make this claim) get to define the Niagra method. It works every time. Just use Niagra and performance improves overnight (when used as directed).

If the Niagra method existed, it would have become widespread by now. It would have become the #1 prescribed solution to project performance problems. If it existed and produced the claimed results, people would start referring to it, and it would spread rapidly. That’s how C++, Java, and HTML became mainstream. People rapidly adopted them because they provided benefits that anybody could realize. In fact, there are many proposed Niagra methods, but none of them have become mainstream because they only work under special conditions.

This raises an obvious point. If traditional development is so bad and is such a big failure, then why has the software industry done so incredibly well? The software industry is an incredible success, that’s absolutely true. It is incredible how much of our daily lives runs on software and how much software has positively influenced our quality of life. The benefits of software, when it does finally ship and it does finally have a high enough level of quality are worth waiting for. Otherwise, of course, there would no longer be a software industry.

There have been incremental improvements to the software development process which have produced incremental improvements and have become widespread. Examples are: the use of software development tools such as source control and issue tracking, breaking down large tasks into small tasks, nightly builds, one-step build process, moving from procedural programming to object-oriented programming and many others. The important point here is that each of these improvements have become widely adopted and made things somewhat better, but still didn’t solve the root cause. The process is still unpredictable and unreliable.

This is akin to the O(n) approach to problem solving. If you have an algorithm which takes 100*n^2 operations, then getting that down to 50*n^2 is a good thing, but changing the algorithm to (for instance) be 200*n is much better. To date, changes to the software process have been of the first variety, shrinking the constant.

Perhaps Agile development is the answer. But isn’t Agile “yet another fad” which will blow over any day now? After all, we have had CMM, 9001, TQM, Critical Chain, and many other “next big thing” ideas in the past. What makes Agile any different? Previous ideas have all been centered around the idea that the framework underlying traditional development is fine, it is the implementation, discipline, people, or management of the people that is the problem. No, it is the underlying framework that is the root problem.

Certainly, people issues are important. Getting a team gelled and working well together with the right combination of skills is essential. But the success rates of these teams isn’t exactly stellar either! Yes, it is better, but they have similar problems with predictability and quality. A root problem remains. I do not claim that solving this problem means that suddenly teams that have insufficient skills and hate each other will start pumping out high quality software on a regular basis. But I do claim that Agile development has the ability to dramatically increase the potential of any team. And to truly succeed you have to do both: create a good team, and use Agile development.

Let's take a deeper look at the fundamental problems with traditional development.

Next: Traditional Development is a Chinese Finger Trap

Apply Elegant Architecture to Your Dev Team: Part II

Previous: "Apply Elegant Architecture to Your Dev Team: Part I"

When you are creating new software, what do you think about? Don’t you think about how it will scale to meet your needs as they grow with high availability? Even if you don’t achieve that on the first try, you are still thinking about it and striving to achieve it. You know that to do it, you will need to make the right technology and architectural choices. Over time, you may need to change some of those choices to keep pace with competitors. Even if your current needs are modest, software development architecture has evolved technologies and patterns to allow software to scale from a single user to hundreds or even thousands of users on multiple platforms at multiple locations with 99.999% uptime.

If you consider your development organization in the same way, how would you apply the same thinking? What technologies are you using? What is the architecture of your organization? Will it scale from its current size to double its size? Will it scale seamlessly to include new teams in new locations? What will happen if you acquire a company? When creating software, you want to design it so that it is flexible and adapts to new circumstances. The same should be true for your development organization.

Another way to look at it is how a particular process would fare if each of the resources available were available at seemingly random times for random periods of time. This exactly describes the world of Open Source Software (OSS). At any given time you have no idea who will be contributing on what or how valuable the contribution will be. You don’t know where the contribution will come from and in many cases you don’t even know who the contributor actually is. This is an extreme example of a development situation. Even though your situation is probably not as extreme as this, by using techniques from the OSS world you will be better positioned to handle unexpected events when they inevitably occur.

Problems are an inevitable and regular part of life. Examples include illness, job change, human error, flight delays, system failure, unanticipated market changes, and natural disasters. The ability to cope with these problems is one measure of the robustness of the organization.

Part of robustness is that as things scale up or down, the impact is minimal. Tools and processes should be selected and designed to work well together whether they are used by 1 person or 10,000 . At all times, it should always feel like each individual is part of a team which is no bigger than 12 people. If practices are not scaleable from 1 to 10,000 then people can develop bad habits that resist scaling. If you develop habits that exist in a scaleable framework, then it is more likely that scaling can and will happen as and when needed.

Next: How Agile Solves Problems

Apply Elegant Architecture to Your Dev Team: Part I

There is another way to think of software development which allows you to leverage your skills as a developer and apply them in a new way. The process of developing software is, in effect, an algorithm implemented with various technologies. You can think of your team, the technologies you use, and your development methodology as a piece of software.

What do you do with software? You improve it. You add new functionality, you increase its performance and usability and you remove bugs. Also, software itself is a document: source code. It is a description of how to perform a set of tasks. You improve the software by changing the source code. Let’s call the combination of your team, tools, and techniques: “the process” and the source code for the process “the process document.”

Not only can you think of your development process as software, you can think of your whole development organization and the people in it as a combination of hardware and software with various communication links. I don’t recommend that you take this advice literally and think this way on a regular basis, or treat people as interchangeable cogs in the machine! However, by thinking about it this way you can leverage your technical design skills to think about how to organize and optimize your development organization and development process. You can leverage well-known design patterns. For instance, consider the communication aspect.

In the world of information transfer there is bandwidth, latency, and connectivity. The best environment for communication is high bandwidth with low latency that is always connected. The worst environment for communication is low bandwidth with high latency that is infrequently connected.

This gives a well-known context for discussing human interaction. On one end of the spectrum is self-communication. If you are responsible for two interdependent modules, then when you make a change to one, you instantly know you need to make a change to the other. You won’t misunderstand yourself or need to have a back and forth conversation to really understand. You just know. On the other end of the spectrum you have two people from different cultures revising a document via e-mail who live exactly half way around the world from each other. In between you have pair-programming, collocation, people working together but sitting on different floors of the same building, folks with a great deal of physical separation in the same time zone, a different time zone, etc.

Then there are different forms of information interchange such as video link, phone, e-mail, IM, wiki, document, etc. This way of thinking can guide your decisions about where to seat people, the value of having high bandwidth links, etc.

Next: Apply Elegant Architecture to Your Dev Team: Part II

Is Your Software Development Organization Mainstream?

Have you ever wondered how your organization compares to other organizations when it comes to the process of developing software? Do you feel like there is an area that really needs improvement, but everybody else says "that's the way everybody does it?" While looking at what is involved in mainstream Agile adoption, I realized that it would be useful to create a list of the current mainstream software development practices for comparison purposes and I thought other folks might find it useful as well.

I define a practice as being mainstream if it is a practice that most people would agree that most people do. It may not be that they do that exact practice, they may do something equivalent or further along the spectrum of what constitutes a good practice, but they at least go to the level described.

The word "practice" is the key word here. This is about what people can be observed as actually practicing in their day to day real work. Things which are described by policy or documented as the correct way of doing something but avoided and worked around don't qualify. A mainstream practice is something that is in common usage that people do because they believe the value is worth the effort and not doing it is sort of like saying “electricity and hot and cold running water aren’t for me, I much prefer candles and fetching water from the well.”

While each of the following individual practices is considered a mainstream practice, that does not mean that it is mainstream to being doing all of the following practices. That is, in some of the areas that these practices cover, any particular development organization may be operating at a lower level than these practices, but for any particular practice, most development organizations operate at or above the level of these practices.

Overall

Basic flow – the common development activities are: talk to customers and/or do market analysis, document market needs via requirements, create a design, implement the design, write tests, execute the tests, do find/fix until ready to ship, ship.

Preparation

Requirements documented using Microsoft Word or Excel – while there are actually quite a few off-the-shelf requirements tools available, most people do not use them. The most common method for documenting requirements is via Word and Excel.

Recording of defects in Bugzilla – there are very few organizations which aren’t using at least Bugzilla to track bugs and Bugzilla is definitely the most popular choice. And recording of defects is pretty much as far as it goes. Enhancements, RFEs, requirements are tracked separately.

Basic initial project planning – this is the simple act of picking a bunch of work to be done, estimating it, dividing it up among the folks that are available to do the work and determining a target ship date. Some teams use MS-Excel, some teams use MS-Project. This does not mean sophisticated project planning. The extent of re-planning is a simple “are all tasks done yet” with the occasional feature creep and last minute axing of important functionality at the last minute.

Basic design – prior to coding features or defect fixes that will take more than two days to do or are highly impactful, a design document is created, circulated, discussed, and updated.

Development

All source code is stored in a source control system – this is not to be confused with “all work is done in a source control system.” It is actually surprisingly common to see the source control system used for archiving of work done rather than as a tool for facilitating the software development process.

Source control and issue tracking integration via a hand-entered bug number in the check-in comment, without enforcement – while there are certainly more sophisticated integrations, most people at least type in the bug number when checking in changes so that they can later find the changes associated with that bug.

Defects have a defined workflow that is defined in and enforced by the bug tracking system – even if it is as simple as Open, Assigned, Closed.

Mainline development – all developers on a project check-in to the mainline (aka the trunk).

Refactoring – while the term “refactoring” has come to mean just about any change to the software, the idea that code should be periodically cleaned up and simplified is pretty much universal at this point.

Nightly build – the entire product is built every night. This does not necessarily mean that tests are also run automatically or that there is an automatic report of the results, just that there is a nightly build.

Quality

Mostly manual testing – this one is the hardest to understand. The process of testing software is one of the most backwards parts of software development.

Unit tests – while the overall testing of software is still mostly manual, unit testing has caught on rapidly in a very short span of time.

Using defect find/fix rate trend to determine release candidacy – this is not actually a particularly good practice, but it is what most people do.

Separation of developer, test, and production environments – you might think this is so obvious it isn’t even worth mentioning, but this principal is violated enough that it is worth mentioning that it is actually a mainstream practice. I mention it to emphasize that if you aren’t doing this, you really need to.

Releasing

Basing official builds on well-defined configurations from source control – the official process for producing production builds includes the step of creating an official configuration in the SCM system and using that configuration to do the build.

Releasing only official builds – all builds that go into production are produced using the official process.

Major and minor releases - creating a major release every 6 to 12 months and minor and/or patch releases on a quarterly basis.

What do you think? Are these right on target or way off base? Do you think there are any missing areas? Do you think the mainstream practice level is higher or lower for some of these? Let me know what you think!

Next: "There is no Bug. It is not the Bug That Bends, it is Only Yourself."

Updating the Agile Manifesto is Required by the Agile Manifesto

The Agile Manifesto, created in 2001, is an attempt to describe a better set of values and principles for developing software. This may sound a bit Zen, but the Agile Manifesto does not actually describe the best way to develop software. Or, if you prefer, it does not describe the best principles to develop software. It is only the attempt to describe the way or the principles.

Another way to describe this is to look at what Agilists say about requirements documents. Written requirements do not actually describe the best product for the customer, they only attempt to do so. In fact, the best product for the customer can only be found via the constant feedback of discussion and delivery. I believe that one of the core values of Agile is the constant search for better software development techniques and thinking. What is the Manifesto if not a user story that attempts to describe the needs of software development itself?

Therefore, doesn’t the Manifesto imply, no demand, that it be constantly reviewed and, if needed, updated? Isn’t that the whole point of Agile? Doesn’t the fact that this is not happening point out a major problem that needs to be immediately addressed?

What happened to changing requirements? The requirements of software development itself haven’t changed since 2001? Nonsense! How can we face the skeptics if we don’t even believe in the Manifesto enough to change it? Shouldn’t we be embracing the idea of updating and revising the Agile Manifesto?

You could take the position that all of the books and blogs and seminars that have been produced since 2001 have turned the Manifesto into merely a historical document. But in practice the Manifesto is much more than a historical document. It is the first footnote or reference in most books, papers, and articles on Agile. Journalists and other media folk refer to it directly and often by URL all the time. As a result, many people read it.

It is perfectly reasonable for somebody new to Agile to assume that Agile Manifesto 1.0 is what we mean when we say “Agile” and that it will be one of their first stops on their trip to understanding Agile. New people read it every day. There are new signatories every day. Consider the impression that people have today when they visit the Manifesto and see a dusty web page that hasn't changed in seven years. Now consider the impression that people would have if they saw a revision history, a forum for proposing and discussing proposed updates, and the date for the next meeting to produce the next iteration of the Manifesto.

I personally have visited the manifesto at least 10 times in the past year including today to see if maybe it was versioned. It isn’t. Each time I visit the Manifesto I think “I’ve changed my thinking on this or that, am I ready to sign it yet?” The answer is still no. I realize now that one reason is that it doesn’t take its own medicine.

The idea for writing this post came as a result of participating in the proposal submission system for the Agile 2008 conference. There is a proposal to discuss revising the Agile Manifesto (account creation required). I have to say that the resistance to revisiting the manifesto is a bit shocking to me. It reminds me of many of the things that I thought that Agilists don’t like such as requirements documents that are set in stone, resistance to change, long iterations, rigidity, etc.

I can’t speak for others, but it doesn’t matter to me if I’m personally involved in the review and revision process itself. For me, it is about practicing what we preach and setting an example for those that are still doing traditional development or doing Agile poorly (in my experience most projects still fall into one of those two camps, unfortunately).

Let’s say the existing Manifesto is a historical document. Let’s enshrine it, put it behind glass, do some sort of ceremony etc. But to me, the fact that there is no version 1.1 or 1.0.1 or 2.0 or anything other than “the document” says to me that something is wrong here. Or conversely, creating a new version, regardless of the size of the change would be a good thing and indicate that we are in fact practicing what we preach.

For those that think it is futile or noise, why comment at all? Why does it matter? Worse case: the folks that want to see this happen go and waste their own time and the signatories say “we aren’t changing it.” So what? What’s the big deal? Is it somehow sending a bad message that some folks think that it is a bit odd that the Manifesto itself isn’t responding to change?

The treatment of this document as some sort of holy artifact feels to me like non-Agile thinking. It feels like a discussion that one would have with somebody about a giant standards document or process document. That reminds me, I did just hear such a discussion at Deep Lean in which somebody in the audience was saying “the coding standards document can’t change! It is a standards document.”

Let’s respond to change instead of following a plan. Let’s talk about if and how the Agile Manifesto should change to take into account all that the Agile community has learned since 2001 and set a good example for those that have not yet discovered the benefits of Agile. What do you think?

You may also be interested in a recent post on Deconstructing the Agile Manifesto.

Deconstructing the Agile Manifesto

Does the phrase “Individuals and Interactions over Process and Tools” conflict with the core values of software development? Is there a better way to convey the original intent of this phrase?

Let’s step back for a moment and take a look at some fundamentals that supersede the Agile Manifesto. First of all, why are we here? It is easy to forget because it is so basic. We are here because we are in the business of creating software for people. And software is a tool . It is a tool which automates a process that would otherwise be done manually by people. There is no actual need for software at all. Anything that software does can be done manually by people. But our core belief is that processes which can be automated should be automated and that there is enough value there that people should pay for it. Otherwise, we would all be doing something else.

Stepping back again, consider your thumbs. From Wikipedia: “The opposable thumb has helped the human species develop more accurate fine motor skills. It is also thought to have directly led to the development of tools, not just in humans or their evolutionary ancestors, but other primates as well.” We are tool users by nature and we should be proud of it. Absolutely, people are important. And that is the basis of our profession: to free people of drudgery so that they may use more of their time for creative pursuits.

That said, the process and tools that we the cobblers of software use ourselves should undergo the same scrutiny that is placed on the software that we create both for ourselves and for the rest of the world (those that are not in the business of developing software). And there is a word for that, it is called “usability.” The idea behind usability is that software is used by people and thus the user model should always trump the developer’s model or the implementation model.

Talking about usability for software development tools is fairly straightforward and well understood, but if those tools are for automating a process that would otherwise be done manually, what is that process? It is the process of software development itself. I believe that one of the advantages of Agile over traditional development is that the processes that are used to implement Agile have much better usability than traditional processes because they are more people oriented.

Thus, comparing “Individuals and Interactions” with “Process and Tools” is the wrong emphasis. It doesn’t compile. It is the wrong conversation to be having. Instead, we should be talking about the usability of the process and tools where that process includes things like self-organizing teams and stand up meetings and the tools include things like retrospectives and stopping the line.

You may also be interested in my thoughts on why it is a bit shocking that the Agile Manifesto hasn't been updated in seven years: "Updating The Agile Manifesto is Required by the Agile Manifesto."

Agile for Business Talk at Eclipscon 2008

If you are going to EclipseCon 2008, consider dropping by to hear my 10 minute presentation about how and why Agile is good from a business perspective.