Agile 2008 Conference

I'm happy to announce that I will be doing three sessions at Agile 2008 in Toronto. If you are going, I hope you will consider attending one of my sessions. In any case, I look forward to meeting you there. Agile 2008 promises to be a truly great conference. The planning process was amazing with lots of great discussion of potential proposals.

Here are the three sessions that I will be doing:

Questioning the Perception of Agile (Full Listing)
There are many misperceptions of Agile. This session is mainly for Agile advocates. It is a facilitated discussion about common misperceptions/misconceptions and ways to alleviate them.

Calling All Agile Skeptics, The Curious, and Die-Hard Non-Agile (Full Listing)
If you are looking for a place to either air your skepticism/questions or respond (helpfully) to skeptics/questions, this will be a facilitated forum for both. Please bring an open mind, your sense of humor, and at least one anecdote to share. Participants will be expected to share the floor and help to keep the atmosphere fun and relaxed.

See Large Scale Multi-Stage Continuous Integration in Real Time (Full Listing)
Multi-Stage Continuous Integration is a way to scale CI to larger teams as well as a way to make CI more effective for teams of any size. For a preview, I have a series of posts on the topic. This session will show a real-time environment using AccuRev, Cruise Control, and Jira with hundreds of scripted developers resolving issues, making changes, causing builds to fail and pass, etc.

The Iterative Design of a Lego Sports Car that Transforms Into a Robot, Part II

In Part I, I talked about how the design of a Lego car that can transform into a robot illustrates the process of iterative design. At the beginning of the project, I had the exact requirements and they didn't change at all from the start of the project to the end. However, just having the right requirements did not mean that I could immediately design something that would meet the requirements. At the end of Part I, I had a Lego car that met many of the basic requirements, but still didn't transform. I was stuck because I didn't have the technology that would meet the rest of the requirements.

Part of iterative design is keeping your eyes open, always looking for ways to intelligently expand the solution space. If the information you need is not already at hand, trying variations of things you already know may not lead to a good solution.

Thinking Outside The Box
I had always eschewed the Bionicle Legos as “not really Lego,” so I hadn't originally considerd those. But then two things happened that breathed new life into my Lego project. My son’s third birthday was coming up, and my wife suggested that we make it a Lego birthday. I was responsible for filling the gift boxes for the guests, and I decided to start by visiting the Lego store to see what they might have.

The Lego store had a wide variety of gifts for $5 or less including Duplo dinosaurs, a pony and princess set, knights on horses, and most importantly (for the project) small Bionicle sets. I had never bought a Bionicle set before, so I had never noticed that many of them had articulated joints. The joints were ball-and-socket parts which were small and had enough friction to make them poseable. They were perfect for the project.

There is another kind of Lego set called Exo-Force which has a completely different technology for doing articulated joints and also has a part which when combined with a Bionicle part provides a third technology for articulated joints.

Creating limbs which can fully fold up is a difficult proposition at best. But, I managed to create a couple of variations that allowed me to create the next version of the project. It didn’t really meet the goals of looking like a sports car or hiding the fact that there was more to it than met the eye. It was quite boxy, had odd proportions, and some of the joints were exposed. It was also fairly fragile. But, it could transform!

Sometimes Flexibility is Constraining
Sometimes software that is too generic or two flexible creates another set of problems such as taking too many resources or being too difficult to configure for the task at hand. The knee and elbow joints of the robot form had the same problem. Because they were fully articulated they were just too darn big and difficult to hide. I knew that these joints didn’t need to have the same degrees of freedom as the shoulder, hip, and ankle joints, but had not previously been able to create a hinge that could fold flat and have the full range of motion required. I repeated the exercise of determining all of the available hinge options and found a couple of new ones. The most promising one was a dual Technic hinge which was perfect for the knees.



For the arms, this hinge was too large. But then I realized that less resistance was required for the arms and a single hinge would do. Perfect! Now I had the robot skeleton, I just needed the car exterior.

Inspired by Elegant Design
Many months later, when looking for a new Lego set for my son, I saw a new car set (#4939) that was perfect for the project. I’m not an artist by any means, so one of the challenges has been finding Lego car designs to emulate. Previously they had always been either too big or too small. At one point I had been very excited about the Ferrari that comes with both red and yellow exterior pieces, but it was double the size I needed. Set #4939 seemed to be the perfect size. It didn’t hurt that it was very cool looking and reasonably priced. I had the feeling that this was it, I was finally going to achieve the goal.

Using set #4939 as inspiration, I was able to layer a car exterior onto the robot skeleton without too much trouble. As a bonus, the head, which had previously never really worked out well, was just there. I was within reach of the goal. But still one problem remained. I couldn’t cover the shoulder and hip joints without preventing transforming. I needed something that would allow the top and bottom parts to slide out and then lock in place. I had created parts that slid from place to place before for other Lego projects, but never something that could also lock in place.

Sliding Into Home
After trying many variations using small hinges to act as the locks, I had something that worked, but it was just too big. It was also very clumsy. Luckily, I was in denial. I forged ahead anyway. I wasn’t sure what to do about it being clumsy, but I was determined to create more room in the model. The most obvious step was to do what they do to create stretch limousines: take a regular limo and stretch it out. I made the whole model a bit longer, wider, and taller while keeping the same overall proportions. I also removed all unnecessary internal parts or replaced them with parts that would still do the job but took up less space.

It wasn’t enough, and I couldn’t stay in denial about the sliding part being clumsy forever. But while I was stretching things out I observed that some of the technic parts would slide around a little bit when I exerted too much pressure on them. Of course, the part that would give depended on the amount of friction. Whichever had less friction was what moved and the other one did not. From that came the solution: two bricks together to anchor the axle, and one brick that moved freely but with enough friction that it would stay wherever it was slid to.


The slider was the last major challenge. Once I had this part, it was just a matter of trial-and-error shifting things around until it all worked. There are two sliders; one for the lower body, and one for the upper body, the following picture shows the car with the upper and lower body pulled out as the first steps of transforming.



After the car has been extended, the legs are unfolded.



Lastly, the arms are extended. The head is also fully articulated.



There is still room for improvement, but at this point I think it is more a matter of aesthetics than mechanical design. I’m happy with the current design and hope that perhaps a Lego fan with more art skills than I will take the design to the next step. My other hope is that perhaps the amazing folks at Lego will start producing transforming Lego sets. And of course, if they were Transformer branded, that would be a wonderful (but not necessary) bonus.

[Get the LDraw model for the car here . ]

Motivation, skill, and experience all play important roles in the design process, but they are no guarantee that you are going to be able to design, let alone build what you have set out to do. It almost always requires trial-and-error, serendipity, creative inspiration, research, and an open mind.

See Also: Designing Software is the Same as Predicting the Future

The Iterative Design of a Lego Sports Car that Transforms Into a Robot, Part I

In the lead-up to the debut of the movie “Transformers,” there was already plenty of Transformers merchandise available. My son, who was 2 ½ years old at the time was instantly entranced. While my wife was strolling him through the bookstore he grabbed a Transformers calendar without her noticing. When she got to the register he then used his charms to convince her to buy it.

The Challenge
This got me to thinking. Wouldn’t it be fun to build my son a car out of Legos that could transform into a robot? After all, how hard could it be? I decided to give it a shot and I set myself the following design goals and constraints:

1. Build a Lego model that has two forms: a sports car and a robot
2. Folks looking at it agree that each form is clearly recognizable as that form
3. On casual inspection, it doesn’t look like it transforms
4. 100% Lego, no other parts, and no glue
5. Easily held in one hand
6. Strong (parts don’t fall off easily during use)
7. Robot form can stand on its own
8. The model can easily transform from one form to the other
9. Model is completely connected and transformation does not require the addition or removal of any parts
10. Fully articulated

Now, a year and several generations of Lego models later, I've finally created a Lego model which meets these deceptively simple goals.

As the project progressed, I wasn’t thinking of it as an exercise in iterative design or a formal project, it was always just something I did for fun in my spare time. I didn’t realize the link to iterative design until I started talking to people about the history of the project.

Transforming Requirements Into Working Software
When working with software, even if you have clear requirements that you have validated with users, it is not always clear how you will implement them. The search for a solution has two possible components: thinking about how to do something and implementing something. When thinking about how to do something, that’s really just fantasizing. Of course, it is fantasizing that is focused on producing something real, and it is more likely to produce something real than fantasizing about winning the lottery, but it is still fantasizing.

Requirements, analysis, specifications, and design are all examples of fantasizing. Yes, it is useful to do it, and you will often hit on solutions or eliminate problems much more quickly and much more economically than creating something real without giving it the same amount of thought, but fantasizing is only an approximation of reality, an interlocking web of educated guesses. Not until you actually attempt to create something real will you find out how all of the individual requirements interact and the real costs and difficulty involved. You don’t always find a clear match between requirements and implementation that is cost-effective to implement. In this case you have four paths forward: implement with the closest match and hope for the best, search for new possibilities, invent new possibilities, or shelve the project.

Transforming Lego Car, Iteration 1
The first attempt to build the transforming Lego car didn’t go very well. I was able to create a reasonable looking car which had lots of interior space for the mechanics needed to transform, but creating the sliding parts and the joints proved to be more difficult than I had imagined. I had lots of different hinges, flat parts, special joints, and Technic parts, but all of the possibilities I came up with were either too big, too fragile, too loose, or didn’t support articulation well enough. I was stumped. After a quick trip to the nearby Lego store to see if there was something I had missed, I reluctantly decided to shelve the project.

From a requirements perspective, I had satisfied requirements 2-6, which is half of the requirements. Of course, it was easy to satisfy "doesn't look like it transforms" because it didn't transform at all. :-) From an Agile perspective, the car was fully functional as a car in its own right, and I had discovered lots of valuable information that would help with later iterations of the car.

Little did I know that a hidden bias had kept me from seeing what was right in front of me. It wasn't until a couple of months later that serendipity broke through that bias and brought together the necessary ingredients for the first transforming version of the car.

Next: Part II (of 2)

Applying Usability to the Software Development Process

In order for something to become widely adopted, it has to provide an advantage that is significantly greater than whatever it replaces, and the advantage has to be easily accessible to a mainstream audience. Another way to describe “easily accessible” is to use the term “usability.” That which has high usability is more likely to become mainstream.

There’s a lot of literature on the usability of both software and everyday objects. But what does usability mean when applied to the process of software development itself? If we think of the myriad steps we use to produce software as the implementation of a software development process, then we can think of that implementation as software and consider its usability.

To consider the usability of a particular process, we first have to have a measuring stick for usability. Here’s a set of categories loosely based on some of the chapters from Joel Spolsky’s excellent book “User Interface Design for Programmers” and also adding in “visibility.”

Feedback
We get feedback our whole lives. When we first come into the world, we need constant feedback in order to learn how to survive. Touching something hot is painful, going without food is painful, falling down stairs is painful. In school we get feedback as to which subjects we are good at and which subjects we are not so good at. We can use this feedback to decide what to do more of, what to avoid and as an aid to improve, if we are able to.

The software development process, along with its rules, is just one process that you are a part of. Process, rules, and laws are all around you. You don’t have to consciously think about them but you are constantly reacting to them and they are second nature to you. On your way to work you drive on the correct side of the road, you stop at stop signs, and you stay in your own lane. When you go to lunch, you pay for your food. After work you go to your house instead of going to somebody else’s house.

It is easy to remember the laws, rules, and processes because you interact with them every day and you get feedback in the form of reminders and possibly even unpleasant consequences when you go astray. A great example of this is the rumble strip that has become commonplace on the side of highways. If for some reason you start to drift off the road, you’ll hear a loud warning noise as your tires hit the rumble strip. If you are really violating the law, you’ll hear an even louder sound accompanied by flashing lights. This is also known as feedback.

With traditional development, the development timeframe is much longer and thus the feedback loop is much longer. With Agile, the feedback loop is usually no longer than a month and often on a weekly basis.

Consistency
An important part of absorbing all of these laws, rules, and processes is that they are clearly defined, easy to understand, and easy to remember. For instance, once you have mastered basic physical laws such as the fact that running at full speed into a door is painful, it is easy to understand the reasoning behind having an upper limit on the speed of vehicles on public roads. You may disagree with that limit, but you can generally guess what the limit is for a given stretch of road and it is posted at regular intervals to help you score your guesses. Sometimes there is even a referee.

In traditional development, the long timeframe makes it difficult to absorb what the actual process is and usually everybody has a different mental model of that process. When people have different mental models for the same process, bad things happen. In Agile development, the process is far simpler and much easier to absorb. You have a chance to see the full process from start to finish on a very regular basis.

Expected Behavior
When something happens unexpectedly, it interrupts our train of thought and slows us down. It is a little bit like taking a wrong turn and becoming lost. We have to take time to figure out where we are and how to get back to familiar territory. Most of the time, when working on a software project we are concentrating on a single task that we alone are responsible for.

Consider software development from your own perspective. I’ll bet that you identify with most or all of the following statements:

• When working on a software project on your own, you make changes and use the new version right away.
• One of the reasons that you got into software development is the fact that you can make a change and see the result right away.
• When you have something new working for the first time, you want to show it to somebody.
• It isn’t easy to create something that the customer thinks is exactly right the first time.
• Even when creating something for yourself, it isn’t easy to create what you want the first time.
  

Most software projects are part of a larger scope, even if we are the sole developer. Somebody has to test it, and there are usually organizational standards that must be adhered to. In any case, you will spend a fair amount of time developing according to the rules of the organization (“the process”) rather than the way you would do it naturally if it was an individual project just for yourself. Whenever the way that you would develop software naturally conflicts with “the process,” the more you have to slow down and think about the correct next step and the greater the usability problem.

With traditional development, there is generally much more process to cope with the greater complexity of trying to develop a much larger increment of functionality over a much longer time period. Because of the large gap between how one would develop software on one’s own and how one develops software as part of a much larger process, the steps in the process are often counter-intuitive and thus it is much harder to establish a natural rhythm.

Visibility
It is hard to know what to do next if you aren’t sure where you are. Visibility is akin to project status. Do you feel like you know exactly where you are and what to do next at every step of the process? Or do you feel pulled in a million different directions at once and lose track of what you’ve already done and what you need to do next? If you do feel like you’ve finally “got” how software is developed in your organization, how long did it take to get to that point?

In a traditional project, it is almost impossible to see at a glance what the real project status and progress are. You know that you won’t really know until sometime after code freeze. In an Agile project, the next milestone is much closer, so it is much simpler to see where you are. Granted, if your real goal is six iterations out, you don’t have the same sort of visibility into the next five iterations, but the chances of risks hiding out until code freeze is substantially reduced.

The Usability of the Software Development Process
Based on just these few criteria, it is clear that traditional software development suffers from poor usability and hinders the ability of otherwise competent people to get their work done. While it is true that people can succeed on projects that have very long timeframes and involve a high degree of stress at the end, that mode of working does not align well with our strengths as human beings.

People are much more productive and much less prone to error when they work at a constant and sustainable pace. This also means that when there are unforeseen circumstances, people are more likely to be able to respond well.

Agile Development provides frequent feedback via short iterations, a simple process which is more in keeping with natural human rhythms and capabilities, and significantly more visibility into project status and progress. This creates a highly usable and people-oriented software development environment.

Next: It is Not the Bug That Bends, it is Ourselves

How Agile Works in a Nutshell

Yet Another Fad
We’ve heard it all before. The Segway, UML, Virtual Reality, TQM, object-oriented databases, Artificial Intelligence. Agile’s no different, right? Just the latest fad? That’s what I thought.

Back in 2005 I had my first run-in with Agile development. At a speaker’s reception, my blissful ignorance of Agile came to an end. I was surrounded by Agilists. They looked perfectly normal to me, but then they started to talk about the advantages of keeping track of requirements on 3x5 cards, pair programming, and lots of other crazy sounding stuff.

3x5 cards! Are you kidding me? What is this, the seventies? How can people who make a living writing software advocate the use of 3x5 cards? And what about pair programming? Two people sitting together all day sharing the same keyboard and mouse switching between coding and sitting on their hands, having to hear phone calls about the results of medical tests. Yuck!

I wasn’t just skeptical, I was outraged that all these people were trying to push this nonsense. Keep in mind that I work at a company that provides software tools to companies that are looking to improve their process. On the surface, Agile seems to be diametrically opposed to that. I felt that it was time for somebody to step forward and do something about it. I nominated myself. To gather evidence I read the books, talked to the experts, and attended the seminars. But then one day a funny thing happened. I had an aha moment and I realized there might actually be something of value buried under all of the rhetoric.

Traditional Development Scenario
Here’s what I realized. Let’s say you want to add new features to stay competitive. In a traditional project, you have a known timeframe for major releases. Too short and you’ll spend too much time on overhead, too long and you’ll miss opportunities. For the sake of argument, let’s pick a timeframe of six months and say that allows you to provide three “big features.” Marketing says the three features with the highest ROI are a Facebook plug-in, a Second Life plug-in, and an RSS feed plug-in.

Midway through the design process, marketing announces that the Second Life plug-in is not as marketable as they had hoped and iPhone support is showing signs of becoming very lucrative. You think, oh well, nothing we can do about that now. Just after you finish coding marketing declares that the Second Life plug-in is going to be a complete flop and when can they get iPhone support?

Now that the functionality has settled down, QA starts writing tests and running them. Planning and development took longer than expected and the release deadline is looming. Testing time is compressed, QA concentrates on the most critical stuff and gives the rest a spot check. Here’s a mystery. If your full test cycle takes two days, why does testing take a month? The answer is that you aren’t really doing a month of testing. Problems have been creeping in all along the way that you are just now finding out about and it takes many test/fix cycles to expose them and fix them. Once the find/fix rate gets down to an acceptable level, you declare victory and deliver the new release.

Now Let’s Try Agile
For simplicity, let’s use two month iterations. You start with the Facebook plug-in. You plan. You design. You create a test plan while the code is being written. You discover potential problems and deal with them. You automate those tests while the code is being written. As the code is written, it is integrated, built and tested continuously; problems are found and fixed immediately. At the end of development, the only problems that remain are the ones that could only be found at the end of development. If you wanted to, you could cut a new release with very little overhead if you wanted to.

Marketing announces that the Second Life plug-in is not as marketable as they had hoped and iPhone support is showing signs of becoming very lucrative. So, you start on the RSS feed support. Now marketing says the Second Life plug-in is worthless but iPhone support is hot, so you add iPhone support. During the whole process, you kept your options open. At the end you have produced more business value than using the traditional process, and you were in a position to start realizing that value much earlier.

I also realized that you don’t need to do any of those things that at first really turned me away from Agile. You can still get the primary benefits without resorting to 3x5 cards or pair programming and you don’t have to do frequent releases. These things are optional and you can use them or not according to your preference and circumstances. My advice to you is don’t pass by the opportunity to find out if Agile can work for you the way it has worked for others. I’m not suggesting you throw out your skepticism; try a pilot project!

Next: No Really, What is Agile?

New Reddit for Agile Development

Reddit is a great way to surf the net for things that you are interested in, and you can participate in determining what's good and what's not. Now you can surf using reddit for all things Agile! Check out the new Agile reddit!

Is Your Dev Team Having Performance Problems? Try Niagra!

Have you ever thought “why do the companies I work at have so many problems with developing software?” Well, I can tell you from interacting with literally thousands of software development organizations that most development organizations have problems with reliably and predictably producing high quality software. What I hear over and over again is “if only we could hire the right people” or “if only people could be more disciplined.” The other thing I hear over and over again are various tweaks to traditional development that people believe will fix things but that “other people just don't get it.”

Well, there are only so many people out there to hire so unless we find some magical place to hire more of “the right people” from, we are going to have to figure out how to use the people we have. And I hate to break it to you but you just aren’t going to get more discipline out of people. Whatever discipline that exists today, that’s all you are going to get. So what’s left? Changes to the way we do things.

Let’s say there was some combination of techniques which would mean that most development groups could use traditional development to reliably and predictably produce high quality results. Let’s call it the “Niagra” method. Furthermore, you (or whoever wants to make this claim) get to define the Niagra method. It works every time. Just use Niagra and performance improves overnight (when used as directed).

If the Niagra method existed, it would have become widespread by now. It would have become the #1 prescribed solution to project performance problems. If it existed and produced the claimed results, people would start referring to it, and it would spread rapidly. That’s how C++, Java, and HTML became mainstream. People rapidly adopted them because they provided benefits that anybody could realize. In fact, there are many proposed Niagra methods, but none of them have become mainstream because they only work under special conditions.

This raises an obvious point. If traditional development is so bad and is such a big failure, then why has the software industry done so incredibly well? The software industry is an incredible success, that’s absolutely true. It is incredible how much of our daily lives runs on software and how much software has positively influenced our quality of life. The benefits of software, when it does finally ship and it does finally have a high enough level of quality are worth waiting for. Otherwise, of course, there would no longer be a software industry.

There have been incremental improvements to the software development process which have produced incremental improvements and have become widespread. Examples are: the use of software development tools such as source control and issue tracking, breaking down large tasks into small tasks, nightly builds, one-step build process, moving from procedural programming to object-oriented programming and many others. The important point here is that each of these improvements have become widely adopted and made things somewhat better, but still didn’t solve the root cause. The process is still unpredictable and unreliable.

This is akin to the O(n) approach to problem solving. If you have an algorithm which takes 100*n^2 operations, then getting that down to 50*n^2 is a good thing, but changing the algorithm to (for instance) be 200*n is much better. To date, changes to the software process have been of the first variety, shrinking the constant.

Perhaps Agile development is the answer. But isn’t Agile “yet another fad” which will blow over any day now? After all, we have had CMM, 9001, TQM, Critical Chain, and many other “next big thing” ideas in the past. What makes Agile any different? Previous ideas have all been centered around the idea that the framework underlying traditional development is fine, it is the implementation, discipline, people, or management of the people that is the problem. No, it is the underlying framework that is the root problem.

Certainly, people issues are important. Getting a team gelled and working well together with the right combination of skills is essential. But the success rates of these teams isn’t exactly stellar either! Yes, it is better, but they have similar problems with predictability and quality. A root problem remains. I do not claim that solving this problem means that suddenly teams that have insufficient skills and hate each other will start pumping out high quality software on a regular basis. But I do claim that Agile development has the ability to dramatically increase the potential of any team. And to truly succeed you have to do both: create a good team, and use Agile development.

Let's take a deeper look at the fundamental problems with traditional development.

Next: Traditional Development is a Chinese Finger Trap

Apply Elegant Architecture to Your Dev Team: Part II

Previous: "Apply Elegant Architecture to Your Dev Team: Part I"

When you are creating new software, what do you think about? Don’t you think about how it will scale to meet your needs as they grow with high availability? Even if you don’t achieve that on the first try, you are still thinking about it and striving to achieve it. You know that to do it, you will need to make the right technology and architectural choices. Over time, you may need to change some of those choices to keep pace with competitors. Even if your current needs are modest, software development architecture has evolved technologies and patterns to allow software to scale from a single user to hundreds or even thousands of users on multiple platforms at multiple locations with 99.999% uptime.

If you consider your development organization in the same way, how would you apply the same thinking? What technologies are you using? What is the architecture of your organization? Will it scale from its current size to double its size? Will it scale seamlessly to include new teams in new locations? What will happen if you acquire a company? When creating software, you want to design it so that it is flexible and adapts to new circumstances. The same should be true for your development organization.

Another way to look at it is how a particular process would fare if each of the resources available were available at seemingly random times for random periods of time. This exactly describes the world of Open Source Software (OSS). At any given time you have no idea who will be contributing on what or how valuable the contribution will be. You don’t know where the contribution will come from and in many cases you don’t even know who the contributor actually is. This is an extreme example of a development situation. Even though your situation is probably not as extreme as this, by using techniques from the OSS world you will be better positioned to handle unexpected events when they inevitably occur.

Problems are an inevitable and regular part of life. Examples include illness, job change, human error, flight delays, system failure, unanticipated market changes, and natural disasters. The ability to cope with these problems is one measure of the robustness of the organization.

Part of robustness is that as things scale up or down, the impact is minimal. Tools and processes should be selected and designed to work well together whether they are used by 1 person or 10,000 . At all times, it should always feel like each individual is part of a team which is no bigger than 12 people. If practices are not scaleable from 1 to 10,000 then people can develop bad habits that resist scaling. If you develop habits that exist in a scaleable framework, then it is more likely that scaling can and will happen as and when needed.

Next: How Agile Solves Problems

Apply Elegant Architecture to Your Dev Team: Part I

There is another way to think of software development which allows you to leverage your skills as a developer and apply them in a new way. The process of developing software is, in effect, an algorithm implemented with various technologies. You can think of your team, the technologies you use, and your development methodology as a piece of software.

What do you do with software? You improve it. You add new functionality, you increase its performance and usability and you remove bugs. Also, software itself is a document: source code. It is a description of how to perform a set of tasks. You improve the software by changing the source code. Let’s call the combination of your team, tools, and techniques: “the process” and the source code for the process “the process document.”

Not only can you think of your development process as software, you can think of your whole development organization and the people in it as a combination of hardware and software with various communication links. I don’t recommend that you take this advice literally and think this way on a regular basis, or treat people as interchangeable cogs in the machine! However, by thinking about it this way you can leverage your technical design skills to think about how to organize and optimize your development organization and development process. You can leverage well-known design patterns. For instance, consider the communication aspect.

In the world of information transfer there is bandwidth, latency, and connectivity. The best environment for communication is high bandwidth with low latency that is always connected. The worst environment for communication is low bandwidth with high latency that is infrequently connected.

This gives a well-known context for discussing human interaction. On one end of the spectrum is self-communication. If you are responsible for two interdependent modules, then when you make a change to one, you instantly know you need to make a change to the other. You won’t misunderstand yourself or need to have a back and forth conversation to really understand. You just know. On the other end of the spectrum you have two people from different cultures revising a document via e-mail who live exactly half way around the world from each other. In between you have pair-programming, collocation, people working together but sitting on different floors of the same building, folks with a great deal of physical separation in the same time zone, a different time zone, etc.

Then there are different forms of information interchange such as video link, phone, e-mail, IM, wiki, document, etc. This way of thinking can guide your decisions about where to seat people, the value of having high bandwidth links, etc.

Next: Apply Elegant Architecture to Your Dev Team: Part II

Is Your Software Development Organization Mainstream?

Have you ever wondered how your organization compares to other organizations when it comes to the process of developing software? Do you feel like there is an area that really needs improvement, but everybody else says "that's the way everybody does it?" While looking at what is involved in mainstream Agile adoption, I realized that it would be useful to create a list of the current mainstream software development practices for comparison purposes and I thought other folks might find it useful as well.

I define a practice as being mainstream if it is a practice that most people would agree that most people do. It may not be that they do that exact practice, they may do something equivalent or further along the spectrum of what constitutes a good practice, but they at least go to the level described.

The word "practice" is the key word here. This is about what people can be observed as actually practicing in their day to day real work. Things which are described by policy or documented as the correct way of doing something but avoided and worked around don't qualify. A mainstream practice is something that is in common usage that people do because they believe the value is worth the effort and not doing it is sort of like saying “electricity and hot and cold running water aren’t for me, I much prefer candles and fetching water from the well.”

While each of the following individual practices is considered a mainstream practice, that does not mean that it is mainstream to being doing all of the following practices. That is, in some of the areas that these practices cover, any particular development organization may be operating at a lower level than these practices, but for any particular practice, most development organizations operate at or above the level of these practices.

Overall

Basic flow – the common development activities are: talk to customers and/or do market analysis, document market needs via requirements, create a design, implement the design, write tests, execute the tests, do find/fix until ready to ship, ship.

Preparation

Requirements documented using Microsoft Word or Excel – while there are actually quite a few off-the-shelf requirements tools available, most people do not use them. The most common method for documenting requirements is via Word and Excel.

Recording of defects in Bugzilla – there are very few organizations which aren’t using at least Bugzilla to track bugs and Bugzilla is definitely the most popular choice. And recording of defects is pretty much as far as it goes. Enhancements, RFEs, requirements are tracked separately.

Basic initial project planning – this is the simple act of picking a bunch of work to be done, estimating it, dividing it up among the folks that are available to do the work and determining a target ship date. Some teams use MS-Excel, some teams use MS-Project. This does not mean sophisticated project planning. The extent of re-planning is a simple “are all tasks done yet” with the occasional feature creep and last minute axing of important functionality at the last minute.

Basic design – prior to coding features or defect fixes that will take more than two days to do or are highly impactful, a design document is created, circulated, discussed, and updated.

Development

All source code is stored in a source control system – this is not to be confused with “all work is done in a source control system.” It is actually surprisingly common to see the source control system used for archiving of work done rather than as a tool for facilitating the software development process.

Source control and issue tracking integration via a hand-entered bug number in the check-in comment, without enforcement – while there are certainly more sophisticated integrations, most people at least type in the bug number when checking in changes so that they can later find the changes associated with that bug.

Defects have a defined workflow that is defined in and enforced by the bug tracking system – even if it is as simple as Open, Assigned, Closed.

Mainline development – all developers on a project check-in to the mainline (aka the trunk).

Refactoring – while the term “refactoring” has come to mean just about any change to the software, the idea that code should be periodically cleaned up and simplified is pretty much universal at this point.

Nightly build – the entire product is built every night. This does not necessarily mean that tests are also run automatically or that there is an automatic report of the results, just that there is a nightly build.

Quality

Mostly manual testing – this one is the hardest to understand. The process of testing software is one of the most backwards parts of software development.

Unit tests – while the overall testing of software is still mostly manual, unit testing has caught on rapidly in a very short span of time.

Using defect find/fix rate trend to determine release candidacy – this is not actually a particularly good practice, but it is what most people do.

Separation of developer, test, and production environments – you might think this is so obvious it isn’t even worth mentioning, but this principal is violated enough that it is worth mentioning that it is actually a mainstream practice. I mention it to emphasize that if you aren’t doing this, you really need to.

Releasing

Basing official builds on well-defined configurations from source control – the official process for producing production builds includes the step of creating an official configuration in the SCM system and using that configuration to do the build.

Releasing only official builds – all builds that go into production are produced using the official process.

Major and minor releases - creating a major release every 6 to 12 months and minor and/or patch releases on a quarterly basis.

What do you think? Are these right on target or way off base? Do you think there are any missing areas? Do you think the mainstream practice level is higher or lower for some of these? Let me know what you think!

Next: "There is no Bug. It is not the Bug That Bends, it is Only Yourself."