Apply Elegant Architecture to Your Dev Team: Part II

Previous: "Apply Elegant Architecture to Your Dev Team: Part I"

When you are creating new software, what do you think about? Don’t you think about how it will scale to meet your needs as they grow with high availability? Even if you don’t achieve that on the first try, you are still thinking about it and striving to achieve it. You know that to do it, you will need to make the right technology and architectural choices. Over time, you may need to change some of those choices to keep pace with competitors. Even if your current needs are modest, software development architecture has evolved technologies and patterns to allow software to scale from a single user to hundreds or even thousands of users on multiple platforms at multiple locations with 99.999% uptime.

If you consider your development organization in the same way, how would you apply the same thinking? What technologies are you using? What is the architecture of your organization? Will it scale from its current size to double its size? Will it scale seamlessly to include new teams in new locations? What will happen if you acquire a company? When creating software, you want to design it so that it is flexible and adapts to new circumstances. The same should be true for your development organization.

Another way to look at it is how a particular process would fare if each of the resources available were available at seemingly random times for random periods of time. This exactly describes the world of Open Source Software (OSS). At any given time you have no idea who will be contributing on what or how valuable the contribution will be. You don’t know where the contribution will come from and in many cases you don’t even know who the contributor actually is. This is an extreme example of a development situation. Even though your situation is probably not as extreme as this, by using techniques from the OSS world you will be better positioned to handle unexpected events when they inevitably occur.

Problems are an inevitable and regular part of life. Examples include illness, job change, human error, flight delays, system failure, unanticipated market changes, and natural disasters. The ability to cope with these problems is one measure of the robustness of the organization.

Part of robustness is that as things scale up or down, the impact is minimal. Tools and processes should be selected and designed to work well together whether they are used by 1 person or 10,000 . At all times, it should always feel like each individual is part of a team which is no bigger than 12 people. If practices are not scaleable from 1 to 10,000 then people can develop bad habits that resist scaling. If you develop habits that exist in a scaleable framework, then it is more likely that scaling can and will happen as and when needed.

Next: How Agile Solves Problems

Apply Elegant Architecture to Your Dev Team: Part I

There is another way to think of software development which allows you to leverage your skills as a developer and apply them in a new way. The process of developing software is, in effect, an algorithm implemented with various technologies. You can think of your team, the technologies you use, and your development methodology as a piece of software.

What do you do with software? You improve it. You add new functionality, you increase its performance and usability and you remove bugs. Also, software itself is a document: source code. It is a description of how to perform a set of tasks. You improve the software by changing the source code. Let’s call the combination of your team, tools, and techniques: “the process” and the source code for the process “the process document.”

Not only can you think of your development process as software, you can think of your whole development organization and the people in it as a combination of hardware and software with various communication links. I don’t recommend that you take this advice literally and think this way on a regular basis, or treat people as interchangeable cogs in the machine! However, by thinking about it this way you can leverage your technical design skills to think about how to organize and optimize your development organization and development process. You can leverage well-known design patterns. For instance, consider the communication aspect.

In the world of information transfer there is bandwidth, latency, and connectivity. The best environment for communication is high bandwidth with low latency that is always connected. The worst environment for communication is low bandwidth with high latency that is infrequently connected.

This gives a well-known context for discussing human interaction. On one end of the spectrum is self-communication. If you are responsible for two interdependent modules, then when you make a change to one, you instantly know you need to make a change to the other. You won’t misunderstand yourself or need to have a back and forth conversation to really understand. You just know. On the other end of the spectrum you have two people from different cultures revising a document via e-mail who live exactly half way around the world from each other. In between you have pair-programming, collocation, people working together but sitting on different floors of the same building, folks with a great deal of physical separation in the same time zone, a different time zone, etc.

Then there are different forms of information interchange such as video link, phone, e-mail, IM, wiki, document, etc. This way of thinking can guide your decisions about where to seat people, the value of having high bandwidth links, etc.

Next: Apply Elegant Architecture to Your Dev Team: Part II

Is Your Software Development Organization Mainstream?

Have you ever wondered how your organization compares to other organizations when it comes to the process of developing software? Do you feel like there is an area that really needs improvement, but everybody else says "that's the way everybody does it?" While looking at what is involved in mainstream Agile adoption, I realized that it would be useful to create a list of the current mainstream software development practices for comparison purposes and I thought other folks might find it useful as well.

I define a practice as being mainstream if it is a practice that most people would agree that most people do. It may not be that they do that exact practice, they may do something equivalent or further along the spectrum of what constitutes a good practice, but they at least go to the level described.

The word "practice" is the key word here. This is about what people can be observed as actually practicing in their day to day real work. Things which are described by policy or documented as the correct way of doing something but avoided and worked around don't qualify. A mainstream practice is something that is in common usage that people do because they believe the value is worth the effort and not doing it is sort of like saying “electricity and hot and cold running water aren’t for me, I much prefer candles and fetching water from the well.”

While each of the following individual practices is considered a mainstream practice, that does not mean that it is mainstream to being doing all of the following practices. That is, in some of the areas that these practices cover, any particular development organization may be operating at a lower level than these practices, but for any particular practice, most development organizations operate at or above the level of these practices.

Overall

Basic flow – the common development activities are: talk to customers and/or do market analysis, document market needs via requirements, create a design, implement the design, write tests, execute the tests, do find/fix until ready to ship, ship.

Preparation

Requirements documented using Microsoft Word or Excel – while there are actually quite a few off-the-shelf requirements tools available, most people do not use them. The most common method for documenting requirements is via Word and Excel.

Recording of defects in Bugzilla – there are very few organizations which aren’t using at least Bugzilla to track bugs and Bugzilla is definitely the most popular choice. And recording of defects is pretty much as far as it goes. Enhancements, RFEs, requirements are tracked separately.

Basic initial project planning – this is the simple act of picking a bunch of work to be done, estimating it, dividing it up among the folks that are available to do the work and determining a target ship date. Some teams use MS-Excel, some teams use MS-Project. This does not mean sophisticated project planning. The extent of re-planning is a simple “are all tasks done yet” with the occasional feature creep and last minute axing of important functionality at the last minute.

Basic design – prior to coding features or defect fixes that will take more than two days to do or are highly impactful, a design document is created, circulated, discussed, and updated.

Development

All source code is stored in a source control system – this is not to be confused with “all work is done in a source control system.” It is actually surprisingly common to see the source control system used for archiving of work done rather than as a tool for facilitating the software development process.

Source control and issue tracking integration via a hand-entered bug number in the check-in comment, without enforcement – while there are certainly more sophisticated integrations, most people at least type in the bug number when checking in changes so that they can later find the changes associated with that bug.

Defects have a defined workflow that is defined in and enforced by the bug tracking system – even if it is as simple as Open, Assigned, Closed.

Mainline development – all developers on a project check-in to the mainline (aka the trunk).

Refactoring – while the term “refactoring” has come to mean just about any change to the software, the idea that code should be periodically cleaned up and simplified is pretty much universal at this point.

Nightly build – the entire product is built every night. This does not necessarily mean that tests are also run automatically or that there is an automatic report of the results, just that there is a nightly build.

Quality

Mostly manual testing – this one is the hardest to understand. The process of testing software is one of the most backwards parts of software development.

Unit tests – while the overall testing of software is still mostly manual, unit testing has caught on rapidly in a very short span of time.

Using defect find/fix rate trend to determine release candidacy – this is not actually a particularly good practice, but it is what most people do.

Separation of developer, test, and production environments – you might think this is so obvious it isn’t even worth mentioning, but this principal is violated enough that it is worth mentioning that it is actually a mainstream practice. I mention it to emphasize that if you aren’t doing this, you really need to.

Releasing

Basing official builds on well-defined configurations from source control – the official process for producing production builds includes the step of creating an official configuration in the SCM system and using that configuration to do the build.

Releasing only official builds – all builds that go into production are produced using the official process.

Major and minor releases - creating a major release every 6 to 12 months and minor and/or patch releases on a quarterly basis.

What do you think? Are these right on target or way off base? Do you think there are any missing areas? Do you think the mainstream practice level is higher or lower for some of these? Let me know what you think!

Next: "There is no Bug. It is not the Bug That Bends, it is Only Yourself."

Updating the Agile Manifesto is Required by the Agile Manifesto

The Agile Manifesto, created in 2001, is an attempt to describe a better set of values and principles for developing software. This may sound a bit Zen, but the Agile Manifesto does not actually describe the best way to develop software. Or, if you prefer, it does not describe the best principles to develop software. It is only the attempt to describe the way or the principles.

Another way to describe this is to look at what Agilists say about requirements documents. Written requirements do not actually describe the best product for the customer, they only attempt to do so. In fact, the best product for the customer can only be found via the constant feedback of discussion and delivery. I believe that one of the core values of Agile is the constant search for better software development techniques and thinking. What is the Manifesto if not a user story that attempts to describe the needs of software development itself?

Therefore, doesn’t the Manifesto imply, no demand, that it be constantly reviewed and, if needed, updated? Isn’t that the whole point of Agile? Doesn’t the fact that this is not happening point out a major problem that needs to be immediately addressed?

What happened to changing requirements? The requirements of software development itself haven’t changed since 2001? Nonsense! How can we face the skeptics if we don’t even believe in the Manifesto enough to change it? Shouldn’t we be embracing the idea of updating and revising the Agile Manifesto?

You could take the position that all of the books and blogs and seminars that have been produced since 2001 have turned the Manifesto into merely a historical document. But in practice the Manifesto is much more than a historical document. It is the first footnote or reference in most books, papers, and articles on Agile. Journalists and other media folk refer to it directly and often by URL all the time. As a result, many people read it.

It is perfectly reasonable for somebody new to Agile to assume that Agile Manifesto 1.0 is what we mean when we say “Agile” and that it will be one of their first stops on their trip to understanding Agile. New people read it every day. There are new signatories every day. Consider the impression that people have today when they visit the Manifesto and see a dusty web page that hasn't changed in seven years. Now consider the impression that people would have if they saw a revision history, a forum for proposing and discussing proposed updates, and the date for the next meeting to produce the next iteration of the Manifesto.

I personally have visited the manifesto at least 10 times in the past year including today to see if maybe it was versioned. It isn’t. Each time I visit the Manifesto I think “I’ve changed my thinking on this or that, am I ready to sign it yet?” The answer is still no. I realize now that one reason is that it doesn’t take its own medicine.

The idea for writing this post came as a result of participating in the proposal submission system for the Agile 2008 conference. There is a proposal to discuss revising the Agile Manifesto (account creation required). I have to say that the resistance to revisiting the manifesto is a bit shocking to me. It reminds me of many of the things that I thought that Agilists don’t like such as requirements documents that are set in stone, resistance to change, long iterations, rigidity, etc.

I can’t speak for others, but it doesn’t matter to me if I’m personally involved in the review and revision process itself. For me, it is about practicing what we preach and setting an example for those that are still doing traditional development or doing Agile poorly (in my experience most projects still fall into one of those two camps, unfortunately).

Let’s say the existing Manifesto is a historical document. Let’s enshrine it, put it behind glass, do some sort of ceremony etc. But to me, the fact that there is no version 1.1 or 1.0.1 or 2.0 or anything other than “the document” says to me that something is wrong here. Or conversely, creating a new version, regardless of the size of the change would be a good thing and indicate that we are in fact practicing what we preach.

For those that think it is futile or noise, why comment at all? Why does it matter? Worse case: the folks that want to see this happen go and waste their own time and the signatories say “we aren’t changing it.” So what? What’s the big deal? Is it somehow sending a bad message that some folks think that it is a bit odd that the Manifesto itself isn’t responding to change?

The treatment of this document as some sort of holy artifact feels to me like non-Agile thinking. It feels like a discussion that one would have with somebody about a giant standards document or process document. That reminds me, I did just hear such a discussion at Deep Lean in which somebody in the audience was saying “the coding standards document can’t change! It is a standards document.”

Let’s respond to change instead of following a plan. Let’s talk about if and how the Agile Manifesto should change to take into account all that the Agile community has learned since 2001 and set a good example for those that have not yet discovered the benefits of Agile. What do you think?

You may also be interested in a recent post on Deconstructing the Agile Manifesto.

Deconstructing the Agile Manifesto

Does the phrase “Individuals and Interactions over Process and Tools” conflict with the core values of software development? Is there a better way to convey the original intent of this phrase?

Let’s step back for a moment and take a look at some fundamentals that supersede the Agile Manifesto. First of all, why are we here? It is easy to forget because it is so basic. We are here because we are in the business of creating software for people. And software is a tool . It is a tool which automates a process that would otherwise be done manually by people. There is no actual need for software at all. Anything that software does can be done manually by people. But our core belief is that processes which can be automated should be automated and that there is enough value there that people should pay for it. Otherwise, we would all be doing something else.

Stepping back again, consider your thumbs. From Wikipedia: “The opposable thumb has helped the human species develop more accurate fine motor skills. It is also thought to have directly led to the development of tools, not just in humans or their evolutionary ancestors, but other primates as well.” We are tool users by nature and we should be proud of it. Absolutely, people are important. And that is the basis of our profession: to free people of drudgery so that they may use more of their time for creative pursuits.

That said, the process and tools that we the cobblers of software use ourselves should undergo the same scrutiny that is placed on the software that we create both for ourselves and for the rest of the world (those that are not in the business of developing software). And there is a word for that, it is called “usability.” The idea behind usability is that software is used by people and thus the user model should always trump the developer’s model or the implementation model.

Talking about usability for software development tools is fairly straightforward and well understood, but if those tools are for automating a process that would otherwise be done manually, what is that process? It is the process of software development itself. I believe that one of the advantages of Agile over traditional development is that the processes that are used to implement Agile have much better usability than traditional processes because they are more people oriented.

Thus, comparing “Individuals and Interactions” with “Process and Tools” is the wrong emphasis. It doesn’t compile. It is the wrong conversation to be having. Instead, we should be talking about the usability of the process and tools where that process includes things like self-organizing teams and stand up meetings and the tools include things like retrospectives and stopping the line.

You may also be interested in my thoughts on why it is a bit shocking that the Agile Manifesto hasn't been updated in seven years: "Updating The Agile Manifesto is Required by the Agile Manifesto."

Agile for Business Talk at Eclipscon 2008

If you are going to EclipseCon 2008, consider dropping by to hear my 10 minute presentation about how and why Agile is good from a business perspective.

Mary Poppendieck on Compensation

I just got back from tonight's Agile Bazaar meeting. Making the decision to go or not was pretty difficult. On the one hand there was an offer to have free beer and pizza and watch the season premier of Lost in HiDef after having been deprived of Lost for so long. On the other hand was driving an hour out of my way for the opportunity to hear Mary Poppendieck give a talk on compensation. So of course, I went to hear what Mary had to say.

She told us it was her first version of the talk. There were some rough edges, but it was well worth the trip. It was a fascinating review of the various forms of incentive-based compensation and performance appraisals. The one area of difficulty for me was realizing that Mary is not actually opposed to different levels of compensation for different individuals (which was what I thought she was saying), but rather some of the traditional methods for determining individual compensation.

Thinking about compensation models is not something I do every day, so I'm not going to attempt to summarize Mary's talk. But if you have a chance to attend Mary's talk on this subject, I highly recommend it. Even if you disagree with her points, it will expose some of the assumptions that you might not have even realized you were making and give you some new ways of thinking about compensation.

I'm not sure if this was a point that Mary was making or not, but my biggest take-away was that one of the most natural links between Agile and compensation is to have at least some amount of link between the monetary performance of the product that an Agile team produces and that team's compensation. One of the hallmarks of Agile is rapid feedback. If one of the best ways to have people work on the right things is to make sure they know what the right things are, isn't the surest measure of "the right thing" the success of the end product rather than arbitrary measurements that are disconnected from product success?

The next meeting will be a talk by Steve Berczuk about the connection between developer workspaces and Agile development. Details when I have 'em.

Scaling Agile and Stand-up Meetings

In the "ask away" section Matt poses the following question:

We're doing Scrum on a 10-12 person team (depending on what the DBAs are working on at the moment) and one of the more interesting problems I've been talking about with people is how to scale the process. I have a friend on a 200+ person project and they do Scrum-of-Scrums every day so one person from each "lower level" scrum team goes to another scrum meeting and so on up to the meeting with all the main project heads. This sometimes ends up taking 2 hours out of people's days if they have to go to a bunch of the scrum-of-scrums. What are your feelings on scaling Agile processes up to larger teams? Do you sacrifice somebody to be the "meeting person"? To me communication has always been the most important part of any Agile process, how do you keep that in large teams?

Scaling Agile
At a high level, it seems like your question is: “how do you scale Scrum?” In my experience, scaling is mostly independent of methodology. There are some Agile practices which can limit scalability, but luckily those practices are not required to get the primary benefits of Agile development. The practices which limit scaling are: relying on collocation and using 3x5 cards to the exclusion of other methods for storing and distributing project information. Other than that, what did the 200+ person do to scale prior to Scrum? Whatever it was, there is no reason that I can see to prevent them from scaling using those methods. Speaking specifically about Scrum, the Scrum-of-Scrums is the method recommended by Ken Schwaber for scaling project information flow.

Another area which is often a problem with scaling in general is the frequent integration required by the short iterations. For that you may consider Multi-Stage Continuous Integration.

Stand Up Meetings
As for the stand-up meetings, those are mandatory in Scrum. I don’t happen to agree with that requirement, but let’s hold that thought for a moment.

First, let’s take a look at the expense side of the Scrum meetings. First, people have to get to it. You have to wait until everybody is there. And then you have to get back to your computer. Scrum discusses how to minimize this time, but practically speaking, there is more overhead than just the ideal 15 minute meeting. If you are at a larger company, somebody has to book the room and let people know where it is. Let’s call the cost of the meeting 20 minutes per person. If you have 12 people in a Scrum, that’s 4 person hours per day. That’s the equivalent of half of a person. Those Scrum meetings had really better be worth it!

If they are worth it, then there is actually no problem and if somebody going to Scrum-of-Scrums has a personal dislike of meetings, even when they provide value, then perhaps somebody else should be doing the Scrum-of-Scrums.

Now let’s take a hard look at the stand-up meeting itself. One of the basic ideas of Agile (and Lean) is continual self improvement. If the value of the meeting exceeds the cost, then there’s no problem with the meetings, especially if they are eliminating other meetings. If the Scrum meeting is the only remaining status meeting, that seems like a good thing. However, continuous improvement means we’re never satisfied. Now that you are down to just the one meeting, you should still ask the question: “is it providing more value than the cost? Is there a better way?”

What is the purpose of the Scrum? To quickly find out if people are getting their assigned work done and if not why not. Isn’t it more efficient to do that via e-mail, IM, an issue tracking system, or other means? Someone might say “but the socialization aspect is worthwhile.” Ok, so why not separate the two? By all means let's have some social activities, but that's not a good reason for having a status meeting.

Or perhaps the Scrum is needed because otherwise folks wouldn’t complete their work, or people wouldn’t speak up when they run into an impediment. In that case the Scrum isn’t a solution at all. For instance, perhaps somebody isn’t completing their work because they don’t like it, but the constant peer pressure of the standup meeting is goading them into completing their work anyway. So then the real problem is lack of job satisfaction or low morale or something along those lines. Until you fix that problem, the standup is just acting as a band-aid for an underlying management problem. If you find that having standup meetings from time to time helps to expose management problems, then by all means have the meeting(s), find the problem(s), but don't have the meetings just for the sake of having them!

Short Iterations
One more thought on this topic: the real measure of project status and health is having an increment of shippable value at the end of every iteration. A standup will only expose problems that are on people’s minds, but the forcing function of the increment of shippable value is where you will get the true picture of how things are going. A one month iteration interval is good, but if you can get it down to 2 weeks or even 1 week, that will do far more to expose real problems than a standup will.

Speaking of short iterations, IMHO that’s really the number one thing which makes Agile scale better than traditional development. It is much easier to manage and coordinate large multi-team projects when the scope of everything is constantly focused on one month iterations (sprints). Of course, it may seem more difficult at times, but that is often due to the fact that problems are much more apparent and can’t “hide out” like they do in long iterations.

In summary, experiment and see what works for your particular situation, but always be on the lookout for opportunities for process improvement. Don’t do things by rote, demand that all activities provide real value.

Have a question on Agile or Software Development? Please visit the "ask away" section.

Agile and Software Development Answers

Ask any question about Agile development or the process of software development in general and I'll do my best to answer it. If you'd like to see a post on a particular subject, let me know and I'll consider it.

In the meantime, there's a lot of information on Agile here: "Zero to Hyper Agile in 90 Days or Less" .

Please note that I will continue to remove all spam comments.

The Chinese Finger Trap Part II: Architecture, Development, and QA

In Part I of this series I introduced the idea that traditional development is like a Chinese Finger Puzzle. Solutions to problems often make the problem worse instead of better. This post will cover two more areas that illustrate this phenomenon.

Future-Proof Architecture, Major Releases and Large Feature Sets
A common sentiment when using traditional development is that because it is so hard to redo architecture, because it takes so long to make changes, because it takes so long to produce a release, we had better get the architecture right the first time. We had better take into account all possibilities and eventualities. Since we are going to spend a long time on the architecture, we better plan to do a lot of features for the release. All of those features mean we need to take more variables into account and have an even more thoroughly thought out architecture. In a traditional project, the interplay between architecture, release timeframe, and feature set create a vicious cycle which tends to increase the scope of all three.

In an Agile project, the practices of short iterations and iterative design reinforce each other to keep the amount of architecture that is required as small and simple as possible yet no less than necessary. The product backlog and short iterations keep the feature set focused on those features which will produce the highest customer and market value instead of trying to build a product which takes into account all possible future requirements.

Read more about iterative design: "Designing Software is the Same as Predicting the Future."

Separation of Development and QA
A common pattern in traditional development is the separation of development and QA. Of course everybody always says that “QA is everybody’s job” and “you can’t test quality into a product” and “QA should be involved right from the start,” but what usually happens in practice is that QA occurs at the end. One contributor to this problem is that “during development” the product isn’t stable enough to test and you want to get the most bang for the buck out of your testing because it is so expensive to do and it takes so long to qualify a release. The natural result from these circumstances is the separation of development and QA which is constantly reinforced. Using traditional development patterns, all efforts to unify development and QA are inevitably defeated. Nobody is consciously trying to make it happen, it just happens.

But sometimes it is also on purpose. The separation of development and QA can feel “more honest.” Sometimes development doesn’t want QA to know what is going on because they are sure that they can catch up and fix things and then deliver something to QA which is good. But in order to get the benefits of QA without the help from the official QA folks, development ends up having to do QA themselves which reduces the time spent on development. In any case, traditional development practices tend to separate development and QA.

Agile practices do just the opposite, they tend to bring development and QA closer together. The writing of tests for a particular change is done either before the coding starts or during the coding, not in a separate phase at the end of all coding. Tests are run constantly using the practice of Continuous Integration. QA doesn’t have to wait until the traditional code-freeze to start the bulk of their work. This creates a positive feedback loop where QA gets to show its value on a regular basis, development spends less time fixing things at the last minute, QA learns more about the product, and development and QA become even closer.

Next: Short Iterations