Podcast: Open Source Technique for Agile Development

Check out this podcast to hear about how "meritocracy" allows developers to contribute outside of their traditional areas, to build trust in their capabilities, and to allow them to naturally gravitate to the areas where they are the most effective.

Podcast: Mainline Chaos

Here's a podcast which covers these topics:

• More on the practice of a "development hierarchy"
• Checkpoints
• Gatekeepers
• Atomic Transactions

Podcast: Hyper Agile Development for Compliance and Distributed Development

It might seem like doing Agile Development is not compatible with compliance requirements. However, Software Configuration Management (SCM) plays a vital role in any compliance effort that involves software development.

This podcast covers the following topics:

• How SCM and Issue Tracking can help with compliance in an Agile project
• Process is a dirty word
• The Hyper Agile practices of "development hierarchy" and "one virtual site"

Podcast: Winning the Jolt Award Using Hyper Agile Development

Here are some thoughts about winning the Jolt Award as well as a discussion of Hyper Agile, the new methodology that enabled AccuRev to deliver AccuWorkflow on time with high quality.

Primary vs. Secondary Agile Development Benefits

Agile is generally talked about as a single package: if you adhere to the principles, you get the benefits. I believe there is another way to look at Agile. Agile introduces a whole new set of practices to the development toolkit. These practices include: product backlog, pair programming, on-site customer, continuous integration, refactoring, test-first development and many others.

While all of these practices have been either associated with or created as a result of Agile, their application and resulting benefits are completely independent of your chosen methodology. Use Scrum or "Waterfall" and you still get the benefit.

For instance, refactoring is a standard feature in Eclipse. Eclipse is an IDE and is completely orthogonal to your methodology. The benefits from these practices are secondary benefits when practicing Agile. In my opinion, the primary benefits of faster turnaround, realizing business value sooner, and quality increases above and beyond the quality increases from the individual practices comes from the single practice of short iterations.

You don't have to do pair programming, use 3x5 cards, collocate, only use small teams, or have a customer on site to get to short iterations. These may help, but they are not the only practices that will. For instance, test driven development is very easy to adopt, contributes greatly to enabling short iterations and does not require any experimentation with personal boundaries.

Let's say you give me the benefit of the doubt and are willing to believe that there are benefits to implementing Agile. Why should you personally spend any time thinking about it?

Next: Agile Development, What's in it for Me?

The Dangers of Moving to Agile Development

My previous post received this comment:

"We have use the scrum process for about 6 months and I feel we are still practicing mini waterfalls. I really need help to defining the role of QA from start to finish. Are we suppose to write test plans? While we are waiting for development, we are unable to write test cases because the specs are not fully define and items that are being added are changing. In the end, we have the product not fully testing and catching up with the next sprint. No test cases are done and no automation is done. Seems like it is what we were doing before."

To me this sounds like a case of going to short iterations too fast! Most of my thoughts on the role of QA in an Agile project (Scrum or not) are articulated in the two recent posts I've already made on this topic, but here are a couple more thoughts...

Opinions may vary on this, but IMHO, if there are no test cases and no automation, you're just not going to succeed with any Agile methodology, Scrum or otherwise. One of the major principles of Agile is that you have working software at the end of every iteration. If it hasn't been tested, how do you know if it is working? It sounds to me like you've got complete chaos.

Do you have an Agile coach on the team? Do you have certified Scrum Masters on the team? If not, I'd highly recommend getting one or both of these.

I believe that for any Agile project to succeed, each story must go through a full and complete mini-lifecycle prior to the end of the iteration. That is, whatever process it would have gone through in your old methodology, it should still go through and it should happen during the term of the iteration. My preference would be that it go through those steps in immediate succession. That is, before code begins on any story, you are clear what you are setting out to accomplish (requirements), you have a plan for doing it (design/spec), and you have created as many test cases as possible. I'm not saying that all of the requirements are known at the start, but that you do as much as you can. Same for designs, specs, test cases. Worst case, all of these should be done for whatever code is done at the end of the release.

For instance, if story 1234 has code associated with it that performs 4 functions, then you better have the requirements, design, and test cases rock solid at the end of that iteration and those test cases better all be automated and passing. By "rock solid" on the requirements and design I mean that at a minimum, the coder better be able to articulate it to you. If they avert their eyes and hem and haw, how will anybody be able to use it?

If at the end of the iteration you don't have near 100% of what you have come to expect just prior to starting your release process (delivery to customer), then you need to stop immediately, do a retrospective, and take immediate corrective actions!!

Good luck and be sure to keep post updates on your progress.

Passing the Baton from Iteration to Iteration

My experience is that with automation, branching, and the supporting process, the goals of determining the quality of an iteration and creating stable baselines can be achieved in parallel. The first step is automation. Without a high degree of automated testing, I'm not sure it makes sense to try to do this in parallel. However, with continuous integration which includes full builds and full test suite runs, there should be few if any problems at the end of the iteration.

Some might say that their automated build and test takes a very long time. That’s a whole separate topic. In the meantime, let's assume that the full automated build and test cycle takes on the order of hours.

By branching, you can start the next iteration without disturbing the previous one. If QA finds any problems, development can fix them on the previous iteration branch. At some point that branch will be declared "done." Those changes are then merged into the current iteration. At that point, there is very little if any difference between where things are and where they would have been if development had started from the "done" baseline.

The advantage here is that development did not have to stop, and you still have a stable baseline. My experience is that once a team gets into a rhythm, the amount of time spent on "touch-up" is minimal.

This also applies to iterations that become releases. The "previous iteration" branch becomes work towards the release and then hopefully soon thereafter the release itself. Meanwhile, work continues towards the next release.

The Role of QA in an Agile Development Project II

At a high level, you should work to keep the ratio of development to QA balanced. The ratio will vary from project to project, so I'm not commenting on that, but once you have a ratio there are several ways to maintain it. In an Agile project, you may quickly find out that the ratio itself is off and need to adjust the ratio. In a project with long iterations, you might not find out that the ratio is out of whack until it is "too late."

In any case, once you find that you are short of QA resources, there are a couple of options. The first option of course is to get more QA resources. Failing that, you can use development resources. It may also be that you have a short-term need for more QA and in that case using development resources might be exactly the right short-term solution.

I think a common rule of thumb, which especially applies in the context of compliance, is that different people do the development and testing of a particular requirement. That is, if Sue is responsible for issue 12345, then Sue doesn't do the testing of it.

As to the exact use of development resources, I advocate that developers take on a sub-set of QA activities, thus allowing for better leveraging of the existing QA resources. For instance, creating automated unit tests or the general generation of automated test cases.

You can of course set whatever policy makes sense for your organization. Having people only write test cases for other people's code is always a good idea. Beyond that, I would assume that QA would take what the developers had done as a starting point, supplementing with additional tests, reviewing what the developers had done, running the official test cycle according to the test plan and reporting on the results.

Next: Multi Stage Continuous Integration

An Introduction to Hyper Agile Development

Hyper Agile (aka Hyper) is a development methodology designed to smoothly scale out from small teams to large teams and to maximize the potential of the development organization from a return on investment (ROI) perspective. I’m still working on documenting Hyper in full. In the meantime, here is yet another preview. Except for the practice of short iterations, each of the individual practices are best practices in and of themselves and can be implemented in conjunction with any development methodology.

Short Iterations - It has been widely stated that the benefits of Agile Development are producing value, maturity, and feedback faster; allowing early feedback to inform requirements to produce a better product, and increasing business flexibility. In my opinion, all of the benefits of traditional Agile Development can be directly attributed to the practice of short iterations, everything else is an enabler.

Automate Everything - The practice of automating everything is an extension of the Agile Development practice of writing automated test cases and fully automating the build process in support of the practice of continuous integration. If our main reason for existing as software developers is to automate manual processes, shouldn’t we be using automation more and more instead of less and less? Of course it makes sense that if we are going to be doing something new, there may not yet be off the shelf automation that fits the bill, but when there is, shouldn’t we use it?

Massively Parallel Virtual Pipelining - this is the process of breaking the lifecycle into as many separate steps as possible and then applying that lifecycle to each user requirement on an individual basis in parallel. This enables a high degree of asynchronous activity which in turn facilitates short iterations. I say a virtual pipeline because various stages of the pipeline may overlap with the same stages in other pipelines and because the various stages may be performed at different locations.

Development Hierarchy - A development hierarchy is simply a representation of the dependencies between groups that includes process steps such as integration, quality assurance, and code reviews. It encompasses a number of best practices including multiple isolation levels, gatekeepers, checkpoints, and always moving from a known good state to a known good state. It provides the structure for the parts of the virtual parallel pipelines that are associated with actual code. The fundamental building block of a development hierarchy is a branch.

Work Item Ranking - Work Item Ranking is done by putting all of the planned items into a single list. The first item is the highest priority item and the last item is the lowest priority item. This is also known as backlog in the Scrum Agile methodology. This practice vastly simplifies project planning, keeps you focused on your target market, and helps to prevent feature creep.

Estimation - At first glance, it might seem like estimation is a relatively simple and mundane task. Think about the task, compare it to your experience, give a rough guess and add a little padding. Actually, there is a surprising amount of literature that focuses solely on estimation. In the end, I think the most important part of estimation is to do it. I suggest you use the PERT method. It is widely used, and it is very simple. Take the least possible amount of time, add 3 times the expected, add in the most time it could possibly take, and divide the sum by 5. That’s it.

Writing Tests Early - Tests and requirements are very closely related. In order to write either one, you need to know the answer to the question “what is this supposed to do?” If the person responsible for writing the tests is struggling, it is a good early warning that it is not clear what the software is supposed to do. If it is not clear what the software is supposed to do, it is unlikely that anybody will be able to write the software to satisfy the end user. Thus, it makes sense to write the tests first to reduce the chance of writing code that does not reflect the needs of the end user.

Quality Quotient - One of the frequent objections to Agile (believe it or not) is related to quality. Since there is no good way to determine the exact quality of a product, it is difficult to empirically compare one iteration against another iteration to help to determine whether or not an iteration is ready for release. A quality quotient is a systematically calculated approximation of the quality of a particular iteration. The data that is compiled as part of the calculation of the quality quotient can also be used to guide future quality and process improvement efforts.

Meritocracy - Meritocracy allows developers to contribute outside of their traditional areas, to build trust in their capabilities, and to allow them to naturally gravitate to the areas where they are the most effective. An efficient way to implement meritocracy is to allow developers to create branches where they can implement their ideas. If they are successful in implementing their idea and the result is accepted, it is then a simple matter to incorporate the change into the development process.

Watered Down Agile Development

I find myself torn between being a supporter of Agile and being associated with the unprofessional aspects of it. For example, we all (or at least most of us I hope) know that discussing religion and politics in a professional environment is not a good idea. Yet that is often what many discussions about Agile seem like.

Actually, that just gave me idea. Next time you find yourself in such a discussion, you could say "hey, let's keep religion out of this" with a smile (or smiley) and try to steer things back to a professional level.

In the end, Agile is either a good idea or it is not and it will be adopted at its own pace in its own way. Trying to force it won't help. Representing it in a professional and appealing way will help.

Some folks complain that Agile is becoming “watered down.” Agile may evolve in ways that we as individuals don't care for. Oh well! But if it is a good idea, it will be adopted. It may be that it is adopted in a watered down fashion. Oh well! If it is adopted in a watered down fashion world-wide, that's actually progress. From that will come the next wave of adoption. And so it goes.

Consider refactoring and continuous integration. These are very widely adopted practices and aren't really even associated with Agile by many (most?) of the people using them, yet they did originate with Agile. You could say that people that are using only those practices are using "watered down" Agile. Is that a bad thing? I would say it is a very good thing. It provides more support for adoption of additional practices and helps to accelerate adoption of Agile in general.

The Role of QA in an Agile Development Project

An interesting question in Agile Development is “how do you incorporate QA?” I guess that all depends on what you see as the role of QA. For instance, if you have your QA people involved from start to finish, then you really wouldn't have to change a thing. For instance, a QA person can:

• help to determine if your stories are well defined
• suggest adding stories related to testability
• make sure that people are creating good unit tests and/or write unit tests
• increase your code coverage by adding more automated tests
• write tests for stories before (test first) or after they are completed
• organize usability testing
• add stories for improving usability
• do exploratory testing on early builds
• verify the completion of stories as they are completed

If they are involved from start to finish in this manner, then you eliminate the need for a well-defined QA stage. If you find at the end of an iteration that QA is "falling behind" then it could be argued that your ratio of development to QA is off. If so, then you can either add more QA resources or have developers take on some portion of the above tasks to restore the balance.

For AccuWorkflow, I had the partial use of multiple QA resources over time, so I had a "virtual QA person" which was different people at different times including developers doing the various tasks I mentioned.

With short iterations, it is likely that there will still be some touch-up work to be done on the previous iteration when the next one starts. Perhaps QA has not finished evaluating the iteration, or perhaps somebody wants to do a demo but runs into a bug that needs to be fixed first.

We were doing variable length iterations where the iterations were between 2 days and a week, so we ran into this hand-off period on a regular basis. The way we handled it was to create a stream for the “hand-off” period.

As an iteration ended, we created a new stream based on the existing development stream. Depending on the SCM system you are using that probably translates to either creating a branch or a label. In any case, the new stream then represented a stable version of the “previous iteration.” If somebody needed to do a little follow-on work, for instance to get things ready for a demo, they used another configuration called “demo” based on the “previous iteration” stream. To prepare for the demo they made changes in the demo stream without affecting the previous iteration or the current one. Once they finished the demo they then merged any changes into the current stream.

Having streams representing multiple iterations also made it possible to easily find out what had changed between iterations which aided in tracking down problems.

Of course, there are all of the standard things that a good QA organization brings that operate at a higher level than stories, but I'll just take those for granted for the purpose of this post.

Next: The Role of QA in an Agile Development Project Part II

Agile Development Drives Better Business Results

Imagine a world where you are predictably delivering high quality products to happy customers faster than ever before and realizing more revenue and higher profits as a result. A world where compliance is automatic, audits are a breeze, and expenses are kept under control.

In this world builds don’t break, developers are happy and productively working on the right things. People are working smoothly towards common goals at multiple sites across the globe. Using parallel development, multiple projects are worked on all at the same time, customer updates are made quickly, and there are fewer regressions due to fixes that didn’t make it into a new release.

More and more folks are experiencing this world as they upgrade their development environment. By development environment I mean the combination of development tools and development methods. Just as C replaced assembler, C++ replaced C, and Java and C# are replacing C++, new development tools are replacing the old ones and Agile development is replacing traditional development methods.

Change is in the air. The terms “Agile Development,” “Extreme Programming,” “continuous integration,” and others can be heard more and more. They are in the titles of conference sessions and books, they are showing up in new publications and web sites, and more and more companies are adopting these new methods.

It has been widely stated that the benefits of Agile Development are producing value, maturity, and feedback faster; allowing feedback to inform functionality to produce a better product, and increasing flexibility. Getting these benefits is much easier using more modern software development tools.

A lot has changed in the software development world in the last 10 years. If you’ve only ever used the older tools such as Bugzilla, CVS, ClearCase, ClearQuest, PVCS, VSS, SCCS, Synergy, or Dimensions, you are missing out on a whole new world of options. There are many exciting new entrants in the world of software development including AccuRev, Ant, Cruise Control, Jira, Mojo, Perforce, Surround SCM, Subversion, Team Foundation, and Team Track. As a simple example, none of the older SCM systems provide atomic transactions, but all of the newer systems do!

There are really only four ways to increase the profitability of a business based on software development: reduce costs via outsourcing, reduce headcount, reduce other expenses, or increase revenues. In the end, reducing expenses can only go so far. By far, the most expensive part of software development is the people. Compared to your people costs, the cost of migrating to a new open source tool or purchasing a new commercial software development tool is a drop in the bucket. The most successful way to increase profits is to increase revenues or increase productivity. These new tools and development methods can help you increase your revenues by helping you get your products to market faster with higher quality.

Refactoring and the Law of Unintended Consequences

Refactoring has the advantage that over time it makes code easier to understand and maintain. This can contribute to the maturity of the code. It has the disadvantage that it requires changing the code and any change has the potential to reduce the maturity and stability of the product.

The so-called “Law of Unintended Consequences” states that any human action will always have at least one consequence that was not intended. The bad part of this is when the unintended consequence causes a problem. Another way to look at this is that each change will have an effect; one of the effects is the reason for making the change in the first place. All of the others can be classified in two different ways: expected or not and reasonably predictable or not. When making a change, there may be effects that are expected even though they were not the reason for making the change. As long as they are considered to be beneficial, that’s probably ok. If there is any reason for not wanting the effect, even if some observers might consider it beneficial, then it is probably best to think of the effect as non-beneficial.

The other classification is whether or not an effect can be reasonably predicted. Some effects might not be immediately obvious. As with chess, the effect of moving a queen into a capture square of a pawn is immediately obvious. Generally, it means that the queen is going to be captured. However, it might also be the case that the queen was moved there in order to guarantee checkmate in five moves. Unless the other player is looking ahead at all possibilities five moves in advance, there is a very good chance that they will miss this. Thus, the worst part of the law is unintended effects which are harmful and cannot be reasonably expected to be predicted.

Some changes have a non-linear effect on a system. That is, the change was very small – perhaps a single character was changed, but the impact was very large, such as destruction of large amounts of data. Because the amount of change often has no connection to the potential impact, it is always best to keep the amount of change to the absolute minimum required.

Even if a change could be reasonably expected to be predicted, sometimes the effort to do so is not invested. In any case, any effects that were not determined prior to incorporating the change are considered to be unforeseen consequences.

There are some strategies for minimizing the risk of unforeseen consequences. The first is to make a reasonable effort to determine effects other than just the intended one. The second is to do an impact analysis and either verify that the current requirements and code coverage are sufficiently high to mitigate the risk or to increase coverage to a reasonably high level.

The Agile Development Waterfall

These days, there is a lot of criticism of the “waterfall” model of development. I believe that is because it is used as a blanket term that actually has three important subcomponents. In a classic waterfall project, the duration of the project is however long it takes with the resources provided to produce the software which meets all of the requirements. There are set process stages that each work item goes through such as requirements gathering, requirements specification, design, development, integration, etc. Lastly, there is the characteristic that all of the work items go through the process stages in lock step.

I submit that all software development projects share an essential part of the “waterfall” model: the steps of the waterfall itself. This is out of necessity. When was the last time you saw code spontaneously pop into existence without being written, or code being written without any goal in mind, even if it was as vague as “make it better?”

All changes to software go through a waterfall of stages. Even if it is a fuzzy idea, change starts with a concept for what needs to be done. From this concept comes a plan. The plan may exist only as a flash of insight in the mind of the developer and they may not even be conscious of it, but there is always a plan. From the plan comes the code. Lastly, the proof of the original concept comes from trying the result. Thus, even in the least rigorous of environments, working on a single issue, there is at least a 6 step waterfall.

All of the stages described may be intermingled, have no clear start or finish, and be repeated multiple times, but they are there. Another way to describe them would be: need, requirements, specification, design, development, test.

The major differences between the classic waterfall model of development and other models of development come in the number and duration of iterations, number of steps in the waterfall, degree of detail in the steps, and the degree of synchronicity between the steps and between the work items.

For instance, the classic waterfall has a single iteration where the duration of the project is however long it takes with the resources provided to produce the software which meets all of the requirements, there is a very high degree of detail including verbose documentation every step of the way, and all of the work items progress together from one step to the next.

In the Agile model, there are frequent short iterations which consist of self-contained projects representing the highest value work that can be done during a single iteration, there is a minimum of detail and documentation, and the work items may or may not progress together from one step to the next, but those steps are still there.

I suggest that the value of the waterfall model, when applied to a single work item, is in producing a well defined series of steps which when followed is known to detect and filter out problems. For any particular development team, those steps may vary a great deal, but by refining the steps and applying them consistently, you have a higher likelihood of producing consistent results.

Next: Sustainable Pace: Supply vs. Demand

Agile Development Objection: Customers Don't Want Frequent Releases

A common objection to Agile Development is that customers don’t want frequent releases. To me, this is one of the stranger objections. It is true that very few customers want frequent releases. Nothing says that a particular customer has to take every release that you produce. In fact, they probably don’t do that no matter how frequently or infrequently you produce new releases. At AccuRev, we’ve had customers that stayed back on 2 year old and even 3 year old releases, despite the fact that bugs they had reported had been fixed and features they had asked for had been added.

Basically, the frequency of releases has nothing to do with the frequency of customer uptake of releases. The exception is when the customer has an urgent need. In that case, you had better be able to attend to that need quickly. Since that is the case anyway, again there is no connection between the frequency of regular releases and customer uptake of releases.

Customers take new releases when there is a good reason to do so. Usually, that is when the delta between where they are and the current release is big enough to warrant the disruption and they have a place in their schedule to put it. As the provider, this window is pretty much unpredictable for you. When that time comes, the customer will take your most recent release or perhaps the one before that under the assumption that more is known about the state of the older release. If you have more than one customer, the chances that your entire customer base is ready to take a release when you release it is pretty low.

One of the advantages of frequent releases is that there is a high likelihood that one of your customers will move to it fairly soon after the release, thus giving you an opportunity to gather feedback right away.

If you are just not ready to release frequently, that’s ok. Instead, use short iterations and just go on to the next iteration instead of releasing it. You can still get most of the feedback related benefits of Agile Development by having folks take a look at a demo of the new iteration and avoid the risk of disrupting your customer base.

Next: Tuning the Frequency of Your Release