Is Extreme Programming an Overreaction? (Part 2)

Although it may be an overreaction, what is it a reaction to? Many statistics show that even today, most software projects fail. It seems reasonable to believe that many companies are looking for a new way of doing things. From the programmer's point of view, it probably seems like it is a failure of management and a lack of proper tools to get the job done. If that's the case, it would make sense to try to minimize management and throw out the old tools.

On the other hand, there's the famous adage that managing programmers is like herding cats. From a management perspective, it can be frustrating as well: "why don't the programmers understand how important it is to get feature XYZ completed by the deadline?"

I think the truth is probably a combination of things; insufficient understanding by management of the complexities of software development, lack of reliable visibility into the plan and current status, overly complex and insufficient tools, insufficient understanding by programmers of the complexities of business realities, and lack of information in the trenches about how the current development plan is connected to the business plan.

It seems to me that part of the problem is plain and simple education. Having been in both camps, often simultaneously, I can shed some light on this subject, but my favorite part and the part I think tool vendors can help the most with is providing tools that provide high visibility of all aspects of the software development process to all stakeholders in real time.

After all, isn't the fundemental reason for software the automation of manual processes? Why in the world should software developers ever have to resort to keeping track of things with 3x5 cards or spreadsheets that provide only a modest benefit over paper and pencil compared with using a software tool that has been tailor made for tracking software development activities? What's next? A return to punch cards?!

As a software development tool vendor, I believe we are obligated to sit up, take notice, and do something about this situation. Otherwise, it seems that we can look forward to state of the art software being created with the software equivalent of sharp sticks and stone knives.

One company that has started to respond to this challenge is the Rally software company with their Rally software product. It automates many of the tasks associated with Agile development. I salute their efforts and hope that all tool vendors will respond in kind. I know that I'll be doing my utmost to make sure that AccuRev takes heed.

Is Extreme Programming an Overreaction? (Part 1)

I've been doing a lot of research and thinking about methodology recently. I also went to SD East this week (great show, great conference) where I was overloaded by Agile this and Extreme that. I guess I was lucky though. I joked to one speaker that I figured I'd made a mistake by leaving out the word "Agile" from my presentation title since it was in practically every other presentation title. He shrugged and said "it seems less here than at other conferences." I can only imagine which conference he recently attended. It must have been called something like "Going Beyond Extreme Programming" or "Lighting Methods for Legacy Agile Developers".

I've read up on Agile methods and Extreme Programming, read articles, talked with prospects and customers about it, and used many of the ideas over the years. For the most part I had heard about those ideas or learned about them prior to having heard of either "Agile" or XP, so I never really thought of myself as an Agile person or an XP person.

I also heard about pair programming, but outside of frequent code reviews, haven't actually done true pair programming. It doesn't strike me as my cup of tea.

In any case, I guess I had subconsciously formed an opinion about Agile methods and then began filtering out new information. After all, I "knew" what it was, so now I could think about other things.

But then I did some more in-depth research on modern methodologies in general, spent some time at a customer that is "going Agile" and followed that up with a week at SD East. I was forced to realize that I had not fully appreciated what it meant to be "Agile."

I can't help but thinking: "is this an overreaction to something?" I've heard things like "but isn't extra training contrary to the manifesto that says 'the only measure of success is creating working code?'" and "keep track of the project with a stack of 3x5 cards." You might say that I'm the one that is overreacting, that I'm only picking up on "the extremes." I don't generally react so strongly to an extreme comment here and there, so I'm pretty sure it is the steady drumbeat that I'm reacting to. By the way, how do you Google a stack of 3x5 cards?

My gut tells me that Agile is a programmer-centric reaction to a feeling that there is too much bureacracy keeping the programmer down; mangement is too heavy-handed, and the development tool stack is too hard to use. If only the programmers were in charge, everything would be better. And to top it all off, the guiding principles are set out in a Manifesto. I agree with some of the principles in the Manifesto, but to me it seems to go a bit too far.

For instance, take the principal that "Working software is the primary measure of progress." That seems far too programmer-centric to me. I agree, there's nothing quite like the rush of seeing a newly implemented feature in action, or the satisfaction of squashing a clever bug. I could spend all day implementing cool features and squashing bugs. But outside of personal hobbies and coursework, isn't the point of developing software to maximize customer happiness and new customer acquisition? At the end of the day, isn't software development a tool for achieving success in business?

Now that I have your attention, stay tuned for Part 2!

Acquiring a New SCM Tool

Folks often ask "what is the best SCM tool" or "is there a comparison of SCM tools available." I would say that generic comparisons of SCM tools are sort of like generic comparisons of cars. Somebody looking for something that is fast and doesn't care about gas mileage probably wouldn't think much of a Prius.

Currently, IMHO, the SCM market is not a commodity market where the products are generally about the same. There is a wide range in both pricing and functionality. So, you really have to know your requirements and the priority of those requirements. A good initial priority to figure out is: which is more important to the organization; price or return on investment? There's no sense in looking at all of the tools on the market if there's not enough budget.

Generally, a good way to acquire a new SCM tool is to start from a well defined set of requirements, create a shortlist of 3-5 candidates that seem close, take a closer look and narrow that down to 2, and then do a proof of concept on those 2 and pick the 1 that most closely meets your requirements.

Depending on the size of the development group and the size of the company, acquiring a new SCM tool can be an eye-opening experience. Finding a tool that meets your requirements is perhaps the easiest part of the process. Some of the harder parts are: figuring out who all of the stakeholders are, getting all stakeholders to consensus on the requirements, getting upper-management buy-in, and securing the budget for the purchase. Getting budget and requirements to match is generally the hardest part and typically involves writing some sort of business case to justify the purchase.

Here are some typical stakeholders: CEO, CFO, Purchasing Agent, Legal, VP of Engineering, Director of QA, key developers, and Release Engineering. Some of these may seem to have nothing to do with SCM, but they do get involved in large purchases that affect the whole company. It is better to make contact with all of these folks early to get the full scope of what is required to change SCM tools so as not to get surprised later and potentially impact release schedules.

If you make a recommendation that fits the budget and best meets the other requirements, but there is still a big gap in the requirements, don't despair! If you've involved all of the stakeholders and built consensus, you have a good chance of leveraging that gap to increase the budget.

Act Like This is the Release

I think the single most important part of Release Engineering (RE) is the principle of "act like this is for the release." There may be times that this is too extreme, but I think that keeping this as a guiding principle will do more good than harm.

For instance, instead of having one process for developers to build during development and another one for release, try to use the same one. That way, when it comes time to release, the path is well worn and more easily reproducible if necessary.

That's hopefully a pretty obvious thing to do, but it isn't always obvious for developers. How many times have you heard "well, it builds for me, it must be your build scripts!" :-) The more closely integrated RE and development are, the better the chances of having a smooth and easily reproducible release.

IMHO, RE should pervade every aspect of development (to the level appropriate) and be involved from start to finish. Now that I think of it, this is akin to "test early and test often"; having QA activities/mindset pervade every aspect of development.

As to where the boundaries are between development, SCM, RE, I think that depends on the organization. However, when possible, I think it works best to have somebody with a big-picture view of all activities to be responsible for SCM/RE and for that to be their first priority. It seems to me that when this is given to somebody in development who does not have the big-picture and it is a second or lower priority, organizations run into trouble. By the way, by "big-picture" I mean somebody that is familiar with how development does things _and_ is very familiar with good release engineering practices and has done a variety of releases for different kinds of projects.

This may not always be possible, but the closer an organization can come to having RE/SCM be an important role (as opposed to an afterthought), staffed by somebody that knows when to branch, label, or lock, when to use parallel development, which SCM product is right for an organization, how to set up a good build system which integrates well into the release process, etc., the better off that organization will be. Of course, you may end up with some developers grumbling about how their freedoms have been restricted, but hopefully they will soon realize that a little more discipline is better than broken builds and long post-freeze settle times.

Check In Early and Check In Often

Developers often put off checking in. They put it off because they don't want to affect other people too early and they don't want to get blamed for breaking the build. But this leads to other problems such as losing work or not being able to go back to previous versions.

My rule of thumb is "check-in early and often", but with the caveat that you have access to private versioning. If a check-in is immediately visible to other users, then you run the risk of introducing immature changes and/or breaking the build.

The nice thing about private versioning is that in most cases the versions checked in are stored on the server and available to other users if they need them, but don't cause problems the way that making them public can.

I find that when developers have access to private versioning, they tend to start relying on it and it requires no effort on the part of the cm admin to "enforce."

The Importance of Branching Models in SCM

Complex code bases, massive parallel development and other problems often plague a project and can bring it to its knees. A number of Configuration Management and Source Control products and processes exist to help try to tackle these growing problems. But which one is best suited for a particular release structure and build environment? A number of branching models exist that each have their particular strengths and weaknesses.

The following article gives an overview of the prevailing branching models in Software Configuration Management.

Software Configuration Management (SCM) Best Practices

Uttam Narsu is an industry expert on software configuration management best practices. While an analyst at Giga and Forrester Research, Mr. Narsu worked with hundreds of differing software development environments and has put together these SCM best practices in this compelling white paper (January 2005).

Please click below to download a copy of this white paper.

SCM Best Practices - Uttam Narsu

Source Control Using CVS, Subversion, and AccuRev

A couple of weeks back, Robert Warner, of Availity gave a presentation to The Jacksonville Developers User Group titled "Source Control Using CVS, Subversion and AccuRev."

The link to his presentation can be found here.

He also, in one of his subsequent blog entries, addressed the audience's question of the criteria against which he and his company had evaluated the tools. The blog entry is simply titled "Source Control."

His is a great overview of the major features and functionallity of some of the SCM product solutions out there.

Enjoy.

Timesafe - Immutability in Software Configuration Management

The link below points to a paper that I wrote a few years ago regarding Immutability in Software Configuration Management.

Abstract. A basic function of Configuration Management (CM) is to accurately reproduce past and present information. Many other aspects of CM depend on this function. Flaws in this function affect and are magnified by other aspects of the system. Ideally, a CM system should perform this basic function automatically and flawlessly without any special knowledge or effort on the part of the users. This paper introduces the Timesafe ® property to describe CM models that possess the capability of accurately and automatically preserving and reproducing past and present information. As a first step towards enabling the proof or disproof that a CM model is Timesafe, this paper provides a formal description of the Timesafe property and the currently known list of counter-examples.

The paper can be found here:

http://www.accurev.com/blog/Timesafe%20-%20Immutability%20in%20CM.pdf

This is one of the core competencies that I believe to be integral in Software Configuration Management products, especially in the dynamic, distributed and parallel development environments that we as CM professionals must try to manage.

What is Software Configuration Management?

The following article is a great introduction to SCM.

http://www.informit.com/articles/article.asp?p=390813&rl=1

Article Brief

Software is easy to change—too easy. And not only is it easy to change, but it is unconstrained by the physical laws that serve as the guardrails of what is possible with hardware systems. Software is bounded only by the limits of the human imagination. Uncontrolled and undirected, imagination can quickly give rise to nightmare. Software configuration management helps to limit this.

A Vocabulary for Software Configuration Management

The beginning of any good conversation requires that a lingua franca be created so that the participants have a common vocabulary and thus assurance that they are comparing apples to apples. SCM is certainly no different. Therefore, in the interest of laying a foundation for further postings, below is an abreviated list of concepts and terms that will help us understand one another:

• Attribute - An attribute is similar to a label. An attribute is a named value that is associated with a version. One use for attributes is to tag a version with a change request number to identify which change it is a part of. Some systems allow attributes to be applied to other objects, such as branches, as well.
• Branch - A branch is a point of divergence. It starts a new, uniquely named series of versions from an existing version. Different systems use different naming schemes. In RCS [Tich85], a new numbering sequence is added. Creating a branch off of 1.1 in RCS might create a new sequence such as 1.1.5.1, 1.1.5.2, 1.1.5.3, etc. In ClearCase [Rati96], there is an initial branch, /main, which begins a sequence that looks like this: /main/1, /main/2, etc. Creating a branch off of /main/2 in ClearCase would record internally that the branch was rooted at /main/2 and the new sequence would look like this: /main/gizmo/1, /main/gizmo/2, etc.
• Change-set - A change-set represents a logical change. It is the set of all changes that must be applied to a previous configuration to derive a new configuration.
• Configuration - CM is typically applied to logical groupings of objects. For instance, you might want to use CM to manage all of the files that are used to build a product or all of the files contained in a web site. A configuration is one instantiation of whatever it is you are managing. You might have a configuration that represents version 1.0 of a product, or a configuration that represents new product development.
• Configuration Definition - A configuration definition is a recipe that is used to specify the exact contents of a configuration. The definition is constructed from the names, versions, timestamps, labels, branches, attributes, and/or change-sets from one or more repositories.
• Configuration Name - A configuration name is a reference that is directly understood by the CM system and can be used to specify the configuration definition. The system uses its configuration name dictionary to automatically translate configuration names into configuration definitions. Object - Typically files and directories.
• Label - A mnemonic name (alias) used to refer to a particular version.
• Object Mastership - When using replication, each object that has a copy at multiple sites is mastered at one and only one site. That is, that site owns the object and updates to that object may only be made at that site.
• Object Name - Typically file and directory names, but can be anything used to uniquely identify objects in a repository.
• Repository - A repository provides storage of all names, versions, labels, branches, attributes, change-sets, and configuration definitions.
• Replication - For efficiency, it is often desirable to make a copy of all of or part of a repository at one or more geographically separated sites.
• Synchronization - Periodically, it is necessary to update replicated repositories with new information. This process is called synchronization.
• Timestamp - The time at which an action was performed. Usually, each version is timestamped in addition to being numbered. Timestamps are often the time at which a transaction was performed.
• Version - Usually a monotonically increasing positive integer starting with 1 used to designate the different contents of an object over time. Any designation which uniquely identifies a version will suffice.

This is a good start... next onto what these terms mean in the context of SCM.