I just happened to read a truly infuriating article in Dr. Dobbs' Journal, which was an extended screed about how XP doesn't work, is heavyweight, and isn't anywhere near as good as Plain Old Traditional Design. Part of me wanted to write a letter to the editor, but that would be pointless -- as usual, I'm months behind, and this article is long in the past. Then I thought about writing a point-by-point counter-screed here in my LJ, but that would be equally pointless, since most of my readers haven't read the article.
So I'm going to instead boil my argument down to the central point, which is this: comparing XP to traditional software design is silly, because true software design is impossible.
No, I'm not kidding here. The point is a little subtle, but the conclusion isn't -- the reason I've become a devotee of agile software processes is that the traditional mechanisms *never* work, and are usually horribly counter-productive.
Somewhere in my pile, I have a button that sums it up nicely. (All wisdom can be found on Nancybuttons.) It says roughly, "It is easy to write software that conforms to the spec. It is impossible to write a spec that describes what you want." That sounds glib, but I think it contains a criticial element of truth.
Folks tend to treat software design as if it was scalable. They look at cute little systems, that can be neatly and elegantly described and spec'ed, and assume that you can do the same thing for large, real-world complex systems. Problem is, it simply doesn't work. Once a system becomes complex enough to be interesting, it rapidly becomes impossible to understand hypothetically how all the pieces will interact. And I don't mean at the API level -- I mean that the complexity of the semantic interactions rises as something like the square of the number of features. You just don't realize, until you start using it, that Feature A and Feature B are critically linked to each other, forcing you to re-evaluate the design of both.
And of course, this isn't even counting the realities of market conditions. It's bad enough when you are dealing with a customer that has a clear vision of the desired end product, so that you have some stability. In commercial software development, the realities are far worse, because you generally don't even really know what you are building until halfway through the project. Hell, it's common to not know what you're building until after you've shipped the first release. Worse, even when you have *five* releases out the door, you often find that market conditions are now demanding a left turn that forces a serious rethink of your underlying assumptions.
One of the great verities of software is that it is constantly evolving. It evolves as the demands change, and it evolves as your understanding of the problem space matures. If you think, upfront, that you understand the problem space well enough to design the program in any real detail, then either the problem is *quite* simple or you're fooling yourself.
Now, this isn't to say that one should do *no* design at all. It's always a good idea to sketch out a rough architecture, to take a stab at what pieces are likely to be necessary, and what the initial control flow is going to be. But it's crucial to not start believing that you have then come up with The One True Design. Probably the worst thing about traditional design methodologies is that people spend so much time on their designs that they then wind up with ego stake in those designs, which usually turns into an impediment when (not if, when) those designs need massive overhauls.
That's one objection to spending a lot of time on design. Another, simpler objection is that it wastes time. I've been on far too many projects that spend *months* navel-gazing about designs, trying to get it all "right". Invariably, I've found, the results are no better than the quick guesses I make in (at most) a couple of days.
Further, there's a real question of habit. The cult of design draws a sharp distinction between Creating Software and Maintaining Software. At Creation time, you do everything very formally, with lots of fancy design work. The problem is, this doesn't help anyone get into the habit of doing the things that are utterly *critical* at maintenance time -- refactoring and rewriting. So instead, most programmers just hack during maintenance.
No, I'm not going to tell everyone that they should be using XP -- indeed, I've only used it in adulterated form. But I do think that there's no excuse for using the traditional design-centric modality instead of an agile one; it's simply institutionalized delusion to believe that you understand a new software problem well enough to design it in detail. You get far better results by taking an intentionally evolutionary perspective, focusing the discipline on constantly making the software *better* and understanding very deeply that you are always chasing a moving target.
Finally, to those who want to argue the point, consider this. I've been programming professionally for 20-some years, at a variety of companies, on many different types of projects. The *only* one of those projects that didn't waste time, and produced a sharply focused, absolutely rock-solid product, was the one using an XP variant. I'm speaking from often-bitter experience here, and my fondness for agile techniques is empirically grounded. I'm not talking about theory here, but down-and-dirty practice...