Background: I've been doing Agile of one sort of another from essentially the beginning. Back in the late 90s, a few of the particularly forward-thinking guys at Looking Glass (Tom Leonard, Doug Church and the like) picked up on this hot new idea called "Extreme Programming". We played with that a little in the Siege Engine project (the barely-started next-generation game engine that we were working on when LG shut down), and then Tom put XP strongly into practice at our next company, Buzzpad, where he was the CTO. We introduced XP piece by piece, consistently finding that it made things work better, and ended up using most of it. Since then, I've been increasingly hardcore in my refusal to work in a non-Agile environment, which has led me to teaching a couple of companies how to do it, and acting as de facto project manager from time to time.
That said, my definition of "Agile" seems to be very different from what most people are experiencing. In particular, the horror story I have been hearing repeatedly (and which shows up a lot in yesterday's thread) is a misguided notion that Sprints are the heart of Agile, and that if you simply do short Sprints and daily Scrums, this will magically cause your problems to go away. This is, of course, nonsense -- it isn't the main focus of Agile, and on its own it doesn't do much to help.
So here's my concise, if intentionally vague definition of the term:
Agile development means letting go of the illusion that software development is predictable, and instead finding disciplines that work, through constant iteration and refinement.The first part of that is the most important, and the one that many many folks are loathe to cope with. The meme of, "If we just try harder, we can predict how this will come out" is horribly pervasive in all of programming. If we just have enough sales engineers, we will know what the customers want. If we spend enough time on specs, they'll represent what the customers want. If we spend more time on upfront design, we'll get it Just Right, and can simply implement that design. If we spend enough time in meetings, we'll know how long this will all take.
It's all nonsense. History has shown, over and over and over and over again, that software is (in most cases -- yes, there are some exceptions) deeply unpredictable. And trying to force it to be predictable not only wastes time, it usually results in worse end results.
Recognizing our own powerlessness is the hardest single step here: most engineers dislike admitting it, and most managers even moreso. But once you admit that large-scale predictability is more or less hopeless (at least at the fine grain), you can instead refocus on the small scale, where we *can* achieve a measure of predictability, and build from there.
Essentially everything else about Agile is actually an implementation detail from there. That's important: those details are *not* the definition of Agile, they should exist only to support the crucial goal of breaking things down into steps that actually work. The corollary of this is that, if your process isn't actually serving that goal, then your process is probably wrong for your situation. The single most annoying thing about some management consultants is that they claim to have one-size-fits-all answers, which is an outright lie. There is no single answer for every company; moreover, the correct answers within a single company will frequent vary from group to group, and evolve over time. Retrospectives and process adjustment are annoying, but they are absolutely essential to doing Agile really well -- you can't predict the right process either, so that too has to evolve in small steps.
Sprints are *especially* an implementation detail. They're important, mind -- they tend to be necessary as a discipline for breaking things into bite-size chunks. But people who carry this to the extreme of, "Anything you can't completely finish in two weeks must never be started" drive me crazy. Yes, you have to have small, measurable milestones, but when this gets in the way of getting work done, you're doing it wrong.
So what are my typical guideposts for doing Agile? In rough priority order:
- All stakeholders have specific responsibilities and specific restrictions. This is often the hardest step to get past, and the most important. It is difficult to convince upper management that no, you're not going to give them exact promises of what the product will look like when, and yes, they have the absolute responsibility to frequently review the state of things and reprioritize. (And often even harder to convince customers of this.) But there is no substitute for it: if the people above you are holding on to illusions, it will nearly always cause a disaster. Contrariwise, if management accepts their role with gusto, and play by the rules, it can make things ten times easier. (Implementing an agile project at Memento was enormously eased by the fact that Mike Mulholland, our Product Manager, despite having spent 25 years doing things the old-fashioned way, absolutely *adored* the Agile approach, understood it quickly, and did his part faithfully.)
- Communication Is All. The primrose path to hell is paved with specs, which tend to interfere with honest communication. Everybody should be talking, all the time, preferably across the usual silos -- engineers, QA and product reps should all be interacting constantly. Artifacts of those conversations can be useful, but should not be used as an excuse to stop talking.
- Frequent process retrospection. That is, the team as a whole should always be looking at what is working and what isn't. When carried to an extreme this becomes pointless navel-gazing, but taking time out each Sprint to reflect is not merely useful, it's crucial. This sort of reflection is where Agile came from in the first place, and since every project's needs are different, every project needs to set aside time to do it honestly.
- Fast and frequent iteration with customers. This isn't *quite* the entire point, but it's not far off. One of the most important illusions to give up is that you actually know what you should be building. (Which requires the customer giving up the illusion that they really know what they want.) So instead, you get something in front of real users (or the nearest thing to them as you can get) as fast as possible, take their feedback, and keep iterating quickly. The ideal is "every sprint is a release", meaning that the results of each sprint get shown to the customer representative, which is doable iff they are sufficiently tied into the process.
- Flexible scope. This one's often not mentioned explicitly, because management doesn't want to hear it, but in practice it is always essential. The central illusion that you must dispense with here is that you can predict what will get done when. Since that isn't true, this means that either the schedule or the scope *must* be flexible. 90% of the time, the correct answer is scope -- you know when you will deliver (since that is often driven by externalities), but can't promise precisely what.
- Story-centric development. You don't try to write deeply detailed specs in advance; rather, you design *enough* to know approximately which way you're going, and otherwise prioritize in terms of medium-grained Stories. These should ideally be smallish -- on the order of a few days' work each -- in order to make them more tightly understandable when the time comes, and should be as independent as possible, so that you can reprioritize flexibly.
- Frequent but predictable reprioritizations. Specifically, the rule of thumb I like is that the Product Manager / Customer Representative is allowed (and encouraged) to revisit the priority order of the Stories between Sprints, but not during them. This means that the engineers can focus on getting work done most of the time, but everyone has an agreement that priorities can and will change pretty often. The highest-priority items should always be at or near the top, so that they get shipped. (See "flexible scope".)
- Deep testing and constant refactoring. These go hand in hand -- the XP phrase that I've always loved is "Coding with Courage". If you try to code too conservatively, you wind up with unmaintainable spaghetti. You always have to have the courage to make the code *right*, and the only way to get that courage sanely is to have enough QA -- preferably mostly automated, but there usually needs to be a mix. (I am often bad about putting enough tests into place, but I recognize that that's a flaw.) Nowadays, I usually recommend making it explicit that engineers are responsible for much of the testing; simply dumping it on QA people isn't acceptable.
- Code Sharing. One of the relatively important and often overlooked tenets of XP is No Silos -- all code is shared by the team to the greatest extent possible. In practice, everybody hates this: managers dislike it because they don't have a simple answer to who to ask which questions, and engineers dislike it because it breaks their deeply-held "My Precious" attitude towards their pet code. But it is essential if you are going to be serious about following the priorities: if people are silo'ed, those silos usually won't match the current need. This requirement in turn drives the need for deep code review and/or pair programming.
- Short Sprints. Yes, I've deliberately listed these last, because I honestly think they are the most implementation-detail aspect of the whole thing, and not always even necessary. Sprints are typically required as an artifact of scale. You need them to provide a regular clock for the management / engineering contracts, to keep everyone from annoying each other. You need them to remind you to reprioritize regularly. You need them to remind you to keep your stories small. But the truth is, in a *very* small team with *very* tight communication, they can simply get in the way. (Querki used sprints out of habit for the first year, but about six months ago I simply dropped them -- they were simply overhead at the moment, and not adding anything.)