You have three primary variables: Precision of Prediction, Time Scale and Speed. On the small Time Scale, you can achieve considerable Precision -- it's fairly easy to say what will get done tomorrow, if things are well-managed. On larger Scales, Precision goes down -- it is harder to be precise about what will happen next month, and much more difficult than most managers would like to believe about next year.
You *can* improve Precision on larger Time Scales, but there is a cost: Speed. Specifically, the act of observing a project closely enough to improve Precision will slow it down. This happens in a lot of ways -- more meetings, more precise specification upfront, more bureaucracy and checkoffs -- but they all wind up costing Speed. Large organizations often choose to make this tradeoff, because they value Precision more highly than Speed.
This tradeoff is, as far as I can tell, inescapable. To me, the heart of Agile Development is the choice to put Speed above Precision. That is *not* the only way to interpret Agile -- but it is pretty much where it started, and IMO that choice is what makes Agile valuable. Doing Agile right does *not* mean sacrificing Quality: that is a common confusion, and sadly there are too many people who think they are being "Agile" when they are actually just being lazy. But it absolutely does mean sacrificing the illusion that you can go fast and still have long-term Precision of Prediction. Instead, you focus on being precise only in the short term, and acknowledge that the further out the Time Scale, the less precise you will be about the outcome. Agile works if and only if everybody (*especially* upper management, but also the engineers) accepts that deep down, and are willing to let go of control freakery about the future.
Okay, yes -- the metaphor here is imprecise, and arguably it's exactly wrong. In real quantum mechanics, the act of observing something's position more and more precisely means you have less and less idea about its velocity. Here, observing the future ever more precisely makes it *more* predictable -- in that the project will go slower and slower. But the fact that observation so strongly affects the outcome kind of begged for the comparison.
This post started with my thinking about the fact that, at this point, I just outright refuse to work anywhere that is *not* practicing Agile; it is no accident that at my last two companies, I wound up teaching project management to the managers, largely in self-defense. When I asked myself why, I realized that I am, at heart, a bit impatient, and refuse to sacrifice too much speed in the name of precision. Which is why I will probably always work for small companies: startups are, by and large, much more willing to accept the uncertainty of the future, in the name of getting stuff done...