Is Agile requirement gathering that different from waterfall?

Whether Agile or not, trying to develop software products without gathering requirements first is planning to fail. Effective Agile requirements gathering is the same as effective requirements gathering has always been. Since day one, developers have tried to write code without adequate requirements or designs. We've all laughed at the cartoon captioned, "You folks start coding, I'll go find out the requirements." It's funny because it's true, which makes it not funny at all. And yet, in some ways, Agile deceives itself into thinking the cartoon approach is the smart way.

Finding out and clearly capturing correct requirements does involve active stakeholder interaction, but does not equate to spending inordinate amounts of time drafting unwieldy documents that nobody reads and that turn out to be wrong anyway. That would be dumb -- as is ignoring stakeholders -- but I realize some people do it thinking it's what they are supposed to do.

Though probably mainly from ignorance and well-intentioned zealotry, Agile's PR often does a disservice by making indiscriminate, broad-brush misrepresentations of non-Agile (with a capital "A") approaches. Thus, most Agilistas lump all non-Agile development into "waterfall," which they monolithically demonize as inflexibly stupid. Chief among their depiction of waterfall's sins is that it supposedly spends interminable time verbosely documenting every last requirement detail before doing any useful work (i.e., writing code); and yet somehow users aren't involved.

Both Agile and non-Agile approaches prioritize primarily based on business value as determined by stakeholders, not by developers.

Of course, the truth is a bit different. Some projects may have slavishly followed stupid practices that fit Agile's critical view of waterfall (and then probably blamed waterfall for their problems). But many, if not most non-Agile projects, have always applied a lot of flexible and iterative concepts that Agile thinks it invented. After all, 50 or so years of non-Agile projects created the infrastructure and backbone production systems that make much of Agile's seemingly speedier development possible.

Classics such as Fred Brooks' The Mythical Man-Month and Steve McConnell's Rapid Development point out the importance of seeing the whole in order to selectively pick the right pieces, build in the correct order, and ensure they all work together properly. To see the whole product, one first must identify the set of top-level real business requirements. These are the deliverable "whats" that the product must satisfy.

Then, as each component piece is worked on, relevant, real business requirements need to be driven down to more detail so that responsive product requirements can be defined. This approach predates Agile, yet is truly agile (as a synonym for flexible) and succeeds because first defining the right top-level requirements enable subsequent selective, just-in-time iterative development of the right pieces.

In contrast, the Agile requirements gathering process often ignores these points. Agile projects oftenfocus just on individual component features that will be built immediately. Such features are often identified essentially in isolation without the guiding context of the whole. Skipping the apparent architecture and design overhead indeed speeds short-term delivery of that feature, but commonly at the price of long-term difficulties integrating with other similarly short-sighted features.

On the other hand, effective Agile projects overcome such typical shortcomings with a "Sprint Zero" to identify the full set of epics/user stories, which are a form of business requirements. This set provides a view of the whole and thereby guides the process of carving out sprints to build individual features to better integrate with other sprints' features. Both Agile and non-Agile approaches prioritize business value as determined by stakeholders, not by developers. However, within the business context, developers' needs and concerns can also influence the nature and sequence of project activities, regardless of whether or not they are characterized as sprints.