Migrating from a waterfall programming model to Agile development means far more than learning new programming techniques. A move to Agile means adopting a new way of seeing software development and project deliveries.
Agile and old-style programmers don’t have to be at each other’s throats. If Agile developers can admit that waterfall created many successful programs and waterfall programmers can see that there’s some advantages to Agile’s flexibility, they might find they have more in common than they’d thought: Creating good, solid software.
It’s easy for developers to get ticked off. Many programmers and testers invest a lot of energy in ensuring that they create the best quality software they can — and often they are so committed to Their Way of Doing Things that they hiss and spit at people who see another path.
Some people have very strong opinions on the matter. For example, Jitendra Subramanyam is no Agile fan. The director of product strategy and research at CAST Inc., a semi-conductor, intellectual property developer firm, says, “The very nature of Agile programming conspires against you.” Bits and pieces of functionality that eventually become interdependent are created and tested separately in different scrums, Subramanyam says. “New functionality is often added on top of old resulting in ‘opportunistic refactoring,’ which further muddies the architectural waters, threatens reliability and performance, and increases the cost to modify and maintain the software.” Yack! Who’d want to do that?!
Agile proponents see the matter quite differently; many view the waterfall model, sequential software development model in which development is seen as flowing steadily downwards (like a waterfall) through several phases, as inherently flawed. Victor Szalvay, product development lead for CollabNet‘s ScrumWorks sees waterfall software development as akin to a production line conveyor belt: “‘Requirements analysts’ compile the system specifications until they pass the finished requirements specification document to ‘software designers,’ who plan the software system and create diagrams documenting how the code should be written. The design diagrams are then passed to the ‘developers,’ who implement the code from the design,” Szalvay told me in an interview. Waterfall may sound ideal, Agile supporters say, but in the real world it doesn’t work well.
Perhaps, though, the truth lies somewhere in-between. As Lewis Cunningham, a system architect for JP Morgan Chase, suggests, “Waterfall projects have been successfully creating software for decades. I personally have been doing this for two decades (holy crap, how did that happen?) and while I have seen some less than perfect project executions, I have not seen many abject failures and certainly not at the rates some would have you believe.”
Cunningham continues, “Now, before all you Agile fan boys out there start sharpening your pencils, fingernails, and swords, let me say I am not an Agile opponent. An Agile development methodology offers some very real benefits… I just want to start with the point that waterfall is not bad (or evil).”
What does kill projects then? “Where I have seen projects fail, it usually had little to do with the chosen methodology. It usually had more to do with personal agendas, cowboy coders, bad management, and a nasty us versus them conflict between IT and business,” writes Cunningham.
Tim Ottinger, an Agile consultant, takes a broader view of what can go wrong with Agile deployments. “Sometimes organizational inertia and control issues get in the way. The largest problem is when management expects that only the programmers need to change. The truth is that Agile development touches product management, customer contact, team management, deployment, and schedule management in a deep and significant way. Getting programmers to work in pairs, test-drive code, and re-factor their systems is trivial compared to dropping individual assignments, reducing scope, minimizing work in progress, and connecting customers to development teams.”
So, what is the right way to deal with programming methodology? Cunningham stresses the importance of keeping in mind that “IT exists for the business. Period. No project, using any methodology, should ever start without that assumption and agreement.” Don’t let that, Cunningham continues, trap you into a set-in-stone approach to project planning. That’s not a problem with waterfall, he says. “It’s a problem with management. It’s a problem with the way project management has been taught,” says Cunningham.
You should also keep in mind that project management can be quite different in Agile projects because team members are given more autonomy.
Johanna Rothman, author of Manage It! Your Guide to Modern, Pragmatic Project Management and conference chair for the Agile 2009 conference, says that Agile uses different means to encourage productivity. “Agile makes extensive use of time-boxes to manage risk and finishing work. Time-boxes are an old technique to help people focus, and we use them all the time, whether we manage projects or not,” says Rothman. “Think of [this as] taking a child to a toy store or you going to the bookstore (or the wine store or whatever) and saying, ‘You have 30 minutes, and not one more’ to manage the cost of the purchase.”
But how can you make sure your kids, errr development team, actually get their work done in time? Well, according to Ottinger, you can’t do it by simply pushing speed, velocity, over other factors. “Naive managers try to push velocity, which is the wrong way to go. Simply putting pressure on velocity will drive developers to cut corners and release messy, incomplete, partially-tested code. Instead of pushing velocity managers need to increase the capacity of the team or the workability of the system. It is a different way of thinking.”
The Agile project managers’ way of approaching problems, explains Susan Elliott Sim, assistant professor at the Dept. of Informatics at the University of California, Irvine is to “divide the work into very small units to mitigate risk, enable course corrections, and constrain scope.” She adds, “These points are especially salient when you look at Agile in the broader context of contemporary and historical approaches to software project management.”
How does this work? Angela Druckman, a Certified Scrum Trainer at CollabNet, says, “Agile project management uses the power of iterations to reduce project risk. Rather than making a lot of predictions and estimations upfront that are likely wrong, an Agile approach allows us to use the inspect-and-adapt cycle to make the best decisions possible with the information we have.”
Who is this we? The project managers? The head programmer? In Agile, the development team isn’t just a project manager and some programmers.
As Laurent Bossavit, an Agile Alliance board member and director of the Institut Agile says, “Most previous approaches to managing the software development process were based on control. Agile is based on letting go. Paradoxically, that works a lot better and ironically provides much better actual control (in the sense of avoiding many of the ills that plague software projects, such as missed deadlines, budget overruns, poor quality, and low user and sponsor satisfaction) than previous top-down, document-heavy approaches.”
Sim concurs. “Instead of sticking to a contract to make sure that the worst case doesn’t happen, customers and developers work together to try to make the best case happen.” With this approach, there is a re-distribution of authority and responsibility among management, developers, and customers, she says. That, in turn, means everyone, not just developers and project managers, need to see software development in a new light.
Bossavit continues in concrete terms, “People who are interested in Agile tend to be keen on ‘practices’ — what people do on a day-to-day basis, as opposed to what kind of documents they write. There are a dozen or so key practices for which Agile is best known — frequent releases, short timeboxed iterations, ‘user stories,’ programming in pairs, test-driven development, visual management with task boards — and beyond that a few dozen ancillary practices.”
One of those practices is testing. Indeed, from where Jonathan Bach, Principal Consultant at Quardev, a software testing firm, sits, “The main difference between Agile and other development methods, is that ‘testing’ happens at the same time as code is written, allowing fast feedback. With ‘waterfall’ style methods, first the code is developed and then it is tested — two activities which could be months apart.” In addition, “code is delivered a lot more frequently, programmers pair up, people tend to work closely together (in the same room), there are brief stand-up meetings to discuss progress, the work is packaged into ‘stories’ or smaller units, and there may even be an on-site customer to participate in development and testing.”
So the name of the game in shifting to Agile is to incorporate testing and quality assurance into the development cycle. Instead of putting it the project’s end, as is so often the case, it’s a constant part of project development.
The end result, writes Elisabeth Hendrickson, one of the principals of Quality Tree Software and a well-known Agility expert, is that “Agile teams produce a continuous stream of value, at a sustainable pace, while adapting to the changing needs of the business.”
