How Covad made the switch to a distributed agile development process

While she doesn't label it a "death march," Cornelia Pool, now CIO of Covad Communications Group, inherited a software project that had been grinding along for 18 months with no implementation in sight.

Cornelia Pool, CIO, Covad

Covad, an Internet, broadband and VoIP provider based in San Jose, Calif., is primarily a Java shop working with the Eclipse IDE, and the project was a reimplementation of its operational support system (OSS) from an Oracle platform to Java. That was a big part of the challenge, she said, with a lot of requirements needing to be redefined.

The first thing she looked at was the software development process -- it was waterfall and it wasn't working. But this project was not an anomaly. "Prior to 2006, we did not make any of our releases on schedule," Pool said. "Business couldn't bank on anything [the] software [group] committed to."

Covad was outsourcing some of its development, "but we could really not pull releases off with good quality, on time, and within budget," she said. "We'd get close to a release and nothing worked, which is very typical of a waterfall model. You can't keep track of what you've got."

Also typical of waterfall, she said, were the causes of some of the problems: "Project management was done from a push/pull perspective -- project management had to drive the teams, and nobody owned the projects per se." Scope creep naturally went along with this, Pool said, "because it takes so long to get to a certain point that the business has already had to change requirements." These issues, she said, affected both in-house and outsourced teams.

More on agile development Agile aims to bridge software requirements communications gap Making agile development work for distributed teams How IBM Software Group moved to agile

One process change they had tried was to implement Rational Unified Process (RUP). "It didn't go well," Pool said. "It reinforced a lot of waterfall; it was very architecture-driven and we got stuck in every phase of a project indefinitely."

When Pool took over the OSS project, the agile development methodology was emerging. Pool had read about it, but she didn't know the Scrum process. She and the project manager decided to adopt some agile characteristics, such as doing a standup meeting every day and making people accountable.

In the daily meeting, they would decide "who's doing what work, and what challenges do we face today." She said they focused on finishing functionality. "After every piece of software [was completed], we had a fully implemented piece of functionality the business could get into."

The team was able to roll out the OSS in 12 weeks. That kind of success, she said, "catches some eyes."

Pool subsequently went for Scrum training and took on her next project, an implementation of VoIP. She selected her team and established a Scrum model for development. "We got the offshore folks involved to illustrate it could work with a geographically dispersed team," she said. That project, she said, was the building block of moving the entire organization to Scrum, which she did after she was promoted to VP and CIO.

"Of course, the first principle of Scrum is co-located teams, which I didn't have the luxury of. That's where innovation comes in," Pool said.

Adapting Scrum to their organization required changing vendor relationships and establishing means of communications for their teams to self-manage projects.

Pool had worked with outsourcer Persistent Systems on the initial project and found them "willing to abandon the ways a traditional vendor relationship works," which was establishing project specs and time lines upfront and both sides signing a contract. While Persistent is Covad's primary outsourcer, Pool also utilizes the outsourcer Cognizant.

The first relationship change was socializing the outsourcers to the idea that they were part of the team and the entire process, from analysis to design to scoping, and she expected those teams to be self-managing. "You can't throw it back on-site if you're blocked; it's part of [the outsourced team members'] responsibility to remove the obstacle. So we were changing their background to being part of team. We were no longer going to give you specific instructions; you have to participate."

It took some time to change this culture, she said, with the offshore folks initially reticent to speak up. But, she said, "we encourage them to challenge our design decisions, and for them to understand here's the environment to participate in. We wanted the teams to be successful, which means the business accepts the software."

Another part of the vendor relationship she needed to change was the traditional status reporting. Pool wanted to know "what did you do today, what you're doing tomorrow, what you're stuck on. I want easy, straightforward communication between the team. I look at burndown charts and daily standup meeting notes -- that's all I need to know about where projects are."

It's now up to the teams to take charge of daily communication and use whatever mediums work for them. "It's not for me to prescribe," she said. "It's up to me to make sure we have Web conferencing, bridge lines, etc., available, and help them think through how to resolve some of these issues. The teams quickly figured out they're not all going to do it the same way."

In terms of tools, Covad started out using Excel for the burndown charts, but eventually wrote a Web-based tool for that, with few bells and whistles. "We really believe in keeping things simple," Pool said. They also use a source code control system from Perforce Software.

Pool said her group is responsible for seven releases a year, hence seven sprint cycles. A project begins with a cross-functional team that sits down with the business side to break down the functionality and requirements and then prioritize them. "Then we look at what resources we have available, see how big a team to put around this, scope size and price it, and then say it will cost so much, and it will take us so many iterations or sprints. The business [side] adopted this pretty quickly. They understand you have to prioritize and when money is done, the money is done. They get this like they didn't in waterfall. When you prioritize you realize you may not need something [you thought you did]."

Because Covad's OSS system consists of hundreds of software components and is designed to run end to end without human intervention, Pool said they release the projects all at once, after they go through integration testing. While she said quality assurance (QA) is involved more and more in the development project, she does have what she calls a regression team for this task. She was able to reduce the size of this team by working with Persistence to develop a suite of automated test cases. "Automation is key to ensure the overall quality," she said.

Like many IT organizations, Pool said there has been pressure to cut costs. They have replaced older hardware, reduced staff and removed applications that were costly to maintain but didn't deliver much ROI. However, in the three-year period since adopting an agile process that includes both in-house and outsourced people, her group has chalked up 23 on-time, successful releases and cut costs by 50%.

Refining their processes and reinforcing how the team is expected to work is critical, Pool said. Sometimes, she said, she'll see standup notes that start to look like status reports, "so you bring people constantly back to how you're supposed to work, until it's so embedded they know no other way to do it."