Agile for infrastructure requires major cultural shift

Agile for infrastructure requires radical change

But Agile networking isn't always an easy sell, as it often represents a total overhaul of organizational hierarchy, workplace habits and professional mindset.

"A lot of companies still have siloed teams that say, 'I just write the code; I don't deploy the code,' or, 'I deploy the code; I don't test the code,'" said Chris Stetson, cloud architect at Cigna Healthcare.

An Agile for infrastructure transition often means dismantling that traditional IT hierarchy and rebuilding it to align with business objectives and NetDevOps workflows.

McKesson Corporation -- a major healthcare and pharmaceutical company, headquartered in Irving, Texas -- tackled such an overhaul in 2017 with a comprehensive, top-down Agile infrastructure initiative. Following a number of mergers and acquisitions, the company sought to unite disparate teams and deliver network services more efficiently, said Brian Silverman, senior network and cloud architect.

Without any kind of formal Agile for infrastructure blueprint, however, Silverman said the shift required creative thinking and improvisation.

"It was mostly just sitting in a room to ideate: 'OK, well, what is it that we're trying to do here? What would that look like?'" he said. "It was a lot of trial and error."

Ultimately, McKesson decided to dramatically reorganize its networking team to align with Agile, taking out a management layer and adding Scrum masters to drive progress on individual features.

In Microsoft's Agile transition, Nkposong said the company similarly broke down traditional organizational silos and embraced a new DevOps outlook.

It was mostly just sitting in a room to ideate: 'OK, well, what is it that we're trying to do here? What would that look like?' It was a lot of trial and error.

"The technical project managers decide what it is we're going to build, and then the engineers take that what and convert it into how and build it with policy," Nkposong said. "Those are the mindset shifts we had to make."

On a more practical level, Microsoft also restructured its private offices into an open-plan space. Silverman and Nkposong both added that strong executive support was instrumental in successfully aligning infrastructure processes with business goals, with company leaders giving them freedom to experiment.

"Each of our teams had the autonomy to learn what works best for them," Nkposong said. "The infrastructure teams have now actually evolved to behave like software development teams -- we've abstracted the network to a large extent."

Network pros "own" the abstracted configuration-as-code templates, reviewing change requests and acting as expert resources for developers, Nkposong continued.

As Fratto put it: "They're applying CI/CD [continuous integration/continuous delivery] to operations processes."

Prepare for culture shock

But Cigna's Stetson said he has also seen network engineers and administrators wrestle with applying software development concepts to their workflows.

"It's a little bit more of a struggle on the infrastructure side," he said.

Silverman agreed, saying Agile for infrastructure has some inevitable challenges, even in successful implementations.

"Especially in the network, so many of the blockers are totally out of our control," he said. "Carrier lead times, the acquisition of physical hardware, deployment -- all of those things made it very challenging and continue to make it very challenging."

The McKesson networking team now tries to isolate those factors and address them outside of the Agile release train (ART). Within the ART workflow, they limit their focus to tasks they can accomplish reliably and efficiently, Silverman said. Another adaptation: McKesson eventually changed the delivery teams' project management framework from Scrum to kanban.

"Kanban handles things like carrier delays much better than Scrum, which just doesn't map as well," he said.

And Fratto pointed out that, on a technical level, Agile networking has another unique challenge.

"When you deploy an application into a network, you're configuring switches, routers, firewalls, VPNs, load balancers, caching servers and DNS -- probably 12 different things with 12 different vendors or product lines," he said. "Automating across that entire spectrum intelligently and integrating it with CI/CD pipelines is almost impossible. That's where the difficulty lies."

Stetson agreed, predicting that the answer will ultimately come via a combination of IaC, network automation and network orchestration, which is the focus of an ongoing ONUG working group.

"We're trying to build a framework, but it's a real challenge," said Silverman, who is the group's co-chair.

Fratto added, however, that even small successes help an organization's early automation efforts gain momentum.

"It doesn't have to be big, systemwide, 'robotics-gone-wild' kind of automation," Fratto said. "It can be very small, atomic jobs, things that you do daily or weekly on a regular basis that drive you crazy."

Fratto said he recently spoke to a vendor engineer who often works with customers unsure about what to automate first. He suggested they start with whatever takes most of their time, such as the help desk system.

"Let's automate that, and take it off your plate. Just start," Fratto said, adding that the principle applies to both DIY and branded automation tools. "Take those small, atomic projects where you can see success very quickly, gain experience and move on to bigger projects."

Before long, users will accumulate a suite of automated tasks.

"It's a crawl, walk, run scenario, but it becomes very effective," he said.