When DevOps first began to reach mainstream enterprises over the last decade, some tech industry observers predicted that advanced IT automation, software-based infrastructure abstractions and everything-as-code would eventually eliminate internal enterprise IT operations specialists, a concept dubbed NoOps.
About five years and one pandemic later, the future has turned out to be more IT operations-driven than the past, not less, according to Red Hat executives at the company's annual Summit conference, held virtually this week. And that means a disconnect has emerged between developer-driven software trends and real-world customer environments.
"Right now, all the operational expertise for running software is removed from the production of the code," said Red Hat CTO Chris Wright in a keynote presentation. "We have to bridge the gap between the knowledge and tools that support effective automation at scale [and] the community that's producing the software."
To that end, Red Hat donated more than $500 million in software tools to Boston University's Hariri Institute for its Open Research Cloud initiative, where Red Hat engineers and academic researchers will determine how best to operate cloud applications at scale.
What these teams learn will be used to make open source software easier for enterprises to consume, and help polish new hybrid cloud services Red Hat unveiled this week, according to company executives.
Red Hat CTO Chris Wright gives his keynote address at the company's annual Summit conference this week.
"What do humans need to do to manage [cloud] at scale … and what technologies can we develop to make the job simpler?" said Red Hat CEO Paul Cormier during a media Q&A session. "You can't do this in the lab -- you have to have a large-scale cloud in operation."
Another growing challenge as cloud-native apps proliferate is that real-world cloud computing environments are far from homogeneous, Cormier said.
"Many big customers told us they were going to go 100% 'all in' on public cloud," he said. "That was five years ago, and they're maybe 20 to 25% of the way there."
Thus, instead of neatly transferring a majority of their IT operations burden to a service provider, companies have incurred a new one in managing a mixture of on-premises, cloud and SaaS environments.
Paul Delory
Early in the DevOps trend, it was relatively simple for developers to deploy and operate the first instances of cloud-native operations in test, development and limited production environments. But as these applications transitioned to production deployments, it turned out IT operations couldn't just be abstracted away, said Paul Delory, an analyst at Gartner.
"A lot of IT shops have found out the hard way that ops is still important, whether that's because they tried to outsource operations to a third party, or because they bought into the NoOps hype, which was always a pipe dream," Delory said. "Somebody needs to understand the complexity underneath the abstractions."
Services bridge IT operations skills gap -- at first
A skills shortage in rapidly evolving cloud-native technologies such as container orchestration and distributed data management only widens the gulf between the enterprise status quo and fully automated cloud-native nirvana.
"The job that operators do is very different from what it was 10 years ago," Delory said. "There are labor shortages at the top of the job market, because there aren't enough people with modern skills [such as] Kubernetes, automation [and] data integration. But at the bottom end of the market, there's a surplus of people with legacy skills [such as] data center maintenance [and] ITIL where demand is declining."
For the Electrical Training Alliance (ETA), a not-for-profit job training company for electricians in Bowie, Md., the new Red Hat cloud services will help the company's small IT team with the IT operations learning curve as the company deploys multi-region Kubernetes, API management, Tekton event-driven pipelines and Argo CD GitOps. But "NoOps" is not the company's goal.
"[Red Hat services] can get you started, but I don't think they're looking to be a long-term resource, at least in my experience," said Stephen Boyd, an IT architect at ETA. "But we have four greenfield tech preview things we were trying to get up and running all at the same time."
This wasn't feasible with Boyd's two-person internal DevOps team and a 15-person staff of independent contractors ETA hired for IT operations. But as Boyd and his colleagues get more comfortable with the technologies, they'll use their experiences with managed services to hire and train more internal IT operations staff.
Ultimately, there are limits to how effective developers can be in managing the full infrastructure stack. Boyd said he discovered that during ETA's transition from a single self-managed OpenShift cluster to multi-region cloud application deployments last year.
Developers many know how to consume an application like Kafka, for example, but they can't necessarily set up Kafka for resiliency at scale.
Stephen BoydIT architect, Electrical Training Alliance
"Developers can push an app to Amazon's cloud or spin up their own OpenShift cluster, but as the team grows and we need to deploy in more regions, that's where ops comes in," Boyd said. "Developers may know how to consume an application like Kafka, for example, but they can't necessarily set up Kafka for resiliency at scale."
To learn those skills, Boyd's team is also considering Red Hat's OpenShift Streams for Apache Kafka service, released in preview this week. But managed services have also come with their own initial growing pains, Boyd said.
OpenShift Dedicated, for example, has two infrastructure options -- a bring-your-own-cloud option called Customer Cloud Subscription, and Standard, in which the underlying cloud infrastructure is fully managed by Red Hat.
Boyd chose the fully managed option but found it left him without enough access permissions to OpenShift clusters to use some features of upstream Kubernetes Operators. Red Hat support helped him overcome that issue, and Boyd predicted Kubernetes Operators would evolve upstream to accommodate such scenarios.
Turning over infrastructure to Red Hat also didn't mean turning a blind eye to resource allocation, Boyd added. The API management service, for example, requires 50 GB of storage -- for ETA, that amounted to 50% of the storage initially purchased for its OpenShift Dedicated cluster.
"We were able to solve it pretty easily by just buying another bucket of storage," Boyd said. "But it's important to still be aware of those requirements and total cost of ownership up front."
Red Hat expands IT operations analytics
Red Hat also updated its IT operations analytics tool, Red Hat Insights, to monitor OpenShift Kubernetes infrastructure and Ansible Automation Platform environments this week. Previously, the tool had been built into the Red Hat Enterprise Linux (RHEL) OS.
For one Red Hat customer that hasn't yet moved to containers or Ansible automation, this expansion adds assurance about using those products in the future.
"The Insights expansion will make it easier to adopt those technologies," said John Mills, an infrastructure architect at Graybar, a distributor of electrical, communications and data networking products based in St. Louis. "Instead of having to have an expert on staff that understands all the industry best practices for those tools, Insights tells you what the best practices are and where your misconfigurations are, and you can just address those."
Insights has already helped IT operations accommodate major changes in its RHEL environment over the last three years, Mills said.
"The growth of content within tools like Insight over the past three or four years [amounts to] an insane amount of code additions," Mills said. "What started out as an OS-centric [set of issues] for things like, 'I don't have enough swap space on my VM' has turned into [more complex issues such as] 'Your database user is misconfigured on how much shared memory they can access.'"
Red Hat must now keep up with similarly rapid change across three platforms, and not just for infrastructure management but for cloud cost monitoring and security, which are also features of Insights monitoring. Many of those features remain under wraps for newer platforms -- for example, Red Hat Insights security monitoring for OpenShift and Ansible won't be available until an undisclosed later date.
"I'm very interested in Insights being integrated with Ansible, but I have to note that a lot of the Ansible features in this announcement are future roadmap items," said Gartner's Delory. "We need to reserve judgment until we see what actually ships."
Paul Delory
