Lessons learned: Strategies to adjust IT operations in a crisis

Shelter-in-place orders enforce remote work

With the overwhelming shift back to localized work environments in the 2010s, many IT organizations in 2020 scrambled to accommodate new restrictions and complications related to COVID-19. In-person, impromptu discussions and weekly co-located meetings became impossible with all staff offsite, which created bottlenecks and, in some cases, a slow-down in productivity.

To minimize disruption to management and workers alike, establish an official communication platform or tool, such as Slack, Teams or Zoom. Make sure all workers use the same features to guarantee equal access and opportunity. Additionally, deemphasize individual specializations, as knowledge can be lost if the specialist leaves or becomes unavailable. Lastly, adjust productivity measurements to emphasize quality over quantity. Instate solid documentation practices and hold all staff to that requirement, but avoid micromanagement.

Service desk management tasks surge

In the case of COVID-19, managing IT operations in a crisis demanded support for a mass migration from office settings to remote work -- which, in turn, created a swell in service desk tickets. Teams that spent two days per week on unplanned work suddenly spent their full week on it. Additionally, service desk tasks diversified in a new -- also unplanned -- direction, as remote workers have a variety of internet connections and providers to maintain their personal setups.

Consequently, many organizations shifted both their attention and budgets toward service desk and collaboration technologies, and away from new and innovative projects. For example, the DevOps team at Stemcell Technologies, a biotech company in Vancouver, BC, used a low-code platform to create an app that helped managers track employee and IT equipment status across a remote workforce.

SecOps floats around IT organizations' priorities

Security, and in particular SecOps, was a top item on IT shops' to-do lists until the start of the COVID-19 pandemic, when firefighting and optimization replaced it. Those that dropped SecOps to focus on collaboration tools neglected to pair those tools with security best practices. Zoom, in particular, posed a security challenge up-front -- and a significant uptick in service desk requests. To effectively manage IT operations in a crisis, admins must foster a close relationship with security teams to resolve issues accurately, swiftly and, most importantly, securely.

As firefighting becomes less urgent, SecOps is again a priority for some organizations. Still, IT teams must recognize that end users are not at the same knowledge level, and clearly explain the security practices that they must follow, especially while they work remotely.

Essential businesses, essential AIOps capabilities

Essential businesses have stayed open and operational throughout the pandemic -- but with the workforce scattered to the wind by necessity, many weren't prepared to adjust. Some organizations pivoted their attention to incident response to handle a variety of expected and unexpected problems that take time away from other important tasks.

For example, KeyBank, a financial services institution headquartered in Cleveland, began its AIOps journey in 2017. Over time, KeyBank replaced its 21 monitoring tools with a Kafka pipeline-fed ElasticStack repository. The addition of AIOps capabilities, via Moogsoft, reduced alert volume in 2020 by a factor of 10 -- following a 98% decrease over the prior two years.

Signify Health, a Dallas-based provider of care services in the home, continues its search for capable SREs, as the IT skills gap makes this type of senior position difficult to fill. Signify has put a focus on AIOps and IT automation to reduce tedium for its IT staff at large -- as well as alert volumes -- with New Relic One. Although these functions and features can prove difficult to master, Signify Health's ultimate goal is to integrate New Relic One with its Atlassian Opsgenie incident response system.

Puppet drives head-first into IT automation

Despite multiple leadership changes in only two years, Puppet Labs has held its focus on IT automation practices and tools. In line with its core mission, Puppet leadership's priority is now the expansion of automation, interoperability and scalability, as well as its popular open source tools.

Puppet CTO Abby Kearns explores Puppet's upcoming projects and her reasons for joining Puppet in such a time of transition -- and uncertainty. The COVID-19 pandemic changed the game for many of Puppet's customers, as well as for Puppet itself, shortening the time frame for a necessary digital transformation: Organizations that were tracking eight-year transitions now face an immediate need that can't wait. And the industry-wide skills gap? It's only going to get worse.