Value stream management tames DevOps chaos at Eli Lilly
The pharma firm straightened out disorganized DevOps processes, which now include data management for COVID-19 clinical trials, with a Plutora value stream management tool.
Pharma giant Eli Lilly discovered that simply using software delivery automation tools doesn't bring about IT efficiency -- achieving that goal required a top-down view and detailed measurements of IT and business processes, a practice known as value stream management.
Eli Lilly began searching for a way to sort out multiple DevOps pipeline tools and processes five years ago, as the company's use of tools such as Heroku PaaS, Atlassian Jira application lifecycle management and CloudBees Jenkins continuous integration tools increased, but so did software testing and delivery delays.
Despite the use of IT automation tools such as Chef, procuring a software testing environment could take up to 30 days, and deployment processes often became bogged down in compliance-related approval processes.
"Our environments were managed via Excel, and even the Excel [spreadsheets] didn't make sense," said Marvin Stigter, head of platform services in the test management office at Eli Lilly. "And then we had project managers sitting up in the middle of the night, coordinating [jobs] between onshore and offshore [teams]."
That year, Eli Lilly execs became aware of Plutora, and the company eventually chose to deploy it over a competing value stream management product from their incumbent IT service management vendor ServiceNow.
Value stream management is an IT product category created by Forrester Research analysts in 2002. The term stems from value stream mapping, a practice with a long history in Lean manufacturing environments such as Toyota.
Marvin Stigter
Value stream mapping documents the repeatable steps required to deliver a product or service to a customer, and then analyzes them to make improvements. Value stream management tools apply these principles to software delivery, measure the performance of DevOps teams against improvement goals, and in some cases, orchestrate DevOps workflow automation.
Value stream management is a growing product category. In its third-quarter 2020 Wave report on value stream management tools, Forrester assessed 11 products, from vendors that also included Digital.ai, Tasktop, Targetprocess, IBM, ConnectAll, CloudBees, Atlassian, GitLab and Blueprint.
Eli Lilly evolves from DevOps efficiency to COVID-19 data management
It took IT pros at Eli Lilly 18 months to master Plutora's value stream management products, Stigter said. The process included creating some custom webhooks to integrate earlier versions of the Plutora product with third-party tools such as Jira. However, once that was finished, the Plutora tool had a transformative effect on Lilly's software delivery, Stigter said.
"Since that time, we have not had one bad release go out," he said. "It was way clearer to everybody how to make the right decision at the right time than before, [where] our release schedule was nobody knew about it until something went wrong."
It was way clearer to everybody how to make the right decision at the right time than before, [where] our release schedule was nobody knew about it until something went wrong.
Marvin StigterHead of platform services, Eli Lilly
Value stream management measured the time it took to complete certain tasks, such as standing up a testing environment or signing an approval request. It also pinpointed the cause of lags. The Plutora tool sent automated email reminder messages to people about 15 minutes before they were expected to contribute to a process, in order to keep pipelines running on time. Plutora also added automated schedule adjustments to optimize these processes, rescheduling certain tasks and updating the right people with notifications to make delivery as fast as possible.
"You do have to be careful with the emails ... you can use it the wrong way as well and get overwhelmed," Stigter said. "But if you implement [them] correctly, it's a huge timesaver."
In fact, value stream management also allowed Stigter to calculate precisely how much of his team's time was saved. Building a new test environment now takes five hours at most, compared to the previous maximum of 30 days. Blackout periods for software updates shrank from as much as two full days down to between two and four hours. Between optimized team workflows and automation bots deployed via Plutora, Stigter estimates the company has saved about $16 million per year since 2017.
This year, Eli Lilly has begun to expand its use of the Plutora tool beyond the software delivery lifecycle to workflows such as clinical trials of a potential treatment for COVID-19. Plutora helped streamline the process of loading report data from clinical trial sites into a data warehouse, tracking how many reports have been loaded and visualizing the inflow of data for business and IT stakeholders.
"Our customers in our senior management [now] get a higher level of detail with what's going on, so we kill all the manual communications saying, 'Hey, where are you? What's going on?'" Stigter said. "When COVID came down, we had [study data] uploaded within two days, which had never happened before. Normally, at a minimum, it's five weeks."
Plutora reveals value stream management roadmap
Stigter and his team have begun to beta test new Plutora product features set for release later this year, including enhancements to how the tool orchestrates multistep automation tasks.
"If you have code in Chef that you want to kick off, now somebody doesn't have to do it manually," Stigter said. "If you have two or three different automations, one after another, [Plutora] will now also go automatically to the next one."
Those multistep automation triggers have been inconsistent at times during tests, Stigter said, but continue to improve.
The upcoming Plutora release will also revamp how Plutora integrates with third-party data sources, replacing a RESTful API architecture with an event-driven system for faster data ingestion, with less custom integration required. This is similar to plans for IT automation vendor Puppet's Relay product, which also aims to streamline IT automation workflows.
Stigter said he looks forward to faster, even real-time, data flows into Plutora's dashboards as a possible result of the event-driven overhaul.
"[The] reports are just not real-time enough, and that's really the lifeline of the tool," he said. "[Without it,] if I say, 'We saved a lot of time during this release over the weekend,' nobody really understands what that means if they were not involved with it."
Dig Deeper on Systems automation and orchestration
Marvin Stigter
