Scottish EPA rebuilds with Tibco Spotfire after cyber attack

Starting over

The cyber attack on SEPA occurred on Dec. 24, 2020. Following the attack, the agency lost all access to internal resources, including large historical data sets, its catalog of about 200 internal Tibco Spotfire reports, dashboards and data models that had been developed over an eight-year period beginning in 2012, and to its other analytics-related capabilities.

However, the loss of the vast majority of the analytics tools SEPA used to monitor the Scottish environment did not mean the organization could suddenly stop operating.

Fortunately, one instance of Tibco Spotfire survived.

The organization's public-facing resources were unaffected, and that included one instance of Spotfire that enabled SEPA to meet its obligations while it redesigned and rebuilt its catalog of analytics tools.

"This was more than lucky," Claire Neil, senior scientist at SEPA, said recently during a session at Tibco Analytics Forum, a virtual conference hosted by the analytics vendor. "SEPA still had its legal obligations as Scotland's environmental regulator, and having an instance of Spotfire allowed the informatics unit to build and develop interim solutions."

Interim systems included tools for lab management, workload planning and tracking, emission-zone data models and compliance verification, she continued.

But more than enabling SEPA to withstand the cyber attack, the agency's surviving instance of Tibco Spotfire enabled SEPA to rethink its analytics strategy and build a better analytics program, one started in 2021 with modern capabilities and the experience gained over nearly a decade using data to inform decisions compared with one started in 2012.

When the dust settled from the cyber attack, we realized we had an opportunity to use our wealth of knowledge and experience to build back better.

"When the dust settled from the cyber attack, we realized we had an opportunity to use our wealth of knowledge and experience to build back better," Neil said.

Before the cyber attack, because SEPA had so many analytics tools, it was becoming increasingly difficult for users to identify the right ones for their work. But because there were no technical problems with the agency's analytics program, there was no impetus for a complicated, time-consuming overhaul.

The cyber attack, however, left SEPA with no choice.

"We were able to learn from past mistakes to design more effective tools and systems," Neil said.

Among them were a system to monitor bathing water quality.

System refresh

Among SEPA's many responsibilities is to provide water quality predictions for 29 of the 87 designated bathing waters across Scotland from June through mid-September (bathing waters are defined as the waters off beaches where 150 or more people are allowed).

Before the cyber attack, SEPA's system for monitoring water quality was complex, according to Roisin Murray-Williams, a senior data scientist at SEPA.

The system collected rainfall data from a gauge network and transferred it through an FTP server. Operators were then able to validate data, build and run models to produce predictions and subsequently post those predictions to signs at beaches and on SEPA's website.

The code, however, was written nearly a decade ago, and the inner workings of the system were a bit of a mystery even to data experts at SEPA. And then after the cyber attack, the entire system and all the code to run it were lost.

Tibco Spotfire's model-building capabilities had been part of the system before the cyber attack but they did not play a role in SEPA's daily monitoring activities, according to Murray-Williams. Afterward, however, Tibco Spotfire became the primary engine for SEPA's bathing water quality forecasting.

"The loss of the established system presented a challenge, but it also presented an opportunity to start fresh," Murray-Williams said. "We could take into considerations lessons learned from the previous system, which was large, complex and somewhat of a black box. Alterations to the system were not easy to carry out and many of the models … threw up some unforeseen issues."

The agency tweaked the system year after year during the offseason, but none of these changes resulted in a true overhaul, she continued.

After the cyber attack, SEPA developed an interactive dashboard with Tibco Spotfire to monitor water quality and make predictions during the 2021 summer season. When the 2021 season was over, SEPA added more depth to its new tool, including automation capabilities that relieve data engineers of the time-consuming responsibility of updating data.

Now, any improvements and new iterations of models and dashboards are much easier to implement, and because they are accessible through a web browser rather than solely kept on premises, they can be used in real time from any location, according to Murray-Williams.

"The Spotfire tool presents the data in a user-friendly manner and an accessible way," she said. "Staff no longer have to open three applications to carry out their tasks -- just one Spotfire tool that does it all. Spotfire initially allowed us to quickly develop a gap-filling product, and now it has been adopted as a long-term solution."

Beyond a new system for monitoring bathing water quality, two other tools SEPA has developed using Tibco Spotfire since the cyber attack in December 2020 enable the agency and its partners to monitor and predict the levels of pharmaceutical pollution in wastewater and use satellite technology to track the surface temperature of the water surrounding Scotland.

Roadmap

As SEPA moves past recovering from the 2020 cyber attack and plans for the future, it plans to take advantage of digital twins through Tibco Spotfire to do scenario planning, according to Neil.

In addition, SEPA plans to add a hub for self-service users to make it easier to find data sets and data assets.

The hub will be created using Spotfire Mods prebuilt applications that enable users to develop customized analytics tools such as reports and dashboards without requiring them to write code.

"Spotfire continues to be an integral software package used by SEPA," Neil said. "It really saved the day in the immediate fallout of the cyber attack. Now, we are building a newer and better system that takes advantage of 10 years of experience and learning from our previous failures and successes. Our plan is to continue to push the boundaries of Spotfire to improve the decision-making process."