At OSIsoft PI World, IT/OT convergence a hot topic
Building IoT connectivity with a PI Server across multiple plants to improve analytics and efficiency is a great goal, but achieving it isn't without its challenges, as evidenced at OSIsoft PI World 2018.
In many enterprises, IT teams and operations technology groups have typically lived in different worlds -- and for good reason. No manager wants to take the blame when a new security vulnerability allows hackers to break into a power plant, factory or water system.
However, as new technologies and legacy systems combine to create internet-of-things systems, the responsibilities and overlap of these teams are blending. And it's becoming evident, as seen at last week's PI World conference in San Francisco, the success of IoT lies in enterprises' ability to successfully achieve IT/OT convergence such as is enabled by the PI Server that lends this conference its name.
Before IoT was even a thing, real-time data management company OSIsoft, which hosts the PI World conference, began creating an intermediate tier for staging data from operations technology (OT) systems in the 1980s. This made it easier to share information across a single plant in a reliable, secure and governable manner.
This core technology, called a PI Server -- PI was originally an acronym for Plant Information -- sits on top of the often proprietary supervisory control and data acquisition (SCADA) systems running on various pieces of industrial equipment to help provide a unified view across the plant. PI Servers have helped engineers make better decisions and simplified the ability to write accurate reports for regulators and business executives. OSIsoft has focused its efforts on building integrations for virtually every piece of industrial equipment and has become a common integration tier used by major OT software providers.
With the emergence of IoT, the benefits of the PI Server are becoming more evident.
"PI is the only game in town when it comes to data integration," said a chief engineer at a major fiberglass manufacturer, who was at PI World. "Other vendors have some integrations, but PI plugs right into all of the equipment across our plants."
Tim Alexander
"The PI System is a huge part of our infrastructure for capturing and staging data, and one of its strengths is that it is good at bringing data in from and pushing data to other tools and systems," said Tim Alexander, brewery operations technology manager at Deschutes Brewery in Bend, Ore., in an interview at the conference.
Deschutes Brewery brings in data from multiple SCADA systems, as well as from various databases. It combines and looks at all this data in PI and also pushes it up to its Microsoft Azure cloud to perform predictive analytics. "Having a tool that can reach out to all of these systems and enable us to compare data directly between them is invaluable to us," Alexander said.
IT/OT convergence: Growing beyond a plant
The vast majority of users have focused on improving their ability to aggregate information at a single plant level. The big theme at this year's PI World involved finding ways to build connectivity across multiple plants to improve analytics and efficiency, resolve problems more quickly and replicate best practices across different facilities within a company.
There is a lot of paranoia about shifting mission-critical data off premises.
Andy Castonguayresearch director of IoT, Gartner
"There is a lot of paranoia about shifting mission-critical data off premises," said Andy Castonguay, research director of IoT at Gartner. "There is a lot of experimentation from many companies, but it is not well-orchestrated in terms of pulling that together, so they end up with multiple point solutions running in parallel."
In many organizations, there are different groups wanting to push forward without top-level endorsement. Yet, they see benefit in coming up with a plan for IT/OT convergence to pull their different efforts together. "There is some remorse in not having paused, given it some thought, and brought in reliably and trustworthy sponsors and vendors and thinking it through in a more collective way," Castonguay said.
The San Francisco Public Utilities Commission (SFPUC) started down this path with a simple application for improving incident response for its sewer systems. In a presentation, Max Chung, distributed control system engineer with the SFPUC, said the organization was having issues notifying biologists when storm runoff water entering the bay and ocean was exceeding certain thresholds.
The SFPUC had implemented an elaborate monitoring system, which kept tabs on water quality across its vast network. Unfortunately, this data was locked inside the SCADA systems and could only be directly viewed by water plant personnel at dedicated workstations. "The older system was dependent on an operator noticing an alert while they might be busy doing other things, and then sending out a manual request." Unfortunately, some of these alerts were never sent in time for the biologists to take effective action.
The SFPUC worked with data monitoring and aggregation company DST Controls to set up a dedicated server for aggregating these alerts. This made it easier to stage the sewage plant data outside of the control network and opened it up to work with other applications. This was connected to an enterprise message access gateway (EMAG) service operated by Verizon. The EMAG provided a much higher level of reliability and accountability than is possible with a public SMS gateway, said Greg Dumas, CTO at DST Controls, based in Benicia, Calif.
The EMAG also allowed SFPUC to send messages longer than 140 characters to biologists. Once notified, biologists could log into the web application for finer-grained details about the incident, so they could then take appropriate action in a timely manner. In addition, this infrastructure also allowed SFPUC to automatically send notifications to executives and local regulators without burdening humans to aggregate the data and file the reports.
The results have been good so far. "In the past storm event in April, we doubled the notifications sent out compared to the manual system," Chung said. "It's been a great success to our regulatory group and everyone in San Francisco."
Renaming things is hard
Naming remains a big challenge for large enterprises as they scale their IoT data infrastructure outside of single facilities and achieve IT/OT convergence. Over the years, different types of OT engineering groups have invented their own naming schemes for devices relevant to different kinds of reports or engineering challenges. These differences can be even more significant across plants, which makes it difficult for executives and engineers to get a clear picture of performance differences, said Bob McIntyre, senior process control engineer at Nutrien Ltd., a large fertilizer enterprise in Saskatoon, Sask.
McIntyre set out to create a single data repository for the company by aggregating IoT data from the different plants. This was no easy task, as each plant had thousands of pumps and other types of industrial equipment. The first stage of this process involved collecting data from different systems into silos on site and putting them into one local database. Nutrien used a PI Server to aggregate the data outside of the plant control network. "Business people need this [data], and we don't want them reaching into the control network," McIntyre explained.
When McIntyre moved to the head office, he established one central PI Server already running that was not being used for any meaningful analytics. The key was to develop a tag-naming standard for describing different kinds of equipment and translating each plant's local conventions into the new standard. "There was a lot of grumbling in the beginning, but everyone did it. And afterward, they loved it," McIntyre said. Eventually, the local plants were able to turn off the local servers, saving them the trouble of maintaining them.
This process of standardizing the IoT device tag names required a lot of upfront effort. But in the long run, McIntyre found it saved that time back many times over in building and maintaining the database. He discussed implementing a tag hierarchy that works on top of the PI Asset Framework, a repository for asset-centric models, hierarchies, objects and equipment. A good hierarchy brings structure to the way tags are organized, but it can be challenging to create the best hierarchy. He cautioned that each company needs to work through the hierarchy that works for their organization. A good approach is to build a hierarchy structure, try it out, note what does not work and then change it. "By the time you get to version four, you will get to something that works for you," McIntyre said. "But once we built that platform, we could stand up other sites quickly."
One of the major benefits of bringing this data together at the enterprise level and achieving IT/OT convergence is Nutrien can better identify opportunities for improvement. For example, it can now compare power usage and production across days. In addition, there can be differences in the production of fertilizer caused by changes in the raw materials that are mined from the earth, humidity and temperature changes -- this can now be more easily monitored.
McIntyre noted that getting all the data into one database in a consistent data structure is a prerequisite for more sophisticated predictive techniques using tools from Esri, Microsoft Azure and SAP.
Many of the PI World attendees were asking for more specifics about the tag-naming schemes. One concern was it might be difficult to update some plants without bringing them offline.
Heather Quale, president of Mera Development Corp., an engineering services firm that worked with Nutrien, suggested that one solution was to temporarily run multiple tagging infrastructures. Once the second one was in place, then the enterprise could quickly cut over to it, without losing data.
Tim Alexander
