your123 - stock.adobe.com
CyArk migrates digital archive to Seagate's Lyve Cloud
On a mission to create 3D digital blueprints of cultural heritage sites, CyArk discovered its storage infrastructure was too brittle for the task. It turned to Seagate for help.
When CyArk realized it had a massive amount of data on its hands and a less-than-stellar method for storing it, the organization turned to Seagate for cloud storage and migration tools.
CyArk, a portmanteau of "cyber archive," is based in Oakland, Calif., and was formed in 2003. It creates layered, 3D blueprints of the world's cultural heritage sites such as Pompeii, Mount Rushmore and the Sydney Opera House and archives them to be used as preservation and conservation tools, according to Avidan Fernandez, production manager at CyArk. The nonprofit organization is currently looking to archive 500 global sites, but the data generated by the cutting-edge equipment has become immense.
CyArk collects data using drones and on-the-ground vehicles outfitted with Lidar-like laser scanners invented by founder Ben Kacyra. It also relies on high-resolution, mirrorless cameras to better capture a site's texture. Collectively, the data is used to generate a virtual representation of sites that can be maintained in the event of a catastrophe, Fernandez said.
The data is collected on portable devices, such as SD and CFast cards, as well as Seagate Backup Plus Fast and LaCie Rugged Thunderbolt portable hard drives, and then gets transported back to a storage server for archiving.
Beginning an archive
Since its beginning, CyArk set out to use the data collected to create an archive, which started out as collecting enough data to build "a little glass cube of a site, and then maybe with a 3D model inside," Fernandez said. But after the partnership with Seagate, the nonprofit could build an actual digital archive. However, CyArk was far from that vision in the beginning.
In 2003, a project could fit on a thumb drive or be shared using Dropbox or Google Drive. As cameras improved, the data they generated also grew, Fernandez said, with a typical project averaging between 2 to 5 TBs of raw data that still needed to be processed. The initial storage setup couldn't handle ever-increasing file sizes, nor was it easy for employees to access.
"We didn't really have a permanent solution," Fernandez said. "We had just a bunch of drives all over the place kind of connected to a single computer."
Modernizing with Seagate
At the 2016 National Association of Broadcasters convention in Las Vegas, CyArk employees saw Seagate's presentation on its LaCie 6Big and 12Big drives, leading to a partnership between the two organizations. The nonprofit used a local contractor to set up a storage server in its main office, for which Seagate donated enterprise HDDs. Seagate also donated some of its LaCie Rugged DAS HDDs, portable ruggedized drives designed to be used in the field, and the LaCie 6Big and 12Big drives.
As it turned out, storage wasn't CyArk's only problem. Each project had two employees on site and three rugged portable hard drives supplied by Seagate, a copy for each person and one extra, in case it was lost or stolen. From here, the data would be transferred via Thunderbolt connection to the array in CyArk's office. The process created fragmented data sets, as the three copies could be missing some data.
The data sets were placed in folders on the server that were hard to organize, creating a second data management headache. And then COVID-19 hit, leading CyArk to, in Fernandez's words, "abandon ship" and move to a smaller office away from where the storage server lived, throwing a wrench in the organization's the mission to build a central repository for data.
CyArk again turned to Seagate with this new set of problems it was facing.
"[Seagate said], 'We need to get you guys on the cloud, immediately,'" Fernandez said.
Migrating historical archives to Lyve Cloud
Seagate suggested that CyArk move to its Lyve Cloud, a storage as a service (STaaS) offering that includes data replication and protection as well as transparent and fixed pricing. Lyve Cloud is an S3-compatible public cloud that is accessed through the Lyve Cloud portal, which provides a dashboard for tracking consumption metrics and bucket statistics and provides multifactor authentication. Because Lyve Cloud is an object storage offering, CyArk can take advantage of its ability to scale. Seagate started CyArk off on a 500 TB account.
The move to cloud didn't happen without hitting a snag. CyArk's archive totaled about 300 TB, all of which needed to be migrated to Lyve Cloud, Fernandez said. Initially, CyArk transferred files over its network connection one project at a time, but the flaw in this plan materialized quickly, as each one of CyArk's 250 projects was taking a week and a half to move. Going this route would have taken CyArk almost five years to transfer all of it on-premises data to Lyve Cloud.
"To copy [300 TB] over on a network connection to Lyve Cloud, it's just impossible," Fernandez said. "And then the office that we're currently occupying is a lot smaller, and the internet connection there is awful."
Seagate worked with CyArk to speed up that process. It shipped a Lyve Mobile Array, a mass storage device that came with up to 96 TB of capacity. The device uses either Thunderbolt 3 or USB 3.2 interfaces for up to 2.8 GBps and 620 MBps, respectively. The device also includes Lyve USM, a PCIe 3.0 connection for up to 6 GBPS or 12 GBps dual port, according to Seagate.
Once the Lyve Mobile Array was in hand, CyArk prioritized its top 50 projects and transferred them through a daisy chain of devices to the Lyve Mobile Array. Once finished, the organization shipped the array back to Seagate and was able to access the data through the Lyve Cloud Portal the following day. CyArk replicated the data migration method until the transfer of all 300 TB of data was complete.
Transferring data to an array and then manually shipping it to Seagate's data centers in Oklahoma City saved on migration time and reduced complexity, according to Fernandez.
By centralizing all of CyArk's data in a single location, Lyve Cloud removed the pitfall of data duplication.
"The full advantage of the cloud system is that we don't need multiple folders for a project because we're on a server," Fernandez said. "You can have abbreviations; you can reorder projects so that everyone can access everything without creating custom solutions."
Archiving into the future
Now that all of its data is migrated to Lyve Cloud, CyArk will start looking at setting policies to give its workforce permissions needed to process and store their work without the old risk of interfering with someone else's.
Upgrades aside, Fernandez said there is still room to improve on its current setup.
"[The Lyve Cloud Portal] is very rudimentary right now," Fernandez said. "I can tell you how much data is on our cloud, but I can't tell you how much is stored in a single bucket or a separate folder. It's all kind of just top-level; there's not that much control. I think there could be a little bit more fine-tuning there."
CyArk uses Cyberduck, an open source cloud storage browser that enables users to connect and transfer files to servers and clouds, for data transfers, but would rather conduct the transfers through the Lyve Portal, he said, which currently can't be done.
Still, Fernandez said the customer service it receives from Seagate, especially as a small nonprofit, is something to note.
"When I do bring up some random request, they are very responsive," Fernandez said. "Seagate has not just been making sure our data is secure and available, but actually helping us organize our projects a little bit better."
