Arcitecta leverages metadata to move critical data faster
Arcitecta has found a way to increase the data transfer speed of its Mediaflux Livewire software, which can be valuable for projects that demand significant collaboration.
Arcitecta has updated its software so that large amounts of data can be transmitted over networks that are either slow or unreliable, enabling collaboration without interruption.
Arcitecta Mediaflux Livewire, initially released in 2022, can now break large data sets up into smaller pieces, use metadata to prioritize what data gets transmitted first and then send the smaller pieces in parallel rather than as one large chunk.
By doing this, the software can make use of slower networks anywhere around the globe. This will allow collaboration and transfers of large files even when the connections are subpar. Mediaflux Livewire is part of the Mediaflux data orchestration platform or can be purchased separately. Price is based on the number of concurrent users.
Organizations need ways to securely transfer large volumes of data, and utilizing slower networks can open overlooked paths to help companies do that, according to Ken Clipperton, an analyst at Data Center Intelligence Group. While there are alternatives out there, an event like Progress Software's MoveIt Transfer breach might create room in the market and have customers looking for new options.
"The ability to move a lot of data over what might be marginal connections as well as to move data fast over great connections is the result of the parallelization of all the elements of the data path," Clipperton said. "That's what Arcitecta is doing with Livewire."
Clipperton likens the parallelization that Arcitecta is doing to the parallelization of phone signals in early modems to increase speed on existing lines. When the top speed was 2,400 bits, early modem pioneers were able to send signals concurrently over 512 channels rather than serially, getting seven to 15 times the data through than was possible before.
Mitigating migration issues
The new features use visual representation at each site to reduce network traffic between transmission sites, automatically tune compression to decrease the data transmitted, offer reliable packet retransmission in the event of a connection failure, and provide network performance monitoring for real-time flow and network health, according to Arcitecta.
The ability to move a lot of data over what might be marginal connections as well as to move data fast over great connections is the result of the parallelization of all the elements of the data path.
Ken ClippertonAnalyst, Data Center Intelligence Group
Livewire uses metadata -- key information about other data that improves organization and searchability -- to better orchestrate data movement. Based on the metadata, users can see that they might only need three of the 18 packets. That data can be moved first with the remaining packets coming later.
Metadata-enabled workflows are part of Arcitecta's overall platform, Clipperton said. Livewire is an offshoot of the larger data management fabric, Mediaflux, which also relies on metadata for faster tracking and querying. Actions can be triggered based on changes in metadata -- for instance, acting as an indicator that an image has been changed on the other end of a transmission site. The metadata is also easier to transmit.
"Obviously, metadata is a lot smaller than data," he said.
While metadata is an important factor for data transmission, Clipperton said, so is parallelization and efficiency for moving the data.
Collaboration isn't for everyone
But networking cables are confined by their size, only capable of moving so much data at a time. Moving petabytes or exabytes of data across the network will still take time, and instantaneous transfers of entire data sets are still a ways off, according to Mitch Lewis, an analyst at Futurum Group.
Some of the largest, fastest data movements use old-fashioned technology. AWS Snowball, for example, stores data to an external device and then ships the device to its new location, Lewis said. But the use cases for Snowball are meant more for a one-time move. Livewire is aimed at moving data more regularly for big projects where data must be transferred to those who need it when they need it. Use cases include geospatial, oil and gas, and medical imaging projects.
"[Arcitecta has] this overlaying data orchestration layer that moves things where they need to be so you can collaborate," he said.
But Livewire isn't for everyone, Lewis said. Large scale data environments that need data management and data movement can utilize Livewire. But smaller organizations are less likely to be moving petabytes back and forth between continents.
Regardless, Livewire does lay down a potential path for future data migration, Clipperton said. Cable networks will still be physically limited in how much data can be moved at a time, but parallelization and metadata will make the process more efficient and faster.
"There is a lot of power in parallelizing data the way NVMe does for flash storage," he said. "It's a lot about multiple parallel channels."
Adam Armstrong is a TechTarget Editorial news writer covering file and block storage hardware, and private clouds. He previously worked at StorageReview.com.
