DreamWorks animates its cloud with NetApp Data Fabric
The 'virtual studio' DreamWorks cloud relies on various components of NetApp Data Fabric, including All Flash FAS storage, NetApp HCI and OnTap cloud-based data management.
Although it's only been around since 1994, DreamWorks Animation has several blockbuster movies to its credit, including How to Train Your Dragon, Kung Fu Panda, Madagascar and Shrek. To get the finished product ready for the big screen, digital animators at the Hollywood, Calif., studio share huge data sets across the internal cloud, built around NetApp Data Fabric and its other storage technologies.
An average film project takes several years to complete and involves up to tens of thousands of data sets. At each stage of production, different animation teams access the content to add to or otherwise enhance the digital images, with the cloud providing the connective tissue. The "lather, rinse, repeat" process occurs up to 600 times per frame, said Skottie Miller, a technology fellow at the Los Angeles-area studio.
"We don't make films -- we create data. Technology is our paintbrush. File services and storage is our factory floor," Miller told an audience of NetApp users recently.
'Clouds aren't cheap'
DreamWorks has a mature in-house cloud that has evolved over the years. In addition to NetApp file storage, the DreamWorks cloud incorporates storage kits from Hewlett Packard Enterprise (HPE). The production house runs the Qumulo Core file system on HPE Apollo servers and uses HPE Synergy composable infrastructure for burst compute, networks and storage.
Miller said DreamWorks views its internal cloud as a "lifestyle, a way to imagine infrastructure" that can adapt to rapidly changing workflows.
"Clouds aren't magic and they're not cheap. What they are is capable and agile," Miller said. "One of the things we did was to start acting like a cloud by realizing what the cloud is good at: [being] API-driven and providing agile resources on a self-service basis."
We don't make films -- we create data. Technology is our paintbrush. File services and storage is our factory floor.
Skottie MillerTechnology fellow, DreamWorks Animation
DreamWorks set up an overarching virtual studio environment that provides production storage, analytics on the storage fabric and automated processes. The studio deploys NetApp All-Flash FAS to serve hot data and NetApp FlexCache for horizontal scale-out across geographies, especially for small files.
The DreamWorks cloud relies on various components of the NetApp Data Fabric. NetApp FAS manages creative workflows. NetApp E-Series block storage is used for postproduction. NetApp HCI storage (based on SolidFire all-flash arrays) serves Kubernetes clusters and a virtual machine environment.
To retire tape backups, DreamWorks added NetApp StorageGrid as back-end object storage with NetApp FabricPool tiering for cold data. The company uses NetApp SnapMirror to get consistent point-in-time snapshots. Along with StorageGrid, Miller said DreamWorks has adopted NetApp Data Availability Services (NDAS) to manage OnTap file storage across hybrid clouds.
DreamWorks-NetApp cloud
"NDAS has an interesting characteristic. Imagine a cloud thumb drive with a couple hundred terabytes. You can use it to guard against cyberattacks or environmental disasters, or to share data sets with somebody else," Miller said.
The need for storage analytics
The sheer size of the DreamWorks cloud -- a 20 PB environment with more than 10,000 pieces of media -- underscored the necessity for deep storage analytics, Miller said.
"We rely on OnTap automation for our day-to-day provisioning and for quality of service," he said.
A DreamWorks cloud controller helps inform development of NetApp Data Fabric under a co-engineering agreement. The cloud software invokes APIs in NetApp Kubernetes Service.
The vendor and customer have joined forces to build an OnTap analytics hub that streams telemetry data in real time to pinpoint anomalies and automatically open service tickets. DreamWorks relies on open source tools that it connects to OnTap using NetApp APIs.
