5 things to know about deploying big data systems in data containers

1. Explore the data container communities

Brian Gracely

Most data container platforms and Kubernetes are completely open source technologies. Consequently, they have large, vibrant communities supporting and using them.

"The first thing anyone should do is check around the community message boards, GitHub sites, Slack channels and blogs to see if the thing they want to accomplish has already been done before," Gracely said. There's a good chance that a lot of experience already exists.

Projects such as Radanalytics, Kubeflow, TensorFlow, PyTorch, JupyterHub and OperatorHub are great places to get started or find existing integrations to make your use case successful. There are also several great free online tutorials about OpenShift, Kubernetes and containers, Gracely said.

It's also worth looking at how other companies have spun up their big data systems in containers.

The ChRIS project at Massachusetts Open Cloud is solving health issues using a large volume and wide variety of data sources. UPS is running analytics on the edge for delivery trucks. And ACI Worldwide is using containers for doing analytics for credit card fraud detection.

2. Plan for security

Containerization is a new concept that is being adopted very quickly, which can introduce a variety of new security vulnerabilities. As a result, organizations need to train people to not only deploy data containers, but also to identify these new vulnerabilities.

Ahyoung An

"Security and monitoring should never be an afterthought when deploying big data apps on containers," said Ahyoung An, senior product marketing manager at MuleSoft.

An said it is crucial to identify and mitigate issues within containers as they occur, as a breach of containers could mean unauthorized access to an app, a breach in sensitive data or vulnerabilities to key systems.

The best way to ensure security-by-design is to use the application network that naturally emerges when a business packages up its applications, data and devices as standardized APIs to make them pluggable and reusable. These APIs can then be deployed in data containers, either together or separately, depending on business needs. When they are plugged into an application network, containers can be designed in a way that show exactly what data applications might expose, so they can be turned off in the case of a breach.

3. Plan for container APIs

With containerization, users can package their big data applications with the tools and resources they require. Any microservices used can be deployed within the data container, creating an isolated ecosystem that cannot be affected by downstream processes.

Once created, these containers can be plugged into an application network as APIs and moved across any cloud or data center.

"These microservices containers further isolate your big data application from becoming a single point of failure, providing logical entry points to different aspects of your application," An said.

4. Integrate big data and app infrastructure

Christopher Crosbie

Deploying big data apps on containers offers advantages to organizations that are looking to increase isolation, flexibility and portability. Having a single unified interface to cluster environments is a draw for using Kubernetes. Additionally, by running a Kubernetes cluster for non-big data workloads, but not at full usage, data science teams and developers can tap into those unused workloads to increase overall resource efficiency across the organization, said Christopher Crosbie, product manager at Google Cloud.

Developers may want to explore best practices for creating custom configurations at the OS level to make it easy to support both application and data containers on the same infrastructure. They will also want to find ways to package containers with libraries for applications, which is handy for more targeted upgrades and migrating workloads.

5. Watch out for infrastructure sprawl

"Big data systems that are containerized and use Kubernetes don't magically solve all issues," Crosbie said.

For example, if data teams are using YARN, adding Kubernetes for big data containerization represents another cluster environment to manage. Additionally, audit logs, application networking and distributed stateful data represent additional areas that teams should be aware of before deploying big data systems in containers.