Object storage and AI are two hot trends in tech. They exist on their own but work well together too. Still, organizations must also consider the challenges of using both.
Object storage, which accommodates huge amounts of unstructured and structured data in virtually any format, is the storage medium of choice for AI and machine learning systems.
Object storage has several benefits. For example, it is almost unlimited in volume, which is key for AI and ML systems that typically involve large amounts of data. Still, some challenges exist, such as the complexities of cloud management.
Top benefits of object storage for AI
For AI/ML applications, finding the right data is important. Metadata containing attributes of the contents of the object helps AI/ML systems find the data needed. An identifier helps find the correct object.
The contents of the object, metadata and the unique identifier are the components of an object.
Traditional file and block storage formats are typically for more transactional applications. Object storage handles large data quantities that don't necessarily need quick and repeated access. This is important for AI/ML because the application needs time to learn its tasks. Subsequently, it needs to examine huge amounts of data before providing an answer.
The capacity of object storage is one of the top advantages. Object contents can contain just about any kind of data file, such as text or video. For situations where the object contains a lot of unstructured data, users might need a process called munging -- or data wrangling -- to reconstruct the data into a format that the AI/ML app can use.
Object storage is easily scalable as needs change, especially in AI/ML situations where users manage large and changing amounts of data. Organizations can also store objects almost anywhere, which is why they are an important storage option in cloud environments. Users can easily create and store multiple copies in different locations, whether on-site or in a cloud.
Potential challenges
Storing vast quantities of data as objects requires a lot of capacity, which is why clouds are an important object storage medium. Enterprises must consider cloud costs when using object storage for AI, however. Given the size of objects, and if the organization stores many in one medium, location could take time. This process might have an impact on network bandwidth.
Consider how the organization will use object storage for AI. In AI/ML applications, the major consideration is the data, not the AI/ML model.
Multiple users who work on the same object might create different versions of it. Any time an object is changed, a new object is automatically created. It then becomes a challenge to decide which version of the object is the one to use.
When organizations store objects in clouds, they must consider issues surrounding cloud management, including costs, security and service levels. By contrast, on-site storage might not present those issues, but the amount of storage needed -- even with data compression of an object -- might necessitate more costly storage.
Organizations must also consider security for objects, such as who receives access.
Additional considerations
Consider how the organization will use object storage for AI. In AI/ML applications, the major consideration is the data, not the AI/ML model.
Factor business considerations for object storage versus block and file formats into such decisions. It is incumbent on administrators to fully understand the business requirements for object storage.
Paul Kirvan is an independent consultant, IT auditor, and technical writer, editor and educator. He has more than 25 years' experience in business continuity, disaster recovery, security, enterprise risk management, telecom and IT auditing.
