data lifecycle

What is a data lifecycle?

A data lifecycle is the sequence of stages that a particular unit of data goes through from its initial generation or capture to its eventual archival and/or deletion at the end of its useful life.

Although specifics vary, data management experts often identify six or more stages in the data lifecycle, as noted in the following example and flowchart:

1. Determine the need. Before generating new data or capturing existing data, ensure that the need for such data is established and confirmed.
2. Generation or capture. In this phase, data comes into an organization, usually through data entry, acquisition from an external source or signal reception, such as transmitted sensor data.
3. Data preparation for use. In this phase, data is processed to prepare it to be used. The data might be subjected to processes such as integration, scrubbing and extract, transform, and load (ETL). It might also be encrypted to ensure its security and privacy.
4. Active use. In this phase, data is used to support the organization's objectives and operations.
5. Management. During the management phase, data might be made available to the broader public or retained inside the organization. It can be stored in a short-term storage platform for continued availability while its security and privacy is constantly maintained.
6. Long-term storage, retention or archiving. In this phase, data is removed from all active production environments and moved to an archive. It is no longer processed, used or published, but it is stored in case it is needed again in the future. The data might be stored on hard disk drives (HDDs), solid-state drives (SSDs) or tape in-house or with a cloud storage service.
7. Purging and destruction. When the data becomes obsolete, every copy of data is deleted and destroyed as part of a removal process. The data might have been archived or it is data that is no longer needed or has been superseded by newer data. The data destruction process might also include the media on which the data resides.

In this data lifecycle flowchart, a step is added during which the need for the data is determined and validated. Considering how much data is generated every hour, this step is important to minimize the collection of too much data that may or may not be used in the organization (and may overload storage devices). Considering that archived data might be required again for legal and/or audit review or other purposes, a step has been added to identify the retrieval and reuse of archived data.

What is the importance of data lifecycle management?

Data lifecycle management (DLM) is becoming increasingly important because big data analytics have become mainstream and the ongoing development of the internet of things (IoT). It is also a key element in data governance.

Enormous volumes of data are being generated by an ever-increasing number of devices, so proper oversight of data throughout its lifecycle is essential to optimize its usefulness and minimize the potential for errors. Finally, archiving or deleting data at the end of its useful life ensures that it does not consume more resources than necessary.

DLM also ensures that data handling addresses the following security and privacy goals:

• Confidentiality of the data. Data, especially sensitive data, is stored in secure environments so that unauthorized individuals cannot access it.
• Integrity of the data. The data is managed and stored in a way that the data content is unchanged and protected from corruption, regardless of who uses the data or how many versions are generated.
• Availability of the data. The data is available to only those individuals or entities who have a need to know and who have the appropriate level of security access.

DLM is also essential for establishing and maintaining compliance with key data security and privacy legislation, such as the General Data Protection Regulation (GDPR), Sarbanes-Oxley Act (SOX) and Health Insurance Portability and Accountability Act (HIPAA).

Achieving compliance with lifecycle management requirements

Considering the above regulations regarding data privacy, and the many others regarding data security (e.g., ISO 27001) at some point in time, especially during an IT audit, it will be necessary to demonstrate compliance with data lifecycle controls. Activities to perform in advance of audits and reviews include the following:

• Ensure that policies regarding data management and lifecycle management are in place and up to date.
• Examine data management procedures to ensure they are up to date and accurately reflect the current state of data management in the organization.
• Generate copies of management reports that provide evidence of how data is managed, stored, archived and destroyed.
• Prepare evidence of how/where data is stored and archived, as well as how it is destroyed, and how the security of the data is established and maintained; this can be organized with internal data management applications or with third-party solutions.

See how data governance and data management work together and what key roles a data management team should include.