Managing data growth — How much data do you really need?

Data is the heart of modern business. It’s used to drive decision-making, develop products and services, evaluate performance and to deliver customers an experience that keeps them loyal.

A recent survey found that enterprise organizations experienced a 63% growth in data volume each month, storing around 2.02PB across local, managed and cloud-hosted storage locations. While total volumes of data handled by small- and medium-sized businesses are smaller, the IDC estimates a 50% rate of growth per year. The more data there is, the more important it becomes to understand and manage data risk. Managing data growth is an increasing problem for businesses of all sizes.

Data is essential for a successful business, but it’s also a liability that needs to be managed. Poor data management practices can leave data exposed to cyber-criminals. Further, businesses that violate privacy laws or suffer a data breach may find themselves under the scrutiny of regulators and privacy rights groups.

How the data piled up

Historically, organizations have collected and retained data because that’s how things were done. They were operating in an environment where the cost of data storage was decreasing all the time, and the ability to analyze data was in its infancy. Collecting more of it meant a greater chance of having what you need when you need it. 

As the cost of data storage decreased, there was no incentive to clean up and manage that data, so businesses created vast data lakes. As these grew, they had less and less visibility of this data and increasingly limited awareness of where it was, how it was being used and how far it had been distributed.  

It’s only with the introduction of more recent privacy regulation such as the European GDPR that introduced data minimization principles — to retain only the data necessary to fulfil its defined purpose – that legacy practices have begun to change.

That historic data remains, however, and current CISOs and IT managers find themselves custodians of networks hosting vast quantities of sensitive and personal information they now need to manage.

New tech, new problems

While legacy data is one issue, new technologies and changing business practices present their own challenges. 

The rapid increase in remote- and home-working arrangements has accelerated digital transformation programs for organizations of all sizes. The ease by which businesses can create new storage environments and adopt new platforms and services in the cloud makes it hard to control data proliferation and maintain visibility of data across all technology environments. 

And the introduction of AI and machine-learning systems further fuels data growth. These systems often produce significant volumes of data themselves, but also require vast stores of data to train and refine their algorithms — through duplicating production data for testing or ingesting new data for the purpose.

Bringing data down to size

Data growth is inevitable as modern business increasingly relies on the insight it offers. However, there’s a balance to strike. Of all the data that businesses collect and store, almost 70% is never used. 

According to one survey, this is in part due to privacy concerns which prohibit the use of personal data beyond the specific purpose it was collected, or that devalue the quality of the data by means of anonymization and more. 

So, what can businesses do to manage their data more effectively and reduce the rate of growth while retaining value? 

The first step is understanding the data they have — what types of data, in what formats and where it resides. 

Following the discovery process, businesses will identify:

• Data that can be purged because it no longer serves any purpose, is out of date or has been superseded 
• Data that has value but isn’t currently being utilized
• Sensitive and high-risk data that needs protecting — such as PII and payment card data, as well as sensitive business information.

Armed with this information, businesses can more critically evaluate the data they create and ingest, streamlining their analytics and operations to make better use of the data they have while lowering the rate of data growth. Efforts which, in the long run, will lead to more sustainable data management practices that can simultaneously support data security, privacy compliance and business development. 

If you’re ready to start managing your data more effectively, book a call to meet with one of our experts today.