The Big Data Conundrum! Automating Data Cleansing

Redundant, Obsolete, and Trivial (ROT) is a Big Data problem facing the 21st century. The popularity of Big Data is changing the way we think. It can offer unprecedented insight into business data and improve decision-making. Experts are still not on the same page on what makes up “big” data. Some argue that it is the complexity of the dataset, while others believe the source of the data is its driving force. One thing we can all agree on is that data isn’t “big” because of its size.

Even though Big Data is voluminous and becoming more difficult for traditional data processing systems to manage. Size is only one of the many features it possesses!

Our Planets Big Data Mad Rush!

“90% of the data we collect is useless 90% of the time,” said Tom Griffin of Sevone.

We are drowning in Big Data! Too much time is being wasted trying to define our data sets, and not much is being done about harnessing the value of the data collected. As mountains of unused data are stored by organizations hoping to make data-driven decisions, it’s quickly becoming a hot mess of useless data!

When data processing systems are not in place to classify and understand Big Data, organizations are simply practicing a ‘data hoarding culture’. Not only is this a huge financial burden to many companies, but it also has regulatory and security ramifications.

Pause! Ask these questions first before collecting more data:

Why do you need to collect the data?
What are your plans for redundant data?
What are the consequences of binning data that might become valuable in the future? How long do you plan to store this data in the hopes that it will one day become useful?
How often will you need to collect this data?
Where will the data be stored?
Are there any regulatory requirements?
What depth does the data need to be?

These and many more questions must be answered before embarking on any data collection journey.

Automated Data Preparation and Cleaning Workflows

Even though using a data warehouse can help ease Big data storage problems. Over time, as the data stored increases, these stacks of unused and unstructured data become a burden to any organization. Batch processing data can be an efficient method in resolving this big data problem. It will ensure that useless data is discarded at the collection stage. Data can also be classified to make subsequent cleaning and analysis more thorough.

Automating data preparation and cleansing can be done in a variety of ways. However, these processes can only be automated with in-depth knowledge of the data structure. Implementing an iterative cleansing system with basic Python commands can help achieve this automaton only if the data scientist can create a data collection model tailored to the organizations’ data analysis strategy.

Building a machine learning model that learns from previous cleansing tasks will enable organizations to filter through complex data before storage. Although there are some intricacies in batch processing, big data systems combined with artificial intelligence can be used for deep learning and preprocessing unstructured or semi structured data.

Here are some simple Python methods that can help in building a cleaning script:

Dropping columns that are not useful for the analysis process.

# Drop x column from data
data.dropna(subset=['x'], how='any',inplace=True)

Finding duplicated data and assign to a variable

# Find duplicated entries in x dataset column A
dupe = x[x.duplicated('A')]

Changing data types. For example, strings to integer, categorical types etc.

# Convert non numeric to float values.

# Convert any type of column to an explicit dtype. Very useful for converting categorical types.

# Convert object columns of a DataFrame to a more specific type (soft conversions).

# Convert Series and DataFrame columns to the best possible data type.

Strip an entire Series in Pandas. This will remove leading/trailing white spaces on the referenced column

# The best way to remove blank whitespaces in pandas DataFrames.

There are so many more methods and functions available to build automated batch processing systems that can resolve your big data problem. Remember you can create custom loops to iterate through vast amounts of data and reclassify for easy stream processing. You can highlight data, remove break lines, concatenate strings, change date-time formats and so many more data management processes.


There is unanimous agreement that big data is changing businesses and industries. That is why data analysts, managers and decision-makers have to automate the process to simplify data analysis. Relying on data warehouses alone is not enough!

Data lakes and warehouses can be expensive and not just in monetary terms. It’s a waste of valuable insights when datasets become redundant because there are no big data systems in place to carry out data analysis.

With machine learning and artificial intelligence models, data can be analyzed in real-time and allocated to the right data warehouse. Using the most effective big data technology, the process of classifying and storage can be automated, allowing more time for data analysis.

Leave a Reply