Introduction
Nowadays, we are in a world where data-driven decision-making is in trend. Machine learning has emerged as a powerful tool for prediction and extracting information from data sets . It is well known that every machine learning model needs to be trained on a data set and for training that data set must be errorless, properly formatted and complete, people generally unnoticed this process which is known as Data Cleaning.
dataset.
![Data cleaning in Machine Learning]()
What is Data Cleaning?
In simple words, Data Cleaning is the process of identifying and removing any missing, duplicate, or irrelevant data from the dataset. Data Cleaning is one of the critical steps in machine learning because it’s difficult to correct or delete inaccurate, damaged, improperly formatted, duplicated, or insufficient data from a dataset.
The main aim of this process is that data must be accurate, errorless and consistent because it can have a bad impact on the machine learning model.
What are the data issues?
In general, a dataset can have many issues, but commonly, we find some, which are as follows.
- Missing Values: The blank space in the column and incomplete dataset because of any reason are treated as missing values. It can skew statistical measures and lead to biased results.
- Outliers: A data point that deviates significantly from the other dataset because of natural variation or any type of measurement error. It can distort statistical errors and inaccurate predictions.
- Inconsistencies: It is variations in format, unit or codes that can arise due to human error or merging data from different dataset. They lead to misinterpretations and errors in analysis.
- Duplicate Values: The reputation of those values that need to be identical is treated here as duplicate values.
Noise
The random variations or errors in the data that are unrelated to the underlying phenomenon being studied are known as Noise. Noise can arise from measurement errors, sampling variability, or irrelevant factors included in the dataset.