Great new way to handle Missing Values

May 22, 2023 15 minute read

Missing values in datasets are a common challenge in data analysis and can significantly impact the quality of research or predictive models. They arise due to various reasons, such as errors in data collection, non-response in surveys, or disruptions in data transmission. The handling of these missing values is crucial, as improper treatment can lead to biased results and incorrect conclusions.

There are several strategies for dealing with missing data, each with its advantages and limitations. They can be listed as follows: Deletion, Imputation, Synthetization. The approach we take to handle the missing data, very much depends on the nature of the missing data. There are three main types of missing data, each with different implications:

Missing Completely at Random (MCAR):

• When the probability of a value being missing is the same for all cases, the data is said to be missing completely at random.
• In MCAR, the missingness of data is unrelated to any of the study variables or to the missing data itself.
• If data are MCAR, most handling methods, including simple ones like deletion or mean imputation, are unlikely to introduce bias.

Missing at Random (MAR):

• Data is missing at random if the missingness is related to some observed data and not to the value of missing data itself.
• For instance, if younger participants are more likely to skip a survey question, the missingness is related to the ‘age’ variable but not necessarily to the missing question’s answers.
• Handling MAR often requires more sophisticated methods, such as multiple imputation, which can model the missingness based on observed data.

Missing Not at Random (MNAR):

• Data is missing not at random if the missingness is related to the value of the missing data itself.
• For example, individuals with lower income might be more likely to skip income-related questions.
• MNAR is the most challenging type of missing data to handle because it requires understanding the mechanism behind the missingness and often needs complex statistical methods or domain expertise for appropriate handling.

In cases where the missing data points are MCAR and missing values don’t make a large part of the dataset, we could simply delete the observations (rows) which have missing values without loosing relevant information from the dataset.

If the missing values are MAR, and/or the missing values make a large part of the dataset, Imputation is a good way to proceed. There are various ways of performing Imputation:

• Mean/Median/Mode Imputation: Replace missing values with the mean, median, or mode of the column.
• K-Nearest Neighbors (KNN) Imputation: Replace a missing value using the KNN algorithm, which finds the nearest neighbors according to a given distance measure.
• Regression Imputation: Use regression models to estimate and replace missing values

Finally, in case the data is MNAR and especially if the missing values are plentiful, there is no clear way of action. In such cases missing datapoint can be either imputed, synthesized or otherwise generated through a number of other techniques which are often tailored depending on the end goal of the data analysis.

While both imputation and synthesizing data address the issue of incomplete data, imputation focuses on filling in gaps in an existing dataset, adhering closely to its original structure. In contrast, data synthesis involves creating new data points, often for broader purposes like data augmentation, privacy preservation, or when actual data is unavailable. Synthesizing data includes methods like data simulation, where a model is built to generate data points that mimic the distributions of real data. Techniques like bootstrapping or generative models (e.g. Generative Adversarial Networks) are often used.

The approach I wish to introduce is referred to as the “Hard-Impute” and it was first published by Mazumder, Hastie, and Tibshirani. It is an Imputation technique, best suited for MAR and MCAR scenarios.

The Hard-Impute

Imputation methods aim at addressing missing values by identifying the patterns and relationships within the available, non-missing data. These identified structures are then utilized to estimate and replace the missing values effectively. An example of a very popular imputation method is the K-Nearest Neighbors Imputation, however it suffers from the same weakness as the KNN clasifier, which is its high sensitivity to irrelevant features.

The Hard-Impute utilizes the correlation structure behind the features of the dataset in order to proxy for the missing values. It does so by iteratively calculating the principal components of the dataset, and using the PCA scores and vector loadings to reconstruct the original values. This might sound like a handful, so let’s dive deeper into that following an example.

Below is a well-known US Arrests dataset from which we have removed 20 values by random. The dataset showcases the number of arrests per 100,000 residents for crimes such as assault, murder, and gender crimes in each of the 50 US states in 1973. The dataset is available here

US Arrests Data

The 1st step of the Hard-Impute Algorithm is to standardize the dataset with a mean of 0 and standard deviation of 1, and to replace the mising values with their column/feature means.

Standardized US Arrests Data

Standardization is a necessary step because PCA is significantly influenced by the different units used to measure the features. Standardization ensures that the principal components reflect the underlying structure of the data in a balanced way, rather than being skewed by arbitrary differences in variable scales and units.

Missing values are replaced with column means (which are 0 by design of standardization)

In the 2nd step, the PCA is ran against the standardized dataset. I will not go into detail about PCA in this discussion, but I would just remind the reader that the 1st Principal Component of a set of features is a linear combination of those features that results in the maximum variance , where x is our standardized data, p is the number of features, i is the number of observations/rows , z is the first principal component score for each observation i, and fi is the vector loading for each feature p.

PCA algorith outputs i principal component scores and p factor loadings. Due to PCA’s inherent ability to retain maximum information from the dataset, we can represent each data point using the formula:

Naturally, our focus is on approximating the missing values (temporarily replaced with the means of their respective features), while leaving the other data points unchanged.

In the 3rd step we would apply the 2nd step iteratively and keep producing the proxies for missing values until the change between the sum of all missing proxy values between 2 iterations falls below a given threshold (in this example it is 0.002)

< 0.002

Our end result is:

We could also reverse the standardization process on the dataset mentioned above (by multiplying each value in the dataset by the standard deviation of its respective feature from the original dataset, and then adding the mean of that feature from the original dataset).

Destandardized Dataset

Evaluations & Limitations

In a practical setting, we can’t assess the effectiveness of the Hard-Impute method by comparing the approximated values to the original ones, as the original values are missing. However, in the case of the US Arrests dataset, we have intentionally omitted 20 original values, enabling us to evaluate performance. The correlation coefficient between the approximated and original values is impressively high at 0.98, demonstrating the method’s efficiency. The scatter plot below visually supports our findings:

Proxied vs. Original values

The Hard-Impute method has one significant limitation: it’s only suitable for numerical features. This means that if the dataset includes categorical or other non-numerical features, these need to be addressed separately. Additionally, it’s important to remember that the method’s efficiency is heavily influenced by the nature of the relationships between features. When features tend to vary in unison, their informational content can be more effectively encapsulated within the principal components, leading to more accurate reconstruction of missing values. For instance, in the US Arrests dataset, the logical co-occurrence of various types of crimes and the proportion of urban population contributes to the high efficiency of the Hard-Impute method. The line graph below illustrates how feature values generally trend together across most US states:

Co-movement of feature value s

Lastly, it’s crucial to address the significance of choosing the optimal number of principal components, denoted as M. With the US Arrests dataset, using M=1 yielded excellent results. However, the Hard-Impute method can accommodate any number of principal components. A strategy for determining the optimal M involves deliberately omitting extra values from the dataset, then applying the Hard-Impute method for all potential M values, and selecting the M that results in the highest average correlation between the approximated and original values.

Wrap-Up

Dear reader, we have now reached the end of this article. I hope you have found it insightful, and I trust you will agree that Hard-Impute is a valuable data preprocessing technique to include in your data science toolkit. For those interested in the Python code behind the algorithm, I encourage you to visit my git project. Additionally, I have published the method as a PYPI Python package, allowing for immediate installation and use of the algorithm.

Cheers!