Data Anonymization with Python

How to make your Data compliant in Data Science Projects

Personal data is the core concept of data protection. Data protection law only applies when data relates to individuals. The GDPR for example increases fines to up to 20 million EUR or, in the case of large companies and groups, up to 4% of the global group turnover of the previous year [1]. When working in the field of Big Data, Data Science or related fields it is essential to know about these laws and how anonymization and pseudonymization give the possibility of still using the data for your use cases.

What is Personal Data?

This is any information relating to an identified or identifiable person. An “identifiable” person is one who can be identified directly or indirectly, in particular by means of association with an identifier such as a name, an identification number, location data or other special characteristics. The possibility of identifying a person is sufficient here.

Personal data are, for example [2]:

• Name
• Address
• E-mail address
• Telephone number
• Birthday

Tension between Data Protection and Big Data

In relation to the topic of Big Data, there is a certain tension with regard to data protection. This is because data is often collected and stored without it being clear in advance what analyses will later take place. However, GDPR for example says data storing and analyses need a certain purpose. A specific purpose is therefore particularly difficult to prove here.

Anonymization vs. Pseudonymization

Anonymizing data offers one solution. When data is anonymized, it is no longer personal data. The situation is different with pseudonymized data. With the appropriate additional knowledge, it is possible to determine the reference person.

Data Manipulation with Python

Let’s start with generating some sample data:

#Import libs
import pandas as pd
import numpy as np


#Create some dummy data
df = pd.DataFrame({
    'id' : [1,2,3,4] ,
    'Name' : ['Ana','Paul','Marc','Ali'] 
})

print(df)

So we have a simple table with an ID and a name column:

   id  Name
0   1   Ana
1   2  Paul
2   3  Marc
3   4   Ali

The most easy way to make your data compliant is to just delete the columns which have GDPR relevant data. Often for analyst purposes it’s okay because you maybe don’t need the data. For a customer price report you may not need the name of a customer — so you can just ignore it.

#1 Possibility - Delete column
df_2 = df.drop('Name', 1)
print(df_2)

id
0   1
1   2
2   3
3   4

Another possibility is to use a label encoder and to anonymize your data:

#2 Possibility - Encode column (Anonymisation)
#Import LabelEncoder from sklearn / DataFrameMapper from sklearn_pandas
from sklearn.preprocessing import LabelEncoder
#!pip install sklearn
from sklearn_pandas import DataFrameMapper

encoders = [(["Name"], LabelEncoder())]
mapper = DataFrameMapper(encoders, df_out=True)
label_col = mapper.fit_transform(df.copy())
df_3 = pd.concat([df.drop(columns=["Name"]), label_col], axis="columns")
print (df_3)

Other possibilities would be to use mechanisms for example in your database like crambling — An approach would be to mix letters. Scrambling techniques are encryption and hashing or to mask your data— Some of the information is hidden using random characters or **** for example. Scrambling and masking and how ever you implement it it could be seen as anonymization or pseudonymization. Here, it depends on who can whether and who can cancel the pseudomization — here, a good data access roles and rules model is the key [2][3].

Summary

Data protection is becoming an increasingly important issue. Companies want to avoid the risk of horrendous fines, however we all personally should handle personal data with confidence, because everyone wants their personal information to be handled responsibly. For this, I have shown you the general basics in this article and how you can do easily anonymize or pseudonymize with python. With regard to the painfully high fines for violations of the GDPR, companies are encouraged to identify risks in advance and to develop the product in accordance with the regulations.

Sources and Further Readings

[1] GDPR.EU, What is GDPR, the EU’s new data protection law? (2021)

[2] Johner Institut, Anonymisierung und Pseudonymisierung (2020)

[3] Complior, PSEUDONYMIZATION AND ANONYMIZATION OF PERSONAL DATA (2020)