Project 10— Day 24 of 30 days of Data Analytics with Projects Series
Welcome back peep. Hope all’s well. This is Day 24 of 30 days of data analytics where we will be implementing a project covering —
Standardization
Encoding
Linear Regression
Data Profiling
Categorical and Numerical Features
Missing Value Analysis
Unique Value Analysis
Univariate Analysis
Bivariate Analysis
Multivariate Analysis
Correlation Analysis
Correlation Coefficients
Lets cover the most important concepts in brief —
- Standardization: is a technique used to transform variables so that they have a mean of 0 and a standard deviation of 1. This is often done to put variables on the same scale and make them comparable. Standardization is typically used for variables that are measured on different scales, such as weight in pounds and height in inches.
- Encoding: is a technique used to convert categorical variables into numerical variables, so that they can be used in a model. There are several encoding techniques such as one-hot encoding, label encoding, ordinal encoding and others.
- Linear Regression: is a statistical technique used to model the relationship between one or more independent variables (also called predictors or features) and a dependent variable. Linear regression assumes that the relationship between the variables is linear and can be used to make predictions about the value of the dependent variable based on the values of the independent variables.
- Data Profiling: is the process of analyzing and summarizing the main characteristics of a dataset. This can include reviewing the data types, number of records, missing values, and other statistical summaries. It is important step in understanding the data and identifying potential issues before building models.
- Categorical and Numerical Features: Categorical features are those that represent a group of items or categories, such as ‘color’ or ‘gender’. Numerical features are those that represent a quantity, such as ‘height’ or ‘weight’.
- Missing Value Analysis: it is the process of identifying missing values in the data. Missing values can occur in datasets due to various reasons such as data entry errors or data not being collected. Identifying missing values is important before building models because they can affect the model performance.
- Unique Value Analysis: it is the process of identifying the unique values in the data. This can be useful in understanding the distribution of the data and identifying potential outliers.
- Univariate Analysis: is the process of analyzing one variable at a time. This can include reviewing the distribution of the variable and identifying potential outliers.
- Bivariate Analysis: is the process of analyzing the relationship between two variables. This can include reviewing the correlation between the variables and identifying potential outliers.
- Multivariate Analysis: is the process of analyzing the relationship between three or more variables. This can include reviewing the correlation between the variables and identifying potential outliers.
- Correlation Analysis: is the process of analyzing the relationship between two or more variables. Correlation coefficients can be used to determine whether two variables are positively or negatively correlated and the strength of this correlation.
- Correlation Coefficients: are measures of the strength and direction of the relationship between two variables. The different types are:
- Spearman’s ρ: a non-parametric measure of the correlation between two variables. This measure is used when the data is ordinal.
- Pearson’s r: a measure of the linear correlation between two variables. This measure is used when the data is interval or ratio.
- Kendall’s τ: a non-parametric measure of the correlation between two variables. This measure is used when the data is ordinal.
- Cramér’s V (φc): a measure of association between two categorical variables. It is used when the variables are nominal and the sample size is small.
- Phik (φk): a measure of association between two categorical variables. It is used when the variables are nominal and the sample size is large.
Example Code Implementation —
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.linear_model import LinearRegression
from scipy import stats
# Standardization
# Assume we have a numerical variable 'X' that needs to be standardized
scaler = StandardScaler()
X_standardized = scaler.fit_transform(X)
# Encoding
# Assume we have a categorical variable 'category' that needs to be encoded using one-hot encoding
encoder = LabelEncoder()
category_encoded = encoder.fit_transform(category)
# Linear Regression
# Assume we have two independent variables 'X1' and 'X2' and a dependent variable 'y'
X = data[['X1', 'X2']]
y = data['y']
# Create a linear regression model
model = LinearRegression()
# Fit the model to the data
model.fit(X, y)
# Predict the dependent variable
y_pred = model.predict(X)
# Data Profiling
data = pd.read_csv('your_dataset.csv')
print("Data Types:")
print(data.dtypes)
print("Number of Records:", len(data))
print("Missing Values:")
print(data.isnull().sum())
print("Statistical Summaries:")
print(data.describe())
# Categorical and Numerical Features
categorical_features = data.select_dtypes(include=['object']).columns.tolist()
numerical_features = data.select_dtypes(include=['int', 'float']).columns.tolist()
print("Categorical Features:", categorical_features)
print("Numerical Features:", numerical_features)
# Missing Value Analysis
missing_values = data.isnull().sum()
print("Missing Value Analysis:")
print(missing_values)
# Fill the missing Values
# Fill missing values using mean
data_filled = data.fillna(data.mean())
# Unique Value Analysis
unique_values = data.nunique()
print("Unique Value Analysis:")
print(unique_values)
# Univariate Analysis
variable = data['Variable']
print("Univariate Analysis:")
print("Distribution:")
print(data['Variable'].describe())
print("Outliers:")
outliers = data[(np.abs(stats.zscore(data['Variable'])) > 3)]
print(outliers)
# Bivariate Analysis
variable1 = data['Variable1']
variable2 = data['Variable2']
print("Bivariate Analysis:")
print("Correlation:")
correlation = variable1.corr(variable2)
print(correlation)
print("Outliers:")
outliers = data[(np.abs(stats.zscore(data[['Variable1', 'Variable2']])) > 3).any(axis=1)]
print(outliers)
# Multivariate Analysis
variable1 = data['Variable1']
variable2 = data['Variable2']
variable3 = data['Variable3']
print("Multivariate Analysis:")
print("Correlation:")
correlation_matrix = data[['Variable1', 'Variable2', 'Variable3']].corr()
print(correlation_matrix)
print("Outliers:")
outliers = data[(np.abs(stats.zscore(data[['Variable1', 'Variable2', 'Variable3']])) > 3).any(axis=1)]
print(outliers)
# Correlation Analysis
correlation_matrix = data.corr()
print("Correlation Analysis:")
print(correlation_matrix)
# Correlation Coefficients
spearman_corr, _ = stats.spearmanr(data['Variable1'], data['Variable2'])
pearson_corr, _ = stats.pearsonr(data['Variable1'], data['Variable2'])
kendall_corr, _ = stats.kendalltau(data['Variable1'], data['Variable2'])
cramer_corr, _ = stats.pointbiserialr(data['CategoricalVariable1'], data['CategoricalVariable2'])
phik_corr = data.corr(method='phik')
What’s covered in 30 days of Data Analytics Series till now —
Day 1 : Data Analytics basics and kickstart of Data analytics with projects series
Day 3 : Data Analytics Ecosystem — Data Life Cycle, Data Analysis complete process ( most important things)
Day 5 : Statistics
Day 6 : Basic and Advanced SQL
Day 8 : Pandas and Numpy
Day 9 : Data Manipulation
Day 10 : Data Visualization — Part 1
Day 11 : Project 1 : Data Visualization — Part 2
Day 12 : Data Visualization — Part 3
Day 13: Tableau — Part 1
Day 14: Tableau — Part 2
Day 15: Tableau — Part 3
Day 16 : Data Analysis Project 2
Day 17 : Data Analysis Project 3
Day 18: Data Analysis Project 4
Day 20 : Data Analysis Project 6
Day 21 : Data Analysis Project 7
Take Complete Hands On Tableau Course : Link
Projects Videos —
All the projects, data structures, SQL, algorithms, system design, Data Science and ML , Data Analytics, Data Engineering, , Implemented Data Science and ML projects, Implemented Data Engineering Projects, Implemented Deep Learning Projects, Implemented Machine Learning Ops Projects, Implemented Time Series Analysis and Forecasting Projects, Implemented Applied Machine Learning Projects, Implemented Tensorflow and Keras Projects, Implemented PyTorch Projects, Implemented Scikit Learn Projects, Implemented Big Data Projects, Implemented Cloud Machine Learning Projects, Implemented Neural Networks Projects, Implemented OpenCV Projects,Complete ML Research Papers Summarized, Implemented Data Analytics projects, Implemented Data Visualization Projects, Implemented Data Mining Projects, Implemented Natural Leaning Processing Projects, MLOps and Deep Learning, Applied Machine Learning with Projects Series, PyTorch with Projects Series, Tensorflow and Keras with Projects Series, Scikit Learn Series with Projects, Time Series Analysis and Forecasting with Projects Series, ML System Design Case Studies Series videos will be published on our youtube channel ( just launched).
Subscribe today!
Tech Newsletter —
If you are interested, you can join my newsletter through which I send tech interview tips, techniques, patterns, hacks — Software Development, ML, Data Science, Startups and Technology projects to more than 30K readers. You can subscribe to Tech Brew :
In the last post we covered Data Visualization and in this post we will cover a project.
Pre-requisite —
Before starting, go through this post to understand charts/plots and which chart to use and when.
(Note : Zoom all the images)
Import Necessary Libraries
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import seaborn as sns
import pandas_profiling
from matplotlib import pyplot as plt
from matplotlib.colors import rgb2hex
import matplotlib.cm as cm
import matplotlib.colors
from collections import Counter
cmap2 = cm.get_cmap('twilight',13)
colors1= []
for i in range(cmap2.N):
rgb= cmap2(i)[:4]
colors1.append(rgb2hex(rgb))
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
# Set style
sns.set(style='whitegrid')
Load data and get information
#Load the data
df= pd.read_csv('/Path to file/vgsales.csv', low_memory = False)#Get information about your data
df.info()Output —
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 16598 entries, 0 to 16597
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Rank 16598 non-null int64
1 Name 16598 non-null object
2 Platform 16598 non-null object
3 Year 16327 non-null float64
4 Genre 16598 non-null object
5 Publisher 16540 non-null object
6 NA_Sales 16598 non-null float64
7 EU_Sales 16598 non-null float64
8 JP_Sales 16598 non-null float64
9 Other_Sales 16598 non-null float64
10 Global_Sales 16598 non-null float64
dtypes: float64(6), int64(1), object(4)
memory usage: 1.4+ MB# Get Columns informationdf.columnsOutput —
Index(['Rank', 'Name', 'Platform', 'Year', 'Genre', 'Publisher', 'NA_Sales',
'EU_Sales', 'JP_Sales', 'Other_Sales', 'Global_Sales'],
dtype='object')Data Description
- Rank — Ranking of overall sales
- Name — The games name
- Platform — Platform of the games release (i.e. PC,PS4, etc.)
- Year — Year of the game’s release
- Genre — Genre of the game
- Publisher — Publisher of the game
- NA_Sales — Sales in North America (in millions)
- EU_Sales — Sales in Europe (in millions)
- JP_Sales — Sales in Japan (in millions)
- Other_Sales — Sales in the rest of the world (in millions)
- Global_Sales — Total worldwide sales.
Statistical Summary of the data
df.describe()Categorical and Numerical Features
Categorical features are those values that be sorted into groups or categories.
Numerical Features are those values that can be measures (can be places in ascending or descending order)
For this, lets get the Categorical and Numerical Features —
df.info()Output —
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 16598 entries, 0 to 16597
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Rank 16598 non-null int64
1 Name 16598 non-null object
2 Platform 16598 non-null object
3 Year 16327 non-null float64
4 Genre 16598 non-null object
5 Publisher 16540 non-null object
6 NA_Sales 16598 non-null float64
7 EU_Sales 16598 non-null float64
8 JP_Sales 16598 non-null float64
9 Other_Sales 16598 non-null float64
10 Global_Sales 16598 non-null float64
dtypes: float64(6), int64(1), object(4)
memory usage: 1.4+ MBYou can see , in our dataset —
Categorical Features are Name, Platform, Genre, Publisher
Numerical Variable are Rank, Year, NA_Sales, EU_Sales, JP_Sales, Other_Sales, Global_Sales
Missing Value Analysis
In this we figure out the missing values in the
df.isnull().sum()Output —
Rank 0
Name 0
Platform 0
Year 271
Genre 0
Publisher 58
NA_Sales 0
EU_Sales 0
JP_Sales 0
Other_Sales 0
Global_Sales 0
dtype: int64One can also calculate the percentage of missing values out of the total.
p = (df.isnull().sum()/df.isnull().count()).sort_values(ascending=False)
t = df.isnull().sum().sort_values(ascending=False)m_data = pd.concat([t, p], axis=1, keys=['Total', 'Percent'])
m_data.head(10)Output —
Unique Value Analysis
One can get the count of the unique values for each column in your data —
for i in list(df.columns):
print("{} -> {}".format(i, df[i].value_counts().shape[0]))Output —
Rank -> 16598
Name -> 11493
Platform -> 31
Year -> 39
Genre -> 12
Publisher -> 578
NA_Sales -> 409
EU_Sales -> 305
JP_Sales -> 244
Other_Sales -> 157
Global_Sales -> 623Univariate Analysis
In Univariate Analysis, single variable/feature is analyzed at a time.
First we will start with Categorical Features in our data and then Numerical Features.
Categorical Features are Name, Platform, Genre, Publisher
Numerical Variable are Rank, Year, NA_Sales, EU_Sales, JP_Sales, Other_Sales, Global_Sales
Categorical Features Univariate Analysis
# Platform Count
plt.figure(figsize=(10,8))
sns.countplot(x='Platform',data=df,palette='mako',order = df['Platform'].value_counts().index)
plt.xlabel('Platform')
plt.xticks(rotation = 60)
plt.ylabel('Count')
plt.legend()
plt.title('Platform Count')
plt.show()Output —
# Genre Count
plt.figure(figsize=(10,8))
sns.countplot(x='Genre',data=df,palette='mako',order = df['Genre'].value_counts().index)
plt.xlabel('Genre')
plt.xticks(rotation = 60)
plt.ylabel('Count')
plt.legend()
plt.title('Genre Count')
plt.show()
Output —
# Name Percentage
plt.figure(figsize=(25,12))
p_r = df['Name'].value_counts().head(10)
plt.pie(x=p_r,labels=p_r.index,colors=colors1,autopct='%.0f%%',explode=[0.07 for i in p_r.index],startangle=180,wedgeprops={'linewidth':1,'edgecolor':'black'},shadow=True)
plt.title('Name percentage ')
plt.legend(loc='upper right',title='Name')
plt.show()
Output —
# Platform Percentage
plt.figure(figsize=(25,12))
p_r = df['Platform'].value_counts().head(10)
plt.pie(x=p_r,labels=p_r.index,colors=colors1,autopct='%.0f%%',explode=[0.07 for i in p_r.index],startangle=180,wedgeprops={'linewidth':1,'edgecolor':'black'},shadow=True)
plt.title('Platform percentage ')
plt.legend(loc='upper right',title='Platform')
plt.show()Output —
# Genre Percentage
plt.figure(figsize=(25,12))
p_r = df['Genre'].value_counts().head(10)
plt.pie(x=p_r,labels=p_r.index,colors=colors1,autopct='%.0f%%',explode=[0.07 for i in p_r.index],startangle=180,wedgeprops={'linewidth':1,'edgecolor':'black'},shadow=True)
plt.title('Genre percentage ')
plt.legend(loc='upper right',title='Genre')
plt.show()Output —
# Publisher Percentage
plt.figure(figsize=(25,12))
p_r = df['Publisher'].value_counts().head(10)
plt.pie(x=p_r,labels=p_r.index,colors=colors1,autopct='%.0f%%',explode=[0.07 for i in p_r.index],startangle=180,wedgeprops={'linewidth':1,'edgecolor':'black'},shadow=True)
plt.title('Publisher percentage ')
plt.legend(loc='upper right',title='Publisher')
plt.show()Output —
plt.figure(figsize=(25,18))
sns.kdeplot(data=df['Year'], label='Year', shade=True,palette='mako')
plt.xlabel('Year')
plt.xticks(rotation = 60)
plt.legend()
plt.title('Game Release by Year')
plt.show()Output —
Bivariate Analysis
In Bivariate Analysis, two variables/features are analyzed together and the relationship/association between them is studied.
# Game Release by Year
plt.figure(figsize=(25,18))
sns.countplot(x='Year',data=df,palette='mako',order = df.groupby(by=['Year'])['Name'].count().sort_values(ascending=False).index)
plt.xlabel('Year')
plt.xticks(rotation = 60)
plt.ylabel('Count')
plt.legend()
plt.title('Game Release by Year')
plt.show()Output —
# Game Release by Genre
plt.figure(figsize=(25,18))
sns.countplot(x='Year',data=df,palette='mako',order = df['Year'].value_counts().iloc[:7].index,hue='Genre')
plt.xlabel('Year')
plt.xticks(rotation = 60)
plt.ylabel('Count')
plt.legend()
plt.title('Game Release by Genre')
plt.show()Output —
#Global Sales by Year
gs_df = df.groupby(by=['Year'])['Global_Sales'].sum()
gs_y = gs_df.reset_index()
plt.figure(figsize=(25,18))
sns.barplot(x='Year',y='Global_Sales', data=gs_y,palette='mako')
plt.xlabel('Year')
plt.xticks(rotation = 60)
plt.ylabel('Global Sales')
plt.legend()
plt.title('Global Sales by Year')
plt.show()Output —
# Global Sales by Genre
gg_df = df.groupby(by=['Genre'])['Global_Sales'].sum()
gg_y = gg_df.reset_index()
plt.figure(figsize=(25,18))
sns.barplot(y='Genre',x='Global_Sales', data=gg_y,palette='mako',orient='h')
plt.xlabel('Year')
plt.xticks(rotation = 60)
plt.ylabel('Global Sales')
plt.legend()
plt.title('Global Sales by Genre')
plt.show()Output —
#Global Sales by Platform
gp_df = df.groupby(by=['Platform'])['Global_Sales'].sum()
gp_y = gp_df.reset_index()
plt.figure(figsize=(25,18))
sns.barplot(y='Platform',x='Global_Sales', data=gp_y,palette='mako',orient='h')
plt.xlabel('Year')
plt.xticks(rotation = 60)
plt.ylabel('Global Sales')
plt.legend()
plt.title('Global Sales by Platform')
plt.show()Output —
Multivariate Analysis
In Multivariate Analysis, more than two variables/features are analyzed together and the relationship/association between them is studied.
# Global Sales by Year via different Platform
plt.figure(figsize=(25,18))
sns.catplot(x="Year",y="Global_Sales",kind="point",data=df[(df.Year > 2007) & (df.Year < 2018)], hue = "Platform",
palette='mako',ci = None,edgecolor=None,height=10, aspect=10.6/8.23)
plt.title('Sales by Year via different Platform')
plt.show()Output —
# NA Sales by Year via different Platform
plt.figure(figsize=(25,18))
sns.catplot(x="Year",y="NA_Sales",kind="bar",data=df[(df.Year > 2009) & (df.Year < 2014)], hue = "Platform",
palette='mako',ci = None,edgecolor=None,height=10, aspect=10.6/8.23)
plt.title('Sales by Year via different Platform')
plt.show()Output —
# Different Sales by Platform
pg = df.groupby('Platform').mean()[['NA_Sales', 'EU_Sales', 'JP_Sales', 'Other_Sales' ]]
plt.figure(figsize=(20,10))
pg.plot.line(color=colors1)
plt.title(' Different Sales by Platform')
plt.legend(loc='upper right')
plt.show()Output —
# NA_Sales vs Global Sales by Genre
plt.figure(figsize=(20,10))
sns.lmplot(x='NA_Sales', y='Global_Sales', hue='Genre',size=10, fit_reg=False, data=df,palette='mako')
plt.title('NA_Sales vs Global Sales by Genre')
plt.show()Output —
plt.figure(figsize=(50,30))
sns.pairplot(data=df,diag_kind = "kde",palette='mako',markers='o',size=7,hue = 'Genre')
plt.legend()
plt.show()Output —
Correlation Analysis
In order to measure the strength of the linear association/relation between two variable, Correlation Analysis is used.
# heatmap correlation
# Total Sales by Different Platforms
sales_platform = df[['Platform', 'NA_Sales', 'EU_Sales', 'JP_Sales', 'Other_Sales']]
sales_compare = sales_platform.groupby(by=['Platform']).sum()
plt.figure(figsize=(15,10))
sns.heatmap(sales_compare,annot=True,fmt=".2f",cmap='mako')
plt.xticks(fontsize=15)
plt.yticks(fontsize=15)
plt.show()Output —
# Total Sales by Different Genre
g_platform = df[['Genre', 'NA_Sales', 'EU_Sales', 'JP_Sales', 'Other_Sales']]
g_compare = g_platform.groupby(by=['Genre']).sum()
plt.figure(figsize=(15,10))
sns.heatmap(g_compare,annot=True,fmt=".2f",cmap='mako')
plt.xticks(fontsize=15)
plt.yticks(fontsize=15)
plt.show()Output —
Data Profiling
It is used to generate profile reports from the input data.
The statistics include
Descriptive Statistics and Quantile Statistics.
Descriptive stats — Standard deviation, Kurtosis, mean, skewness, variance etc
Quantile Statistics — Min-max, percentiles, median, IQR etc
df.profile_report()Output —
Linear Regression
It’s a technique to estimate the relationship between two quantitative variables. It is used when you want to establish:
- Strength of the relationship — How strong the relationship is between two variables
- The value of the dependent variable at a certain value of the independent variable.
where,
y is the predicted value of the dependent variable for any given value of the independent variable which is X.
B0 is the intercept and B1 is the regression coefficient
x is the independent variable
e is the error of the estimate
It works on the assumption that the relationship between the independent and dependent variable is linear: the line of best fit through the data points is a straight line as shown in the diagram.
reg_X = df.loc[:,"NA_Sales":]
reg_y = pd.DataFrame(df.loc[:,"Global_Sales"])
X_train, X_test, y_train, y_test = train_test_split(pd.DataFrame(reg_X.loc[:,"NA_Sales"]), reg_y,random_state = 0)
lr = LinearRegression().fit(X_train, y_train)
x = np.array(reg_X["NA_Sales"])
plt.figure(figsize=(20,10))
plt.scatter(reg_X.loc[:,"NA_Sales"], reg_y, marker= 'D', s=50, alpha=0.9, cmap='Blue')
plt.plot(reg_X.loc[:,"NA_Sales"], lr.intercept_+ lr.coef_ * x.reshape(-1,1) , 'Purple')
ax = plt.gca()
ax.xaxis.grid(True,alpha=0.4)
ax.yaxis.grid(True,alpha=0.4)
plt.title('Linear Regression')
plt.xlabel('NA_Sales')
plt.ylabel('Global_Sales')
plt.show()Output —
Standardization
Its a scaling technique which involves feature transformation by subtracting the mean and dividing by standard deviation. It rescales the features values as 0 and 1 which is useful in optimization algorithms.
n_list = ['NA_Sales','EU_Sales','JP_Sales','Other_Sales','Global_Sales']
s= StandardScaler()
scaled_array = s.fit_transform(df[n_list[:-1]])
scaled_arrayOutput —
array([[50.48050838, 57.13692978, 11.93805759, 44.60608534],
[35.28443669, 6.7941883 , 21.76729621, 3.82822442],
[19.08427325, 25.19778483, 12.00272364, 17.29711476],
...,
[-0.32408584, -0.29020692, -0.25149161, -0.25486439],
[-0.32408584, -0.27041811, -0.25149161, -0.25486439],
[-0.31184082, -0.29020692, -0.25149161, -0.25486439]])Correlation Coefficients
It’s the measure of the strength of the relationship between two variables.
Spearman’s ρ
The Spearman’s rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson’s r. It’s value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.
To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.
Pearson’s r
The Pearson’s correlation coefficient (r) is a measure of linear correlation between two variables. It’s value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.
To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.
Kendall’s τ
Similarly to Spearman’s rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It’s value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.
To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.
Cramér’s V (φc)
Cramér’s V is an association measure for nominal random variables. The coefficient ranges from 0 to 1, with 0 indicating independence and 1 indicating perfect association. The empirical estimators used for Cramér’s V have been proved to be biased, even for large samples.
Phik (φk)
Phik (φk) is a new and practical correlation coefficient that works consistently between categorical, ordinal and interval variables, captures non-linear dependency and reverts to the Pearson correlation coefficient in case of a bivariate normal input distribution.
Encoding
It is used to transform categorical features into numerical ones. Since ML models only work with numerical values which makes Encoding a very important technique.
cat_list = ['Platform','Genre','Publisher']
df_copy = df.copy()
df_copy = pd.get_dummies(df_copy, columns = cat_list[:-1], drop_first = True)
df_copy.head(20)Output —
That’s it for now. Day 25 coming soon: Power BI.
Let me know if you have questions in the comment section below. Subscribe/ Follow, Like/Clap as it would encourage me to write more in my free time
Stay Tuned!!
Read More —
11 most important System Design Base Concepts
6. Networking, How Browsers work, Content Network Delivery ( CDN)
13. System Design Template — How to solve any System Design Question
System Design Case Studies — In Depth
Complete Data Structures and Algorithm Series
Some of the other best Series —
30 days of Data Structures and Algorithms and System Design Simplified
Data Science and Machine Learning Research ( papers) Simplified **
100 days : Your Data Science and Machine Learning Degree Series with projects
Complete Data Visualization and Pre-processing Series with projects
Exceptional Github Repos — Part 1
Exceptional Github Repos — Part 2
Tech Newsletter —
If you are interested, you can join my newsletter through which I send tech interview tips, techniques, patterns, hacks — Software Development, ML, Data Science, Startups and Technology projects to more than 30K readers. You can subscribe to Tech Brew :
For Python Projects —
For complete 60 days of Data Science and ML : Day 1 — Day 60 : Quick Recap of 60 days of Data Science and ML
Follow for more updates. Stay tuned and keep coding!
For other projects, tune to —
Build Machine Learning Pipelines( With Code)
Recurrent Neural Network with Keras
Clustering Geolocation Data in Python using DBSCAN and K-Means
Facial Expression Recognition using Keras
Hyperparameter Tuning with Keras Tuner
Custom Layers in Keras
