Image Recognition Using PCA

Dimensionality reduction

Feature Selection

A method to select the most relevant input feature variables to reduce the computational cost of model training and improve the prediction performance.

Covariance, measure how two variables change simultaneously in a linear relationship with each other.

If covariance is highly positive, it’s an important feature. The higher the covariance, higher the feature contribution in the output.

Correlation shows the magnitude and the relation between the simultaneous change in two variables.

It depicts strength of the feature and in the range [-1,1]. So, if the correlation value is more towards +1, the features will be more positively correlated. And if the correlation value is more towards -1, the features are negatively correlated.

Feature Extraction

Avoiding the curse of dimensionality

1. Reduction of computational cost
2. Improved performance
3. Prevention of overfitting
4. Better data understanding

2-dimensional data

To reduce the dimension to one, if we choose any single feature from f1 & f2, we’ll lose a lot of variance in the dataset contained in one of the features.

In this case, we have f1’ = m*f1+n*f2 and f2’ = a*f1+b*f2. For a given f1’ and f2’, we will have minimum variance in one of the dimensions (best-fit line) and choosing the one dimension with the larger variance will suffice over choosing both dimensions in earlier case.

The line should fit in the initial feature vector space in such a way that the variance in one of the axes is very low.

Principal Component Analysis

Using PCA, we can project a high-dimensional data on to a low-dimensional space.

The projection of x on the axis u,

PCA finds the best fitting line by maximizing the sum of the squared distances from the projected points to the origin i.e.,

OR intuitively we want to minimize the distance between the input data and the best-fit line. The best fit line is one of the principal components.

n-dimensional data

The feature set,

Axes transformation, new features axes will be,

Each transformed feature is a linear combination of all the input features, known as Single Value Decomposition. The covariance matrix of the input data can be generated using the transformed vectors.

Eigenvalues and Eigenvectors

An eigenvalue is a measurement of variation. The number of transformed features from the above section is equal to the number of eigenvalues and corresponding eigenvectors. The set of eigenvalues,

Then, one should choose k largest eigenvalues from the covariance matrix of transformed features, where k95%) of the dataset.

NOTE: The eigenvalues are simply numbered from 1 to k, it doesn’t correspond to the original set of eigenvalues and eigenvectors.

Dimensionality: n → k, where n>k

Summary of PCA

1. Standardize the range of continuous initial variables.
2. Compute the covariance matrix to identify correlations between input variables.
3. Compute the eigenvalues & eigenvectors of the covariance matrix to identify the principal components.
4. Create a feature vector to decide the chosen principal components.
5. Recast the data along the principal component axes to create a new feature space.

Image Recognition

Image Data Representation

1. Matrix of pixel values, where each pixel is a feature.
2. Matrix size depends on the scale of image.
3. If the image is grayscale, each pixel is a single value, otherwise three values (RGB).

Applying PCA

1. Centre and scale the features of image, standardizing the feature set (zero mean and unit standard deviation)

2. Find the covariance matrix using the input features (pixels of an image).

3. Find the eigenvalues and corresponding eigenvectors.

4. Sort them in descending order, the first principal component explains the most variance in the dataset.

5. To reduce the dimensionality of the dataset, choose a subset of eigenvalues, capturing the maximum variance of the input dataset.

6. Transform the subset to create a feature vector and recast the data in the new feature vector space.

Recognition algorithm

We can reconstruct the image by multiplying the reduced-dimensional representation (X_train_pca) with the corresponding principal components and adding them up (linear combination of feature vectors), resulting in an approximation of the original image.

Benefits of using PCA in Image Recognition

1. Low-dimensional space of image will accelerate the training and testing of ML models.
2. Helps to remove noise and irrelevant information, improving model generalization.
3. PCA can be used as a feature extraction technique, which allows to represent images more efficiently while retaining maximum features.

NOTE: It is crucial to choose the number of principal components one wants to use, and the optimal number depends on the task and the dataset provided.

~Ashutosh