Pseudo-Inverse of Matrix

The pseudo-inverse always exists, even for matrices that are not invertible.

Introduction

The pseudo-inverse, also known as the Moore-Penrose inverse, is a generalization of the matrix inverse that can be used for matrices that are not invertible. It is a fundamental tool in linear algebra and has many practical applications in science, engineering, and data analysis.

Intuition

The intuition behind the pseudoinverse can be understood geometrically. Suppose we have a matrix A that is not full rank, which means that its columns do not span the full range of the vector space. The pseudoinverse of A can be thought of as a projection of the vector b onto the column space of A, followed by a “back projection” of the result onto the entire vector space. The resulting vector is the “closest” vector to b that lies in the column space of A. Here is a small example of what each step looks like.

import numpy as np

A = np.array([[2, 2], [4, 6], [4, 10]]) # define matrix A
b = np.array([1, 2, 4]) # define vector b

U, S, V = np.linalg.svd(A, full_matrices=False) # compute SVD of A
Sinv = np.diag(1/S) # take reciprocal of singular values
Aplus = V.T @ Sinv @ U.T # compute pseudoinverse of A

x = Aplus @ b # compute solution to Ax = b

print("U =\n", np.round(U,2))
print("S =\n", np.round(S,2))
print("V =\n", np.round(V,2))
print("Sinv =\n", np.round(Sinv,2))
print("Aplus =\n", np.round(Aplus,2))
print("x =\n", np.round(x,2))


# Output 
U =
 [[-0.2  -0.59]
 [-0.54 -0.6 ]
 [-0.81  0.55]]

S =
 [13.18  1.55]

V =
 [[-0.44 -0.9 ]
 [-0.9   0.44]]

Sinv =
 [[0.08 0.  ]
 [0.   0.65]]

Aplus =
 [[ 0.35  0.37 -0.29]
 [-0.15 -0.13  0.21]]

x =
 [-0.08  0.42]

The pseudo-inverse of a matrix A is denoted by A⁺. A⁺ and satisfies the following mathematical properties:

A * A⁺   * A   = A
A⁺ * A   * A⁺  = A⁺
(A * A⁺) ^ T   = A * A⁺
(A⁺ * A) ^ T   = A⁺ * A

In short, the important property that A⁺A and AA⁺ are both projection matrices. This means that they satisfy the conditions of a projection matrix, namely that they are symmetric, idempotent, and have a rank equal to the number of linearly independent columns or rows of A.

The pseudo-inverse has several important properties that are helpful (hence the importance of the concept of pseudo-inverse). Specifically, if we want to find the vector x that minimizes the distance ||Ax — b||, then the solution is given by x = A⁺b. A few important points -

General properties

1. The pseudo-inverse always exists, even for matrices that are not invertible.
2. The pseudo inverse is unique.
3. The pseudo-inverse of a product of matrices is the product of their pseudo-inverses in reverse order, i.e., (AB)⁺ = B⁺A⁺.
4. The pseudo inverse is the minimum norm solution to an underdetermined system of linear equations.
5. The pseudo inverse is used to solve the least-squares problem, which involves finding the solution to a system of linear equations that minimizes the sum of the squared errors.

Practical applications

1. Solving linear systems of equations that have no exact solution.
2. Data compression where it is used to approximate a matrix with a lower-rank matrix.
3. Image processing where it is used to restore missing or degraded information in an image.
4. Control theory where is used to control the motion of robots and other systems.
5. Machine learning where it is used in algorithms such as linear regression, principal component analysis, and support vector machines.

The pseudo-inverse of a matrix is a powerful mathematical tool that has many practical applications in science, engineering, and data analysis. It is a generalization of the matrix inverse that can be used for matrices that are not invertible, and it has many important properties and applications.

Use cases

In this article, we will explore some of the key applications of the pseudo inverse, including solving linear systems of equations, data compression, image processing, control theory, and machine learning. We will also provide examples of how the pseudo inverse is used in each of these applications using Python.

Solving Linear Systems of Equations

One of the most important applications of the pseudo inverse is in solving linear systems of equations that have no exact solution. In many real-world applications, it is common to encounter systems of equations that are underdetermined or overdetermined, meaning that there are either more equations than unknowns or more unknowns than equations.

When a system of equations is underdetermined, it has infinitely many solutions, and it is impossible to find an exact solution. However, the pseudo inverse can be used to find the minimum norm solution to the system, which is the solution that has the smallest Euclidean norm.

For example, consider the following system of equations:

3x + 2y = 5
2x + 3y = 7

This system is underdetermined since there are two unknowns and only two equations. A system of equations is considered “underdetermined” when the number of equations is less than the number of unknowns. In the case of a system with 2 equations and 2 unknowns, it is called “underdetermined” when the equations are not sufficient to determine unique values for all of the unknowns. However, the pseudo-inverse of the coefficient matrix can be found using the pinv function from the numpy library:

import numpy as np

A = np.array([[3, 2], [2, 3]])
b = np.array([5, 7])
A_pinv = np.linalg.pinv(A)
x = np.dot(A_pinv, b)

In an underdetermined system, there are more unknowns than equations, and so there may not be a unique solution that satisfies all of the equations. However, we can still find a solution that minimizes the distance between the solution and a target vector using the pseudo-inverse. For example, if the matrix A has more columns than rows, the system Ax = b is underdetermined. We compute the pseudoinverse of A using the np.linalg.pinv function, and then use this pseudoinverse to compute the solution to the system as x = A⁺b, where A⁺ is the pseudoinverse of A and b is the target vector.

The pinv the function returns the pseudo-inverse of the matrix A. The solution to the system of equations can then be found by multiplying the pseudo inverse by the vector of known values b.

Data Compression

Another important application of the pseudo-inverse is in data compression, where it is used to approximate a matrix with a lower-rank matrix. This is done by computing the singular value decomposition (SVD) of the matrix, and then truncating the SVD by keeping only the largest singular values and corresponding singular vectors.

The pseudo inverse is closely related to the SVD and can be used to reconstruct the original matrix from the truncated SVD. This technique is used in many data compression algorithms, such as image and video compression, as well as in data compression for storage and transmission.

For example, consider an image represented as a matrix A of size m x n. The SVD of A can be computed using the svd function from the numpy library:

U, s, V = np.linalg.svd(A)

The compressed image can be represented as A_k = U[:, :k] @ np.diag(s[:k]) @ V[:k, :], where k is the number of singular values to keep. To reconstruct the original image from the compressed version, we can use the pseudo inverse of A_k, which can be computed using the pinv function:

A_k_pinv = np.linalg.pinv(A_k)
A_reconstructed = A_k_pinv @ A_k

Image Processing

The pseudo inverse is also used in many image processing applications, such as image restoration, deblurring, and inpainting. These applications involve recovering missing or degraded information in an image, and the pseudo-inverse can be used to solve the underlying linear equations that govern the image formation process.

For example, consider the problem of image deblurring, where a linear system, such as a point spread function has blurred an image. The blurred image can be modeled as a convolution of the original image with the point spread function, and the goal is to recover the original image from the blurred one.

This problem can be formulated as a linear system of equations, where the vector of observed pixel values is related to the vector of original pixel values by a matrix that represents the convolution operation. The pseudo-inverse of this matrix can be used to recover the original image from the blurred one, by solving the linear system of equations.

Control Theory

The pseudo inverse is also used in control theory to solve the problem of least-squares inverse, which involves finding the minimum norm solution to an underdetermined system of linear equations. This is important in applications such as robotics, where it is necessary to control the motion of a robot arm using sensors and actuators.

For example, consider a robot arm with n joints that is being controlled by m sensors. The goal is to determine the joint angles that correspond to a desired position and orientation of the end effector. This problem can be formulated as an underdetermined system of linear equations, where the vector of joint angles is related to the vector of sensor readings by a matrix representing the robot arm's kinematics.

The pseudo-inverse of this matrix can be used to find the minimum norm solution to the system, which corresponds to the joint angles that minimize the deviation from the desired position and orientation. This solution can be used to control the motion of the robot arm and achieve the desired task.

For example, let’s consider a simple robot arm with two joints and two sensors. The kinematics of the robot arm can be represented by the following matrix A:

A = np.array([[np.cos(theta1), np.cos(theta2)],
              [np.sin(theta1), np.sin(theta2)]])

where theta1 and theta2 are the joint angles/orientations/positions, and the sensors measure the sine and cosine of the joint angles. If we want to move the end effector to a desired position x, we can formulate the problem as follows:

x = np.array([1, 1])
b = np.dot(A, x)

The goal is to find the joint angles that correspond to the desired position x. This can be done using the pseudo inverse of A:

A_pinv = np.linalg.pinv(A)
theta = np.dot(A_pinv, b)

The solution theta gives the joint angles that minimize the deviation from the desired position and orientation.

Machine Learning

The pseudo inverse is also used in many machine learning algorithms, such as linear regression, principal component analysis, and support vector machines. These algorithms rely on finding the pseudo-inverse of a matrix to estimate parameters, reduce dimensionality, and classify data.

For example, consider the problem of linear regression, where the goal is to find a linear function that predicts the output variable from the input variables. This problem can be formulated as a linear system of equations, where the vector of output values is related to the vector of input values by a matrix that represents the linear function.

The pseudo-inverse of this matrix can be used to estimate the parameters of the linear function by solving the linear system of equations. This solution can be used to make predictions on new input data and evaluate the performance of the algorithm.

For example, let’s consider a simple dataset with two input variables and one output variable:

X = np.array([[1, 2], [2, 3], [3, 4], [4, 5]])
y = np.array([2, 3, 4, 5])

We can estimate the parameters of the linear function using the pseudo inverse of X:

X_pinv = np.linalg.pinv(X)
w = np.dot(X_pinv, y)

The solution w gives the weights of the linear function, which can be used to make predictions on new input data:

x_new = np.array([5, 6])
y_new = np.dot(x_new, w)

Conclusion

In conclusion, the pseudo-inverse of a matrix is a powerful mathematical tool with many practical applications in science, engineering, and data analysis. Its importance lies in its ability to handle not-invertible matrices and find solutions to systems of linear equations that have no exact solution.

For my other life-saving content, please check out

Don’t forget to follow, like, and share the article. Thank you!

🔔 clap | follow | Subscribe 🔔

Become a member using my link: https://ithinkbot.com/membership