Visual Perception: Epipolar Geometry
Epipolar Geometry basics like Essential Matrix, Fundamental Matrix, Triangulation, Feature Matching are explained with their MATLAB code implementations
We learned how to use a projective camera and how to calibrate it. In this post, we will learn Epipolar Geometry, which is the intrinsic projective geometry between two views. Two views mean now we will learn how to work with more than 1 camera to perform different tasks!
Let’s take a look at Epipolar Geometry structure and its base terms:
By looking at these terms, we can understand that we will have an epipolar line for each projected point, whereas we have only 1 epipole for 1 image plane. And all the epipolar lines on the same image plane intersects on the epipole.
Another brief note I want to add is that don’t forget that until now we worked only with 1 camera which means we worked with monocular cameras while for epipolar geometry we need at least 2 monocular or 1 stereo camera which provides us multiple views.
What are we able to do using this geometry?
- We can reconstruct 3D world points using the pairs of 2D points coming from the left and right images.
- We can find camera intrinsic and extrinsic parameters using 2D matched points
Let’s start with our first task to find the match of a 2D point we have in 1 image in the other image plane
Essential Matrix
Essential Matrix is the base 3x3 matrix that encodes the epipolar geometry of two views where the cameras are already calibrated.
Since CC’, Cq, and C’q’ vectors are three vectors that lie on the same epipolar plane we have the following equations:
Finally, we have the relationship between the Essential matrix and a 2D image point pair!
Properties of Essential Matrix
- l = εq’ and l’ = Transpoze(ε)q
- εe’ = 0 and Transpoze(ε)e = 0
- It’s a 3x3 matrix with 5 DOF (degrees of freedom) since we already know camera intrinsic parameters (we work with calibrated cameras). Therefore 3 rotation + 3 translation parameters-1 scale factor = 5 unknown parameters
- It is a singular matrix with rank 2
- While Homography Matrix maps a point to a point, Essential Matrix maps a point to a line
Estimation of Essential Matrix
Now it’s time to learn how to estimate the Essential Matrix between two views. Then we can use this matrix to find the right epipolar line of a left image plane point and vice versa, likewise the epipoles.
Fundamental Matrix
We learned what Essential Matrix is and how to estimate it. But what if we don’t have calibrated cameras and we need to work in this epipolar plane having uncalibrated cameras? As we know, Essential Matrix only works with calibrated cameras and this is where Fundamental Matrix comes out!
In a very similar way with Essential Matrix, we can calculate 3x3 Fundamental Matrix which has 8 DOF this time since we don’t have intrinsic parameters too! [1]
Properties of Fundamental Matrix
- l = Fq’ and l’ = Transpoze(F)q
- Fe’ = 0 and Transpoze(F)e = 0
- It’s a 3x3 matrix with 8 DOF (degrees of freedom) since we don’t know camera intrinsic parameters at that time.
- It is a singular matrix with rank 2
To repeat, don’t forget that we can use Fundamental Matrix for every task we talked about for Essential Matrix.
Some additional and important notes :
- We can calculate camera projection matrices using Fundamental Matrix using the following equations: P = [I | 0] P’ = [[e’]xF | e’] where [e’]x is the skew matrix of e’, P is the left side camera projection matrix and P’ is the right side projection matrix.
- The relationship between Essential Matrix and Fundamental Matrix is E = K’ F K.
- You can come across “5-Point Algorithm” and “8-Point Algorithm” in different resources, please note that they refer to the Essential Matrix estimation and Fundamental Estimation
We learned about the Essential Matrix and Fundamental Matrix and finally, it is time to examine the MATLAB code implementation to estimate and use them! In the below code you will find Fundamental Matrix estimation, epipolar lines calculation using Fundamental Matrix, epipoles calculation using Fundamental Matrix, camera projection matrices calculation using Fundamental Matrix
You can reach the code and example point matches used for this implementation by this Github link
Triangulation
Triangulation is the reconstruction of a 3D point using 2D point matches and Projection Matrices.
Now it’s nothing more than to solve that equation again with DLT and SVD as we almost did in every problem in this series. Here is the MATLAB implementation of that part. Note that I had to obtain 2D point pairs first since I don't have any example points and I wanted to share that part in the code too.
You can reach the related code and 3D points used for that example from this Github link.
Feature Matching
Until here, we learned how to use 2D image point pairs to obtain Essential Matrix, Fundamental Matrix and to apply Triangulation. But a very important base point is missing. How can we obtain 2D image point pairs?
After obtaining different images from 1 view, we need to extract the features in the image and match them to obtain these 2D point matches!
I won’t dive into this part but will give you some important terms just to give you some idea of what to do.
- Harris Corner Detection algorithm is a very popular one to detect features in an image.
- After detecting these features in different images, you can describe these features using different methods like SIFT, SURF, FAST, BRIEF, ORB. This step is important before matching the features. Here is a very good post that explains these methods step by step. https://readmedium.com/introduction-to-feature-detection-and-matching-65e27179885d
- Using these feature descriptors, you can use Brute Force Matcher or FLANN methods to match these keypoints and obtain 2D features matches.
- Applying RANSAC optimization after obtaining your matches is always a good idea to remove outliers (false matches)
I implemented the Harris Corner detection algorithm in MATLAB if you want to take a look:
You can reach the related code and the example images in this Github link.
After learning Visual Perception fundamentals, you can take a look at Visual Perception GUI in the next post.
Thank you!
[1] https://stackoverflow.com/questions/49763903/why-does-fundamental-matrix-have-7-degrees-of-freedom
[2] Multiple View Geometry in Computer Vision, Richard Hartley, Andrew Zimmerman
