Image Template matching methods with Python using OpenCV

Background

Template matching is a computer vision technique for finding areas of an image that are similar to a patch (template). A patch is a small image with certain features. The goal of template matching is to find the patch/template in an image. To find it, the user has to give two input images: Source Image (S) — The image to find the template in, and Template Image (T) — The image that is to be found in the source image.

It is usually done by comparing the similarity between the template and different regions in the larger image using a similarity metric, such as cross-correlation or mean-squared error. The location of the region with the highest similarity score is considered the best match.
It is often used in image processing applications, such as object recognition, tracking, and image alignment. The main advantage of this method is its simplicity and speed, making it suitable for real-time applications. However, it can also be sensitive to variations in illumination, rotation, and scale, and may not always provide the most accurate results in these cases.

The threshold depends on the accuracy with which we want to detect the template in the source image. For instance, if we are applying face recognition and we want to detect the eyes of a person, we can provide a random image of an eye as the template and search for the source (the face of a person). In this case, since “eyes” show a large number of variations from person to person, even if we set the threshold as 50%(0.5), the eye will be detected. In cases where almost identical templates are to be searched, the threshold should be set high.(t>=0.8)

Methods

There are several other methods for image matching, including:

Feature-based matching: This method uses distinctive features in the images, such as corners, edges, and blobs, to find the correspondence between the images. Feature-based matching methods include SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), and BRISK (Binary Robust Invariant Scalable Keypoints).

# SIFT (Scale-Invariant Feature Transform)

import cv2
import numpy as np
from matplotlib import pyplot as plt

# Load the two images
img1 = cv2.imread('origin.png', cv2.IMREAD_GRAYSCALE)
img2 = cv2.imread("template.png", cv2.IMREAD_GRAYSCALE)

# Detect SIFT keypoints and descriptors in the images
sift = cv2.SIFT_create()
keypoints1, descriptors1 = sift.detectAndCompute(img1, None)
keypoints2, descriptors2 = sift.detectAndCompute(img2, None)

# Use the BFMatcher to find the best matches between the descriptors
bf = cv2.BFMatcher()
matches = bf.knnMatch(descriptors1, descriptors2, k=2)

# Define a threshold for the ratio test
threshold = 0.75

# Filter the matches using the Lowe's ratio test
good_matches = []
for m, n in matches:
    if m.distance < threshold * n.distance:
        good_matches.append(m)

# Draw the matches on the images
img_matches = cv2.drawMatches(img1, keypoints1, img2, keypoints2, good_matches, None, flags=cv2.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS)

# Display the results
plt.figure(figsize = (20,10))
plt.imshow(img_matches)

# ORB (Oriented FAST and Rotated BRIEF) Method:

import cv2
import numpy as np

# Load the two images
input_img = cv2.imread('origin.png')
template_img = cv2.imread("template.png")

# Convert the images to grayscale
input_gray = cv2.cvtColor(input_img, cv2.COLOR_BGR2GRAY)
template_gray = cv2.cvtColor(template_img, cv2.COLOR_BGR2GRAY)

# Detect ORB keypoints and descriptors in the images
orb = cv2.ORB_create()
keypoints1, descriptors1 = orb.detectAndCompute(input_gray, None)
keypoints2, descriptors2 = orb.detectAndCompute(template_gray, None)

# Use the BFMatcher to find the best matches between the descriptors
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = bf.match(descriptors1, descriptors2)

# Sort the matches based on their distances
matches = sorted(matches, key = lambda x:x.distance)

# Draw the matches on the images
img_matches = cv2.drawMatches(input_img, keypoints1, template_img, keypoints2, matches[:10], None, flags=cv2.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS)

# Display the results
plt.figure(figsize = (20,10))
plt.imshow(img_matches)

Deep learning-based matching: This method uses deep neural networks to learn features that are invariant to changes in the images, such as changes in scale, orientation, and illumination. Deep learning-based matching methods include Siamese networks and triplet networks.
Optical flow: This method uses the movement of pixels between consecutive frames to determine the correspondences between the images. Optical flow methods are commonly used in video processing and can be used to track objects or to estimate the motion of the camera.
Structural similarity index (SSIM): This method uses a perceptual quality metric to measure the similarity between two images. The SSIM index is based on the structural information in the images, such as the mean, standard deviation, and correlation between the pixels, and it provides a measure of the visual quality of the images.
Correlation-based matching: This method uses cross-correlation or normalized cross-correlation to find the correspondence between the images. Correlation-based matching is similar to template matching, but it can be used for more general image matching tasks, not just for finding a match between a template and an image.

import cv2
import numpy as np

# Load the two images
input_img = cv2.imread("origin.png")
img2 = cv2.imread("template.png")

# Convert the images to grayscale
input_gray = cv2.cvtColor(input_img, cv2.COLOR_BGR2GRAY)
template_gray = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)

# Find the dimensions of the template image
w, h = template_gray.shape[::-1]

# Use the cv2.matchTemplate() function to find the correlation between the images
result = cv2.matchTemplate(input_gray, template_gray, cv2.TM_CCOEFF_NORMED)

# Set the threshold for the correlation coefficient
threshold = 0.2
loc = np.where(result >= threshold)

# Check if there are any good matches
if len(loc[0]) > 0:
    # Draw a rectangle around each good match
    for pt in zip(*loc[::-1]):
        cv2.rectangle(input_img, pt, (pt[0] + w, pt[1] + h), (0, 0, 255), 2)
    print("Good match found.")
else:
    print("No good match found.")

# Find the location of the maximum value in the result image
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)

# Draw a rectangle around the maximum value in the result image
top_left = max_loc
bottom_right = (top_left[0] + img2.shape[1], top_left[1] + img2.shape[0])
img_matches = cv2.rectangle(input_img, top_left, bottom_right, 255, 2)

# Display the results
plt.figure(figsize = (20,10))
plt.imshow(img_matches)

Histogram

Template matching with histograms involves comparing the histograms of the template image and the target image regions to find the best match.

Here’s an example of how to perform template matching with histograms using OpenCV Python:

# Template matching with histograms involves comparing the histograms of 
# the template image and the target image regions to find the best match.
import cv2
import numpy as np

# Load the template image
template = cv2.imread("template.png")

# Convert the template image to grayscale
template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)

# Calculate the histogram of the template image
template_hist = cv2.calcHist([template],[0],None,[256],[0,256])

# Load the target image
target = cv2.imread('origin.png')

# Convert the target image to grayscale
target = cv2.cvtColor(target, cv2.COLOR_BGR2GRAY)

# Define the size of the template
template_height, template_width = template.shape

# Define the size of the target image
target_height, target_width = target.shape

# Create a result image to store the match results
result = np.zeros((target_height - template_height + 1, target_width - template_width + 1), np.float32)

# Loop through all the regions of the target image
for i in range(target_height - template_height):
    for j in range(target_width - template_width):
        # Get the region of the target image
        region = target[i:i + template_height, j:j + template_width]

        # Calculate the histogram of the region
        region_hist = cv2.calcHist([region],[0],None,[256],[0,256])

        # Calculate the similarity between the histograms
        similarity = cv2.compareHist(template_hist, region_hist, cv2.HISTCMP_CORREL)

        # Store the similarity in the result image
        result[i][j] = similarity

# Find the location of the best match
_, _, _, match_location = cv2.minMaxLoc(result)

# Draw a rectangle around the best match
cv2.rectangle(target, match_location, (match_location[0] + template_width, match_location[1] + template_height), (255,0,0), 2)

# Display the result
plt.figure(figsize = (20,10))
plt.imshow(target)

The choice of the image matching method depends on the specific requirements of the application, such as the type of images being used, the desired level of accuracy, and the computational resources available. In practice, a combination of methods may be used, and the best method for a particular application may need to be determined through experimentation and optimization.

Limits

Template matching has some limitations that can affect its performance:

Sensitivity to scale and rotation: Template matching is often sensitive to changes in scale and rotation between the template and the larger image. If the template and the target image do not have the same scale or orientation, the method may not find a good match.
Illumination variations: Changes in lighting conditions can also impact the performance of template matching. Shadows, reflections, and other illumination variations can cause the appearance of the template to change, making it more difficult to find a good match.
Background clutter: If the background in the target image is cluttered or has patterns similar to the template, this can interfere with the ability of the method to find the correct match.
Noise: Noisy images can also impact the performance of template matching, as the noise can affect the similarity score and cause false matches to be detected.
Limited to small templates: Template matching is most effective when the template is small relative to the target image. If the template is too large, it can become difficult to find a good match, as the search space becomes too large.

In general, template matching can be a fast and effective method for finding small, well-defined objects in images, but it may not always provide the most accurate results in more complex scenarios. In these cases, alternative methods, such as feature-based matching or deep learning-based object detection, may be more appropriate.

Optimise

To avoid the issue caused by the different sizes of the template and original image we can use multiscaling. In the case where, just because the dimensions of your template do not match the dimensions of the region in the image you want to match, does not mean that you cannot apply template matching.

Multiscaling is a mechanism used in template matching to improve the accuracy of the matching process by searching for the template at multiple scales. The idea is to search for the template at different sizes, starting from a smaller size and increasing the size until the template size becomes the same as the original size. This allows the template matching process to detect the template even if it appears at different scales in the target image.
Multiscaling can be achieved by using the cv2.resize() function in OpenCV to resize the template to different scales. The resized templates are then used to perform template matching on the target image, and the results are stored for each scale. Finally, the scale with the highest similarity score is selected as the best match. The threshold value is used to determine the best match.

Here are the steps to perform template matching with multiscaling in OpenCV:

Load the target image and the template: Use the cv2.imread() function to load the target image and the template as grayscale images.
Store the height and width of the template: Store the height and width of the template in separate variables. These values will be used to draw the rectangle around the best match in the target image.
Create a list to store the result of each scale: Create an empty list to store the result of each scale.
Loop through different scales of the template: Use a for loop to iterate through different scales of the template, ranging from a smaller scale to a larger scale. The scale values can be obtained using the np.linspace() function.
Resize the template: Use the cv2.resize() function to resize the template to the current scale.
Check if the resized template is smaller than the target image: Check if the resized template is smaller than the target image. If the resized template is larger, break the loop.
Perform template matching: Use the cv2.matchTemplate() function to perform template matching on the target image using the resized template. Use the cv2.TM_CCOEFF_NORMED method.
Store the result of the current scale: Store the result of the current scale in the results list.
Find the scale with the highest similarity score: Use the max() function with a lambda function to find the scale with the highest similarity score.
Get the location of the best match: Use the np.where() function to get the location of the best match.
Draw a rectangle around the best match: Use the cv2.rectangle() function to draw a rectangle around the best match in the target image.
Display the result: Use the cv2.imshow() function to display the result, and use the cv2.waitKey() and cv2.destroyAllWindows() functions to wait for a key event and destroy the window, respectively.

By following these steps, you can perform template matching with multiscaling in OpenCV and improve the accuracy of the matching process.

import cv2
from matplotlib import pyplot as plt
# Load the template and the image
template = cv2.imread('template.png', 0)
image = cv2.imread('origin.png', 0)

# Define the scales to use for the search
scales = [0.2, 0.5, 1.0, 2.0]

# Loop over each scale
for scale in scales:
    # Resize the template to the current scale
    template_scaled = cv2.resize(template, None, fx=scale, fy=scale)

    # Perform template matching using the scaled template
    result = cv2.matchTemplate(image, template_scaled, cv2.TM_CCOEFF_NORMED)

    # Find the maximum correlation value and its location
    min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)

    # Calculate the top-left and bottom-right corners of the bounding box
    top_left = max_loc
    bottom_right = (top_left[0] + template_scaled.shape[1], top_left[1] + template_scaled.shape[0])

    # Draw a rectangle around the matched region
    cv2.rectangle(image, top_left, bottom_right, 255, 2)

# Display the result
plt.figure(figsize = (20,10))
plt.imshow(image)

In this code, we first load the template and the image to be searched. We then define a list of scales to use for the multi-scale search. For each scale, we resize the template to the current scale, perform template matching using the cv2.matchTemplate function, and find the maximum correlation value and its location. We then calculate the top-left and bottom-right corners of the bounding box for the matched region, and draw a rectangle around it on the image. Finally, we display the result using cv2.imshow.

nphausg - Overview

You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…

github.com