Free AI web copilot to create summaries, insights and extended knowledge, download it at here

Abstract

n>Figure 1. SimCLR Overview. Image retrieved from <a href="https://arxiv.org/abs/2002.05709">https://arxiv.org/abs/2002.05709</a>.</figcaption></figure><p id="9fb5">SimCLR is the first paper to suggest using contrastive loss for self-supervised image recognition learning through image augmentations.</p><p id="c5b4">By generating positive pairs by doing data augmentation on the same image and vice versa, we can allow models to learn features to distinguish between images without explicitly providing any ground truths.</p><h1 id="ae8d">Momentum Contrast (MoCo)</h1><figure id="5185"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*6n6oUs05g-0uzYe-naeZBw.png"><figcaption>Figure 2. MoCo Overview. Image retrieve from <a href="https://arxiv.org/abs/1911.05722">https://arxiv.org/abs/1911.05722</a>.</figcaption></figure><p id="496b">The previous InfoNCE loss is proposed on a mini-batch of one positive and a number of negatives. He et al. extended this concept by portraying the contrastive learning as analogous to learning to match the best key with a given queue. The intuition led to the foundation of momentum contrast (MoCo), which is essentially a dictionary/memory network of key and values with key stored across multiple batches and slowly eliminating the oldest batch in a queue-like manner. This allows the training to be more stable as it is similar to a momentum where the change in keys is less drastic.</p><h1 id="0b09">Decoupled Contrastive Learning</h1><figure id="8f61"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*4WWk1HTMlMwVTU5jxUUGBw.png"><figcaption>Figure 3. The improvements of by swapping InfoNCE with DCL. Image retrieved from <a href="https://arxiv.org/abs/2110.06848">https://arxiv.org/abs/2110.06848</a>.</figcapt

Options

ion></figure><p id="e8ea">Previous papers in contrastive learning either required large batch sizes or a momentum mechanism. The recent paper decoupled contrastive learning (DCL) hope to change this by bringing a simple change to the original InfoNCE loss: simply removing the positive pair from the denominator.</p><p id="3f86">While seemingly simple, DCL actually allows better convergence and ultimately formed a even better baseline compared to previous papers SimCLR and MoCo.</p><h1 id="886d">Testing Each Concept</h1><p id="da62">Codes of the papers above have been provided by the authors. To test these concepts, one can simply download different datasets to see how well the unsupervised learning method works.</p><p id="04b7">It is important to keep note that these tasks often require your own dataloaders instead of using the ones given by PyTorch. One place you can retrieve datasets such as ImageNet, CIFAR100, and other notable image datasets is <a href="https://gas.graviti.com/open-datasets">here</a>.</p><h1 id="94b7">End Note</h1><p id="a61f">And there you have it! A short article on the recent advances of contrastive learning papers in unsupervised learning for image recognition. This article only covers some of the work under this scope. Many great papers such as <a href="https://arxiv.org/abs/2104.14548">NNCLR</a> and <a href="https://arxiv.org/abs/2003.04297">MoCo v2</a> are also worth a read. Hopefully you had fun in understanding these elegant yet effective concepts!</p><p id="4da8"><i>Thank you for making it this far </i>🙏<i>!</i> <i>I will be posting more on different areas of computer vision/deep learning, so <a href="https://taying-cheng.medium.com/membership">join and subscribe</a> if you are interested to know more!</i></p></article></body>

Contrastive Learning in 3 Minutes

The exponential progress of contrastive learning in self-supervised tasks

Deep learning research has been steered towards the supervised domain of image recognition tasks, many have now turned to a much more unexplored territory: performing the same tasks through a self-supervised learning manner. One of the cornerstones that lead to the dramatic advancements in this seemingly impossible task is the introduction of contrastive learning losses. This article dives into some of the recently proposed contrastive losses that have pushed the results of unsupervised learning to heights similar to supervised learning.

InfoNCE Loss

One of the earliest contrastive learning losses proposed was the InfoNCE loss by Oord et al. Their paper Representation Learning with Contrastive Predictive Coding proposed the following loss:

where the numerator is essentially the output of a positive pair, and the denominator is the sum of all value of positive and negative pairs. Ultimately, this simple loss forces the positive pairs to have a greater value (pushing the log term to 1 and hence less to 0) and negative pairs further apart.

SimCLR

SimCLR is the first paper to suggest using contrastive loss for self-supervised image recognition learning through image augmentations.

By generating positive pairs by doing data augmentation on the same image and vice versa, we can allow models to learn features to distinguish between images without explicitly providing any ground truths.

Momentum Contrast (MoCo)

Figure 2. MoCo Overview. Image retrieve from https://arxiv.org/abs/1911.05722.

The previous InfoNCE loss is proposed on a mini-batch of one positive and a number of negatives. He et al. extended this concept by portraying the contrastive learning as analogous to learning to match the best key with a given queue. The intuition led to the foundation of momentum contrast (MoCo), which is essentially a dictionary/memory network of key and values with key stored across multiple batches and slowly eliminating the oldest batch in a queue-like manner. This allows the training to be more stable as it is similar to a momentum where the change in keys is less drastic.

Decoupled Contrastive Learning

Figure 3. The improvements of by swapping InfoNCE with DCL. Image retrieved from https://arxiv.org/abs/2110.06848.

Previous papers in contrastive learning either required large batch sizes or a momentum mechanism. The recent paper decoupled contrastive learning (DCL) hope to change this by bringing a simple change to the original InfoNCE loss: simply removing the positive pair from the denominator.

While seemingly simple, DCL actually allows better convergence and ultimately formed a even better baseline compared to previous papers SimCLR and MoCo.

Testing Each Concept

Codes of the papers above have been provided by the authors. To test these concepts, one can simply download different datasets to see how well the unsupervised learning method works.

It is important to keep note that these tasks often require your own dataloaders instead of using the ones given by PyTorch. One place you can retrieve datasets such as ImageNet, CIFAR100, and other notable image datasets is here.

End Note

And there you have it! A short article on the recent advances of contrastive learning papers in unsupervised learning for image recognition. This article only covers some of the work under this scope. Many great papers such as NNCLR and MoCo v2 are also worth a read. Hopefully you had fun in understanding these elegant yet effective concepts!

Thank you for making it this far 🙏! I will be posting more on different areas of computer vision/deep learning, so join and subscribe if you are interested to know more!