GAN — Some cool applications of GAN

We make impressive progress in the first few years of GAN developments. No more stamp-size facial pictures like those in horror movies. In 2017, GAN produced 1024 × 1024 images that can fool a talent scout.

Larger resolution with better quality in the first 2 years.

In the coming years, we will probably see high-quality videos generated from GANs. The commercial applications will come! As part of the GAN series, we look into some cool applications and hope that they become the inspiration for your GAN application.

Create Anime characters

Game development and animation production are expensive and hire many production artists for relatively routine tasks. GAN can auto-generate and colorize Anime characters.

Towards the automatic Anime characters creation with Generative Adversarial Networks

The generator and the discriminator composes of many layers of convolutional layers, batch normalization and ReLU with skip connections.

Pose Guided Person Image Generation

With an additional input of the pose, we can transform an image into different poses. For example, the top right image is the ground truth while the bottom right is the generated image.

The refined result column below is the generated images.

The design composes of a 2-stage image generator and a discriminator. The generator reconstruct an image using the meta-data (pose) and the original image. The discriminator uses the original image as part of the label input to a CGAN design.

CycleGAN

Cross-domain transfer GANs will be likely the first batch of commercial applications. These GANs transform images from one domain (say real scenery) to another domain (Monet paintings or Van Gogh).

For example, it can transform pictures between zebras and horses.

CycleGAN builds 2 networks G and F to construct images from one domain to another and in the reverse direction. It uses discriminators D to critic how well the generated images are. For example, G converts real images to Van Gogh style painting and Dy is used to distinguish whether the image is real or generated.

Domain A ➝ Domain B:

We repeat the process in the reverse direction Domain B➝ Domain A:

StarGAN

StarGAN is an image-to-image translation for one domain to another. For example, given a happy face, we want to transform it into a fearful face.

In (b), the generator generates a fake image based on an input image and a target domain label (say angry). In (c), given this fake image and the original domain of the image (say happy), it reconstructs the image using the generator. In (d), we feed real and fake images to the discriminator to label it as real or not as well as its domain classification. The cost function will involve reconstruction errors as well as the discriminator cost in identifying the images and their labels.

PixelDTGAN

Suggesting merchandise based on celebrity pictures has been popular for fashion blogger and e-commerce. PixelDTGAN creates clothing images and styles from an image.

Super resolution

Create super-resolution images from the lower resolution. This is one area where GAN shows very impressive result with immediate commercial possibility.

Similar to many GAN designs, it composes of many layers of convolutional layer, batch normalization, advanced ReLU and skip connections.

Progressive growing of GANs

Progressive GAN is probably one of the first GAN showing commercial-like image quality. Below is 1024 × 1024 celebrity look images created by GAN.

It applies the strategy of divide-and-conquer to make training much feasible. Layers of convolution layers are trained once at a time to build images of 2× resolution.

In 9 phases, a 1024 × 1024 image is generated.

StyleGAN2

StyleGAN2 generates high-resolution images.

High-resolution image synthesis

This is not image segmentation! It is the reverse, generating images from a semantic map. Collecting samples are very expensive. We have trying to supplement training dataset with generated data to lower development cost. It will be handy to generate videos in training autonomous cars rather than see them cruising in your neighborhood.

Network design:

GauGAN

GauGAN synthesizes photorealistic images given an input semantic layout.

Text to image (StackGAN)

Text to image is one of the earlier application of domain-transfer GAN. We input a sentence and generate multiple images fitting the description.

Text to Image Synthesis

Another popular implementation:

Face synthesis

Synthesis faces in different poses: With a single input image, we create faces in different viewing angles. For example, we can use this to transform images that will be easier for face recognition.

Image inpainting

Repair images have been an important subject decades ago. GAN is used to repair images and fill the missing part with created “content”.

Learn Joint Distribution

It is expensive to create GANs with different combinations of facial characters P(blond, female, smiling, with glasses), P(brown, male, smiling, no glasses) etc…The curse of dimensionality makes the number of GANs to grow exponentially. Instead, we can learn individual data distribution and combine them to form different distributions. i.e. different attribute combinations.

DiscoGAN

DiscoGAN provides matching style: many potential applications. DiscoGAN learns cross domain relationship without labels or pairing. For example, it successfully transfers style (or patterns) from one domain (handbag) to another (shoe).

DiscoGAN and CycleGAN are very similar in the network design.

Pix2Pix

Pix2Pix is an image-to-image translation that get quoted in cross-domain GAN’s paper frequently. For example, it converts a satellite image into a map (the bottom left).