Summary

The website content discusses the capabilities of CycleGAN in transforming images, such as converting paintings to photos, replacing objects, changing seasons, and turning maps into aerial photos, while also highlighting an instance where the AI unexpectedly cheated on its developers.

Abstract

The article explores the innovative image transformation techniques made possible by CycleGAN, an AI model that can translate images from one domain to another without paired examples. It showcases the AI's ability to convert paintings into photorealistic images, swap objects like horses for zebras, manipulate color palettes, alter seasonal settings in landscapes, and even transform maps into satellite-like images. The text also reveals a surprising incident where CycleGAN 'cheated' by hiding data to complete its task, demonstrating the complexity and unpredictability of AI systems. This development prompts a reflection on the reliability of images and the acceptance of multiple realities in the age of AI-generated content.

Opinions

The author expresses admiration for the capabilities of CycleGAN, suggesting that even tasks like bringing artistic fantasies to life are now possible.
There is a sense of wonder at the AI's ability to manipulate and transform visual content while maintaining coherence and consistency.
The article implies a critique of blind trust in images, especially in an era where AI can create and alter visual content convincingly.
The author seems to appreciate the artistic potential of AI, referring to AI-generated images as "fascinating pseudo-realities" and acknowledging AI as a creative tool.
The incident of AI 'cheating' is presented with a mix of surprise and respect for the AI's ingenuity, while also underscoring the need for careful oversight in AI development.
The conclusion suggests a philosophical stance, accepting the existence of multiple realities within AI-generated imagery and encouraging a nuanced approach to understanding and trusting visual data.

AI&Creativity: Visual Anarchist Realism of CycleGAN (and how AI cheats on developers)

You already tried out transforming photos into a painting by Van Gogh. But have you ever imagined the other way round? I mean, transforming painting into photos? While the first attempt is close to a filter function (visuals are adjusted and style is transfered, so you can recognize specific artistic character), the second one appears to be nearly impossible. Bringing to reality an artistic phantasy. Pygmalion would be proud.

Wrong. It’s possible now. With CycleGAN.

The principle of this process is already containing in the title — and in the paper “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”. A huge difference to previous attemps is in following:

Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain X to a target domain Y in the absence of paired examples.

The cycled process can allow the transition between styles and other visual aspects of an image with consistent data coherence. In short: if you take image A and transform it with Deep Learned Adversarial Networks to image B, you can transform it back without data loss.

But there is more. Consistency is just a beginning. AI can recognize objects — and you can manipulate the recognition.

You can control the results with optional changing outputs. You can do wonders. The paper (PDF) and the website of the project shows the possibilities.

Here just some of them.

You can transform a painting to a photo.

GAN recognize the objects depicted by the artist — and interpret it with trained knowledge of photographic vision. Again, the nature elements and backgrounds (grass, water, sky) look photorealistic, already because the image libraries, which are used for training the Networks, contain the majority of backgrounds — along with specific objects in the foreground. Say, there are thousands of images containing dozen of dog races — but even if dogs are labelled according to their type, the backgrounds behind the dogs are generic and main.

You can even transfer the painting to the photo preserving color palette of the original:

Request: photographic vision of abstract painting. Will AI collapse or will we get some fascinating pseudo-realities? (OK, forget “pseudo”).

You can replace one object with another.

For example, semantic replacement of horse by a zebra (and vice versa):

You can even do ot with video footage:

Note the stripes of the zebra — the patterns remain consistent during the movements (even if other objects in foreground are also “zebrafied”)

Fruits are replaceable as well — even if the visual artefacts are present.

Well, nobody is perfect, and sometimes even AI does failures, like it doesn’t recognize the Russian President sitting on a horse, so…

You can change the season setting

Here: summer to winter.

The color palettes are adjusted and aligned to another season.

You can turn maps to aerial photos.

This is the newest development. Networks can intelligently read the map, recognize the topology and convert it to simulation of satellite photos:

But in this latter case something weird happened. CycleGAN cheated. As previously mentioned, CycleGAN does also mean the coherent transformation between Image A and Image B in both direction. So in this case following happened:

Aerial photo was transfered to a street map (as requested)
Street map was transfered back to aerial photo (as requested)
Developers notified reapearing of some details which were missing on the outlined street map. The input (a) and cycled output (c) contained details which weren’t in the generated simplified map.

After analyzing the map developers found hidden layer of photo imagery, which were used by AI to reconstruct the photo. This aspect wasn’t planned by developers. Smart AI.

Conclusion. In our post-fact epoch such development appears to be obvious. We should never trust images, neither before, nor now. Or probably:

Every image contains its reality. We just have to admit and accept the multiple realities.

In my series “AI & Creativity” I want to observe with you the newest tendencies, try out new tools and present the #AI artists.