3D Photography Inpainting: Exploring Art with AI.

Usage of the new model, with examples and Colab Notebook.

We are living in a Great Epoche of Experiments. Well, science, society, and culture experience new narratives all the time. Think about Renaissance or Dadaist movements. But data science, empowered by the current state of the computational periphery, allows us to do something beyond our imagination. Machine learning was the initial start of the new journey into the unknown. Deep (and unsupervised) learning is a step further.

Trained on specific datasets, AI can visualize its perception of reality — or augment existing concepts. Sure, it depends on training sources and capabilities. But the same is true about us humans — we understand what we know. And we know what we understand.

A Glimpse behind the Artwork.

I always wondered about the ability of AI to interpret reality — and art — in a particular case. Also: isn’t it an exciting thing to explore “hallucination” by machine?

Like our brain complements the lack of information in a blind spot (see also: Scotoma) with “filling in” non-perceived visuals, so AI “inpaints” missing areas with something it “assumes” should be there.

Sometimes it works in surprising directions. ArtBreeder, a StyleGAN based web application “corrected” my uploaded image in an unexpectable way:

AI Experiment “Image Inpainting” by NVIDIA AI delivered me some creepy results:

Here is another fantastic inpainting glitch, observed by Jonathan Fly:

3D Ken Burns effect by Simon Niklaus et al. provided us with a wholly new thrilling dimension — using a single image, it analyzes the occlusion, depth, and geometry overall, which allows us to “enter inside” a photo.

You can bring historic photos to life:

Or even you can take a step further, behind the image:

3D Ken Burns effect works perfectly on photos or images with a clear perspective. It becomes complicated when depth is not easily definable:

3D Photography Inpainting

With their paper “3D Photography using Context-aware Layered Depth Inpainting” (PDF) has a team from Virginia Tech, National Tsing Hua University, and Facebook brought the exploration of an image to another level.

3D Photography using Context-aware Layered Depth Inpainting:
Project Website / Paper / GitHub / Colab Notebook

Team: Meng-Li Shih (1,2)
Shih-Yang Su (1)
Johannes Kopf (3)
Jia-Bin Huang (1)

1) Virginia Tech   
2) National Tsing Hua University   
3) Facebook

On their Project Website, you can see a variety of demonstration videos.

And here is a video comparison between various existing models:

The novel method with multi-layered depth brings more dimension into the game. The background behind the movable objects is still filled in with inpainting, but the three-dimensionality of the results are stunning.

Using CNN-based depth estimation and image preprocessing, the model continues invisible areas in a connected way: context-aware continuation of lines and patterns works very convincingly.

Compare the image by Magritte I did with the new model:

Or sneak inside the photo portrait of Salvador Dalí and Frederico Garcia Lorca:

Exploring art with 3D Photo Inpainting

As you see, the interpretation of hidden layers by machine is fascinating.

The results in the demo video are more than convincing, but that modified portrait of Picasso intrigued me at most:

The photo used here depicts Picasso in front of “La Cuisine” (1948) in his Rue des Grands-Augustins studio. And the painting areas behind the artist are filled in by AI.

So what does happen within the brain of the machine? I did some experiments and was overwhelmed.

The Landscapes behind Jesus in Da Vinci’s “The Last Supper” are clear and unique:

Another Magritte — feel the space in surreal dreams:

René Magritte is famous for his illusions and plays with layers of realities. In his work “L’Heureux donateur” the beholder cannot recognize, whether it’s a silhouette of a man with a hat painted over with a night scene — or is it a cut-out in the shapes of the man, leading us to the landscapes beyond.

3D Photo Inpainting brings its interpretation:

Andrew Wyeth’s desolate world becomes more vivid:

Piranesi’s utopian post-urban structures get their depth:

In these examples, you can clearly see the depth layers, being generated by CNN in a very sophisticated and accurate way. Geometry and content-aware image inpaiting show the future of image exploration.

The method also includes 3D maps of the images, which can be used in other implementations.

Twitter is full of experiments with 3D Photo Inpainting at the moment. Here by RoadRunner01 (btw. a wonderful must-follow Twitterer, source of newest AI models and experiments):

Also, historical photography becomes more expressive with the use of this method. Old Ireland in Colour experiments with DeOldify (AI-based colorization) and 3D Ken Burns effects on historical Ireland photography. 3D Photo Inpainting is used here perfectly:

Artist and Digital Explorer Mario Klingemann explores in a thread “The Garden of Earthly Delights” by Bosch:

As a summary of the last week (as 3D Photo Inpaiting was published), he writes:

Indeed, it’s time for exploration and experiments with Artificial Intelligence interpreting art.

UPDATE. This Colab by Manuel Romero is designed for converting images to video without time-consuming 3D meta-data. Multiple file upload and download is available too. If you want to convert bunch of photos — this Colab is for you.

What will you find, dear readers?