The Next Rembrandt: bringing the Old Master back to life

Case study: behind the scenes of digital design

Blurring the boundaries between art and technology, The Next Rembrandt is a 3D printed painting, made solely from data of Rembrandt’s body of work. It was created using deep learning algorithms and facial recognition techniques. Capturing the zeitgeist of 2016, it went round the world and back again. We look at the story behind the painting that has won over 60 advertising awards.

As well as being a great example of what happens when fine art meets AI, The Next Rembrandt also challenges the notion of what creativity really is — sparking no end of discussion in the creative, art and technology industries. For Dutch agency J. Walter Thompson Amsterdam, it was a chance to test the boundaries and redefine what an advertising campaign is and could be.

Creative Partner Bas Korsten (J. Walter Thompson Amsterdam): “The Next Rembrandt really touched a nerve. It raised all kinds of questions about creativity. The reactions were sometimes heated, and certainly not all positive. Especially not from the art world. However, that bit of controversy helped us getting people to write, tweet and talk about it.”

What was the client brief? Bas Korsten: “Going back to the start — the brief behind The Next Rembrandt was simple. ING prides itself on being an innovative bank. It also plays a big role in sponsoring arts and culture in the Netherlands. A domain that is known for being traditional rather than contemporary. Focusing mainly on Dutch heritage and the Old Masters. And there is absolutely nothing wrong with that. But how does it fit with a technology-driven financial institution? And that was our brief exactly: bring innovation to ING’s sponsorship of Dutch arts and culture. We chose the Netherlands’ most iconic artist, Rembrandt.”

What design choices did you make and why? Innovation Director, Emmanuel Flores (J. Walter Thompson Amsterdam): “In terms of design choices, it was a fairly complex process. We started the design journey by gathering the complete collection of images of all 346 Rembrandt paintings. We used high-resolution scans provided by TU Delft and the Mauritshuis Museum, but we also used images from other sources. This meant that the resolution across all of the images was inconsistent. To solve this problem, the team used a Deep Neural Network algorithm to upscale the images, increasing the resolution by 300% and reducing visual noise.”

“After gathering all the data, we had to determine the subject of our 3D painting, by analysing who it was Rembrandt painted. After classifying over 400 faces, the data concluded that the subject should be around 30 to 40 years old, male, wearing black clothing, a hat and facing the right. The next task was to develop software that could understand Rembrandt, based on his use of geometry, composition and painting materials. Together with Microsoft, we started to analyse specific features such as eyes, nose, mouth and ears, by mapping 67 landmarks per face.”

Over 6,000 facial landmarks were used to classify the features, from the perspective of relevance and recurrence. The computer learned how to create a Rembrandt face based on these ‘typicalities’. Once the individual features were generated, we assembled them into a fully formed face and bust , according to Rembrandt’s use of proportions.

In order to move from a 2D image to a 3D printed painting, the team had to add another dimension. They did this by combining three distinct layers: canvas, ground layer and brushstrokes. Brushstrokes were generated by training the system to find similarities between the original paintings and the new portrait, using x-ray scans. Similar geometric patterns were matched to create this third dimension. This was done using a small area as a template, and repeated across the whole painting. More than 165,000 painting fragments were matched using this technique.

Emmanuel Flores: “The final step was printing the painting. Canon has a high-end, custom-made 3D printer that is especially designed to re-create existing paintings. So far, it had never been used to create a new painting. Until now. We printed thirteen layers of a special paint-based UV ink, to bring The Next Rembrandt to life.”

Were there any technical challenges? If so, how did you solve that? Bas Korsten: “The idea was kind of ingenious in its simplicity. Create a brand new Rembrandt from the data of all his 346 works of art. Sounds simple, right? The execution was anything but. It took a team of 20 data-analysts, developers, professors in AI and 3D printing experts 18 months. However, the biggest challenge was keeping the idea behind The Next Rembrandt alive. Even though there were so many forces working against it. Time, budget, technology, critics. But, most of all, the overwhelming amount of data we needed to go through. Perseverance and not taking ‘no’ for an answer, are the only reasons why this project succeeded.”

Were any ideas discarded? If so, why? Bas Korsten: “It was a journey of trial and error. We had plenty of ideas that were researched or tested, but discarded in the end. In fact, we started with the idea of creating a robot arm to paint a new portrait. But a robot arm only has nine degrees of freedom. The human hand has 26. Which meant it couldn’t paint in great detail yet. So we moved on to the next idea.”

What are the results so far for? How effective was it? Bas Korsten: “We were very successful in creating a big bang in the press, and on social, for both ING and tech partner Microsoft. With 1.8 billion media impressions and around 2,000 newspaper/magazine articles, including a handful of front pages. An earned media value of €12.5 million.”

What awards has the project won so far? - Cannes Lions: 2 Grand Prix, 3 Gold, 6 Silver, 4 Bronze, 1 Innovation Lion - D&AD: 1 Yellow Pencil, 2 Wood Pencils and 2 Graphite Pencils - One Show: 2 Gold, 3 Silver, 2 Bronze and 3 Merit - Eurobest: 1 Grand Prix, 5 Gold and 2 Silver - LIA: 1 Grand LIA, 5 Gold, 1 Silver and 1 Bronze - Epica: 1 Grand Prix, 3 Gold, 1 Silver and 1 Bronze - Clio: 1 Gold and 2 Bronze - Andy Awards: 1 Gold and 1 Bronze - FWA: Site of the Day - Awwwards: Site of the Day - WPPED Cream: 2 Grand Prix, 1 Winner - ADCN: 1 Grand Prix, 2 Gold, 1 Silver and 2 Bronze

Plus the project was top listed in these annual reports: 2nd Most-Awarded Digital Campaign — Gunn Report 2016 6th Most-Awarded Campaign Across All Media — Gunn Report 2016 Most-Awarded Digital Campaign of 2016 — The Directory Big Won Ranking 8th Most-Awarded Campaign of 2016 Across All Categories — The Directory Big Won Ranking

What are you most satisfied with? Bas Korsten: “With the change the project has brought to the agency. People genuinely feel that anything is possible now. Which is an invaluable mindset to have in a creative agency.”

Will there be a part two to this project? Bas Korsten: “‘We’re not 3D experts, Artificial Intelligence (AI) adepts or Rembrandt lovers per se. We’re just people who like to help clients solve business issues. We just founded a school in India for the victims of child prostitution: the School for Justice. Whether it’s a painting or a school, we’ll rise to the challenge — whatever the brief requires.”

This Dutch Digital Design case study was written by editor Nicole Pickett-Groen.

Credits

Agency: J. Walter Thompson Amsterdam Film production: New AMS Film Company Web development: Superhero Cheesecake

Executive Creative Director: Bas Korsten Concept: Bas Korsten, Robert Nelk, Mark Peeters Creative copy: Bas Korsten, Kasia Haupt Canning Creative art: Guney Soykan Innovation Director: Emmanuel Flores Design: Vinesh Gayadin Digital Director: Jesse Houweling PR Director: Jessica Hartley Editor: Tim Arnold Photography: Robert Harrison Developers: Morris Franken, Ben Haanstra