Machine Learning Art
Can AI generate 3D videos?
New AI video generator creates 3D content
Most AI generators use “Automated Text to Speech” to make videos that look like humans made them. Machine Learning is used in these software tools to turn text-based content into videos automatically. The new AI video generator benefits from 4D Art
- July 2022 — AI art tools update can be found ➡️ HERE ⬅️
Recent improvements to generative adversarial networks (GANs) have made it possible to make images that look like real photos. These methods have been improved to make it possible to make high-quality videos and 3D scenes that look the same from different points of view. But, even though generative models have important uses in visual effects, computer vision, and other fields, no one has yet shown that they can be used to make 3D videos.
Project Page (scroll down)
What is 3D AI video?
The first 4D GAN that can learn to make video data that is consistent across multiple views from single-view video data. A 3D-aware video generator that can make 3D content that can be animated with learned motion priors and allows you to change your point of view. A time-conditioned 4D generator that uses the brain’s new implicit scene representations and a time-aware video discriminator are two important parts of the framework.
The generator takes as input two latent code vectors for 3D identity and motion, and it outputs a 4D neural field that can be queried continuously at any spatio-temporal xyzt coordinate. The 4D fields that are made can be used to make realistic video frames from any camera angle. To train the 4D GAN, the authors use a discriminator that uses the time difference between two randomly chosen video frames from the generator (or from real videos) to score how realistic the motions are.
The model is trained with an adversarial loss, in which the discriminator tells the generator to make videos that look real from all of the sampled camera angles. The researchers test how well our method works on difficult, unstructured video datasets. They show that a trained 4D GAN can make plausible videos that can be viewed from different angles and whose visual and motion quality is on par with that of the best 2D video GANs.
Model architecture
The 3D content and motion codes, as well as the query camera view, are used by the generator (left) to make an RGB image. Based on the position of the camera, we build a ray image from which we take samples of x, y, and z positions along the rays to send to the fourier feature layer. The content code is changed into w, and the middle features of MLP layers are changed. Along with the time query t, the motion code is processed by the motion generator and added to the generator branch. The video discriminator takes two random frames from a video and the difference in time between them. It then predicts whether the frames are real or fake. The image discriminator looks at each frame and tells you whether it is real or fake.
🟠 4D GAN makes high-quality 3D content that can be controlled in terms of time and camera externals.
🔵 How do you make AI videos for free?
Keywords: computer vision, Artificial Intelligence, datasets, Machine Learning, AI art, art, digital art, datasculpting, datasculptor, 3d, 2D, 3D videos, AI art generator app, video generator, 4D GAN, 3D-Aware Video Generation
I invite you to explore the concept of “AI creativity” by reading and learningfrom the many articles found on 🔵 MLearning.ai 🟠
Check out my instagram with new material every week
- If you enjoyed this, follow me on Medium for more
- Want to collaborate? Let’s connect on LinkedIn
- https://linktr.ee/datasculptor
- 3D Machine Learning generated model on sketchfab
Data Scientists must think like an artist when finding a solution when creating a piece of code. Artists enjoy working on interesting problems, even if there is no obvious answer.
All our writers (members) receive the opportunity to be promoted on our social media, which increases the popularity of articles published on MLearning.ai
- Linkedin (16.1K+ ML-professionals)
- Twitter (5.1K+ followers)
- Instagram (2.2K + followers )
- Sketchfab * — individual vRooML!
- Youtube
- Apple Podcasts
- Substack
Project Page:
https://arxiv.org/pdf/2206.14797.pdf
Title: 3D-Aware Video Generation
The Authors: Sherwin Bahmani Jeong Joon Park Despoina Paschalidou Hao Tang Gordon Wetzstein Leonidas Guibas Luc Van Gool Radu Timofte
ETH Zürich Stanford University KU Leuven University of Würzburg