Machine Learning Art
How do you make a 3D scans of humans
state-of-the-art method 2D image to 3D human
There are several applications for 3D scans of humans wearing clothes, which is increasing demand. For example, accurate 3D people models would enhance immersive AR and VR applications, gaming, telepresence, virtual try-on, free-viewpoint photorealistic visualization, and creative picture editing.
Automatic scanning using multi-camera setups, manual production by an artist, or a combination of both is the traditional method for obtaining human models. Artists are frequently used to ‘clean up’ scanning errors. Because such tactics are difficult to scale, designers are looking for other automated alternatives that are less expensive and easier to implement.
- April 2022 — AI art tools update can be found ➡️ HERE ⬅️
State-of-the-art method, PHORHUM. End-to-end trainable deep neural network approach for photorealistic 3D human reconstruction from a monocular RGB picture.
3D human reconstruction
The Pixel-aligned algorithm calculates the whole 3D geometry and the unshaded surface color and scene lighting for the first time. Furthermore, the authors present patch-based rendering losses to enable trustworthy color reconstruction on visible areas of the human and comprehensive and convincing color estimates for non-visible regions after observing that 3D supervision alone is insufficient for high-quality color reconstruction. Furthermore, their technique mainly tackles previous work’s methodological and practical constraints in modeling geometry, albedo, and lighting effects in an end-to-end model where components may be successfully disentangled.
the article is a summary of the paper — Project Page (scroll down)
The attributes of single-picture 3D human reconstruction approaches are summarized( the below image) . Only the PHORHUM technique can predict albedo surface color and shading. Additionally, the process offers the most practicable training setup, does not require picture matting at test time, and delivers signed distances rather than binary occupancy, which is a more helpful representation.
The method’s overview
From an input picture I, the feature extractor network G generates pixel-aligned features zx for all locations in space x. Given a place and its feature, the implicit signed distance function network f computes the distance d to the closest surface. (the below image) Furthermore, f returns albedo colors for surface locations. Given the surface normal nx and light l, the shading network s predicts the shading for surface points. On the right, the authors display the shaded 3D geometry and the reconstruction of geometry and albedo colors.
The approach is the first to compute the 3D geometry, surface albedo, and shading from a single picture as part of an end-to-end model prediction. PHORHUM works well with a broad range of clothes and different body types and skin tones, and reconstructions capture the majority of the detail in the input image. However, while sparse 3D supervision works well for restricting geometry, rendering losses are required to reconstruct perceptually correct surface color, according to the authors.
Limitation of the dataset
When the attire or position of the person in the input image deviates too much from our dataset distribution, the limits of their technique become obvious. Their training gear does not adequately cover loose, large, or non-Western apparel. In addition, the rear of a person does not always match the front side in terms of semantics. These issues might be solved with a bigger, more geographically and culturally varied sample. I strongly advise you to read the complete essay below. The link also provides some fascinating future ideas for 3D scans of humans.
Title: Photorealistic Monocular 3D Reconstruction of Humans Wearing Clothing
the Authors : Thiemo Alldieck, Mihai Zanfir, Cristian Sminchisescu
Google ResearchProject page:
https://arxiv.org/pdf/2204.08906.pdf
Keywords: computer vision, Artificial Intelligence, datasets, Machine Learning, AI art, art, digital art, 3D scan, 3D human, 2D to 3D,
I invite you to explore the concept of “AI creativity” by reading and learning from the many articles found on 🔵 MLearning.ai 🟠
- Check out my instagram with new material every week
- If you enjoyed this, follow me on Medium for more
- Want to collaborate? Let’s connect on LinkedIn
- https://linktr.ee/evartology
Data Scientists must think like an artist when finding a solution when creating a piece of code. Artists enjoy working on interesting problems, even if there is no obvious answer.
All our writers (members) receive the opportunity to be promoted on our social media, which increases the popularity of articles published on MLearning.ai
