Machine Learning Art

2D to 3D scene reconstruction from a single image. DEMO

hallucinating scenes that are outside the camera’s field of view

Estimating 3D from a picture is a fundamental challenge in computer vision. While we, as humans, instinctively perceive a scene from a single image, thinking about geometry and semantics simultaneously, decades of study has shown that this is extremely difficult. As a result, several algorithms incorporate specialized depth sensors, such as Lidar or depth cameras to help with 3D estimates. However, these sensors are often more costly, less compact, and more intrusive than cameras used in smartphones, drones, vehicles, and other devices. Being able to predict a 3D environment from an image would therefore open the door to new applications.

• July 2022 — AI art tools update can be found ➡️ HERE ⬅️

What is 3D scene reconstruction?

It is possible to create three-dimensional models from several photos using 3D reconstruction. It’s the inverse process of converting 3D sceneries into 2D photos.

Understanding 3D geometry and semantics from images together provides the path for improved mixed reality, picture editing, and mobile robotics applications.

3D from a single image

3D Semantic Scene Completion (SSC) addresses scene comprehension by attempting to deduce its geometry and semantics simultaneously. While the job has lately gained popularity, present solutions still depend on depth data (i.e., occupancy grids, point clouds, depth maps, and so on).

What is a scene 3D?

There is more to a 3D scene than geometry. A scene would be pitch-black if it had no lights, hence there must be lights in a scene. A file called the scene file contains all of this information (the description of geometry as well as camera and light information) in rendering.

MonoScene

3D Semantic Scene Completion (SSC) framework, in which the dense geometry and semantics of a picture are inferred from a single monocular RGB image. The authors solve the hard problem of converting a 2D scene to a 3D scene while also figuring out what it means. The framework is based on 2D and 3D UNets that are connected by a new 2D-3D features projection that is inspired by optics. They also use a 3D context relation to make sure that the data is consistent in both space and meaning. In addition to making architectural contributions, the authors also show new global scenes and losses in local frustums. Experiments show that they do better than the literature on all metrics and data sources while hallucinating plausible scenes that are outside the camera’s field of view.

Project Page (scroll down)

MonoScene framework

The authors infer 3D SSC from a single RGB image by using 2D and 3D UNets connected by Features Line of Sight Projection (FLoSP) and a 3D Context Relation Prior (3D CRP) to improve spatio-semantic awareness. Scene-Class Affinity loss improves the global semantics, geometry, and Frustums in addition to cross-entropy. Proportion loss makes sure that classes are spread out in local frustums, which is a form of supervision that goes beyond occlusions.

🔵 The first SSC approach capable of handling both outdoor and interior scenes from a single RGB picture

🔵 A 2D Feature Mechanism Line-of-Sight Projection connects 2D and 3D networks.

🔵 A 3D Context Relation Prior layer that improves network context-awareness.

🟠 New SSC losses are being developed to maximize scene-class affinity and local frustum proportions.

3D Context Relation

SSC is addressed by MonoScene by employing consecutive 2D-3D UNets, which are connected by a new feature projection and which provide enhanced contextual awareness as well as additional losses in the process.

Photo editing or mobile robotics apps may be improved by better understanding 3D geometry and picture semantics. However, mistakes in scene interpretation may have disastrous consequences (e.g. autonomous driving), therefore such algorithms should constantly be backed up by other methods.

Keywords: computer vision, Artificial Intelligence, datasets, Machine Learning, AI art, art, digital art, datasculpting, datasculptor, 3d, 2D, 3D from a single image, 3D scene reconstruction,

I invite you to explore the concept of “AI creativity” by reading and learningfrom the many articles found on 🔵 MLearning.ai 🟠

• Check out my instagram with new material every week
• If you enjoyed this, follow me on Medium for more
• Want to collaborate? Let’s connect on LinkedIn
• https://linktr.ee/datasculptor
• 3D Machine Learning generated model on sketchfab

Data Scientists must think like an artist when finding a solution when creating a piece of code. Artists enjoy working on interesting problems, even if there is no obvious answer.

All our writers (members) receive the opportunity to be promoted on our social media, which increases the popularity of articles published on MLearning.ai

1. Linkedin (15.8K+ ML-professionals)
2. Twitter (5.1K+ followers)
3. Instagram (2.2K + followers )
4. Sketchfab * — individual vRooML!
5. Facebook
6. Youtube
7. Apple Podcasts
8. Substack

🔵 Submission Suggestions

Project Page:

https://arxiv.org/pdf/2112.00726.pdf

DEMO:

https://huggingface.co/spaces/CVPR/MonoScene

@inproceedings{cao2022monoscene,
    title={MonoScene: Monocular 3D Semantic Scene Completion}, 
    author={Anh-Quan Cao and Raoul de Charette},
    booktitle={CVPR},
    year={2022}
}