Machine Learning Art
3D Scenes From Image Data
a novel approach to model via 4D-tensor
TensoRF with CP decomposition tackles the problem of rendering realistic images in a completely novel way. Rather than depending on MLPs, they construct a scene as a vector and matrix tensor that is decomposed into multiple low-rank tensors. It takes less space to store the data and it is easier for these algorithms to render more accurate scenes with more detail.
A novel approach that achieves photo-realistic rendering, fast reconstruction, and compact modeling. Project Page (scroll down)
🔵 Super Fast Convergence
Given a set of multi-view input images with known camera poses, the tensorial radiance field is optimized per scene via gradient descent, minimizing an L2 rendering loss, using only the ground truth pixel colors as supervision.
🔵 Super Compact Memory Footprint
In contrast to previous works that directly reconstruct voxels, the tensor factorization reduces space complexity from O(n3) to O(n) (with CP) or O(n2) (with VM), significantly lowering memory footprint.
🔵 Super Vivid Details
The approach can also achieve high-quality radiance field reconstruction for 360o objects and forward-facing scenes. All results without compression are available at OneDrive.
Abstract. We present TensoRF, a novel approach to model and reconstruct radiance fields. Unlike NeRF that purely uses MLPs, we model the radiance field of a scene as a 4D tensor, which represents a 3D voxel grid with per-voxel multi-channel features. Our central idea is to factorize the 4D scene tensor into multiple compact low-rank tensor components. We demonstrate that applying traditional CP decomposition — that factorizes tensors into rank-one components with compact vectors — in our framework leads to improvements over vanilla NeRF. To further boost performance, we introduce a novel vector-matrix (VM) decomposition that relaxes the low-rank constraints for two modes of a tensor and factorizes tensors into compact vector and matrix factors. Beyond superior rendering quality, our models with CP and VM decompositions lead to a significantly lower memory footprint in comparison to previous and concurrent works that directly optimize per-voxel features. Experimentally, we demonstrate that TensoRF with CP decomposition achieves fast reconstruction (< 30 min) with better rendering quality and even a smaller model size (< 4 MB) compared to NeRF. Moreover, TensoRF with VM decomposition further boosts rendering quality and outperforms previous state-of-the-art methods, while reducing the reconstruction time (< 10 min) and retaining a compact model size (< 75 MB).
@misc{TensoRF, title={TensoRF: Tensorial Radiance Fields}, author={Anpei Chen and Zexiang Xu and Andreas Geiger and and Jingyi Yu and Hao Su}, year={2022}, eprint={2203.09517}, archivePrefix={arXiv}, primaryClass={cs.CV} }
project page:
https://apchenstu.github.io/TensoRF/
I invite you to explore the concept of “AI creativity” by reading and learning from the many articles found on MLearning.ai
- Check out my instagram with new material every week
- If you enjoyed this, follow me on Medium for more
- Want to collaborate? Let’s connect on LinkedIn
- https://linktr.ee/datasculptor
- 3D Machine Learning generated model on sketchfab
Data Scientists must think like an artist when finding a solution when creating a piece of code. Artists enjoy working on interesting problems, even if there is no obvious answer.
All our writers (members) receive the opportunity to be promoted on our social media, which increases the popularity of articles published on MLearning.ai
