Machine Learning Art
Real-time Dynamic 3D Reconstruction
OcclusionFusion: Occlusion-aware Motion Estimation for Real-time Dynamic 3D Reconstruction
Can’t you guess what this is about already? It’s about computer vision, and it’s about tracking! This article explains Occlusion Fusion- a technique used to accurately track the motion of an object in the scene.
- April 2022 — AI art tools update can be found ➡️ HERE ⬅️
This technique can handle long motion sequences with the minimal error by combining the motion of both visible and occluded regions. The idea behind it is to model the output of a neural network that has been trained on images like these:
RGBD-based real-time dynamic 3D reconstruction suffers from inaccurate inter-frame motion estimation as errors may accumulate with online tracking. This problem is even more severe for single-view-based systems due to strong occlusions. Based on these observations, we propose OcclusionFusion, a novel method to calculate occlusion-aware 3D motion to guide the reconstruction. In our technique, the motion of visible regions is first estimated and combined with temporal information to infer the motion of the occluded regions through an LSTM-involved graph neural network. Furthermore, our method computes the confidence of the estimated motion by modeling the network output with a probabilistic model, which alleviates untrustworthy motions and enables robust tracking. Experimental results on public datasets and our own recorded data show that our technique outperforms existing single-view-based real-time methods by a large margin. With the reduction of the motion errors, the proposed technique can handle long and challenging motion sequences.
Limitations
Although the method improves the quality of motion tracking by introducing a graph-based full motion prediction network, there are still some failure cases. The method cannot handle topology change, which is an open problem for node-graph-based reconstruction systems.
Conclusion
We propose OcclusionFusion, a single-view RGB-D based real-time dynamic 3D reconstruction method that outperforms the current online techniques by a large margin. We use a complete 3D motion field to guide the object reconstruction and tracking, where the motion of the occluded regions is estimated online by a pre-trained lightweight graph neural network. The graph neural network combines the motion of visible regions and temporal information by involving an LSTM structure to accurately and efficiently predict the complete object motion. Moreover, the graph neural network predicts confidence together with the motion by modeling the network output as a Gaussian distribution, which effectively enhances the robustness of the reconstruction system. Experimental results show that long and challenging sequences can be tracked well in real time using our technique with a single view input.
@inproceedings{lin2022occlusionfusion,
title={OcclusionFusion: Occlusion-aware Motion Estimation for Real-time Dynamic 3D Reconstruction},
author={Wenbin Lin, Chengwei Zheng, Jun-Hai Yong, Feng Xu},
journal={Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2022}
}
project page:
https://wenbin-lin.github.io/OcclusionFusion
I invite you to explore the concept of “AI creativity” by reading and learning from the many articles found on MLearning.ai
- Check out my instagram with new material every week
- If you enjoyed this, follow me on Medium for more
- Want to collaborate? Let’s connect on LinkedIn
- https://linktr.ee/datasculptor
- 3D Machine Learning generated model on sketchfab
Data Scientists must think like an artist when finding a solution when creating a piece of code. Artists enjoy working on interesting problems, even if there is no obvious answer.
All our writers (members) receive the opportunity to be promoted on our social media, which increases the popularity of articles published on MLearning.ai
