Machine Learning Art
Is AI better than animals at seeing in the dark?
The limits of robot vision in extreme darkness
Studying the behavior of animals at night
In the darkest of nights, it is difficult for us to see. The moon provides about 3/4 of the light in a very dark situation, and long exposure periods have been used in these conditions to provide sufficient light from our environment.
- April 2022 — AI art tools update can be found ➡️ HERE ⬅️
Some creatures, such as bees, can navigate well by the light of the stars (0.001 lux) on the darkest moonless nights, yet the finest CMOS cameras require at least 3/4 moonlight (> 0.1 lux) to picture moving things at night. Due to the minuscule quantities of light existing in the surroundings, seeing in the darkest situations (moonless, clear nights) is exceedingly difficult. However, long exposure durations (e.g., 20 sec or longer) can be used in such gloomy circumstances to capture enough light from the environment.
The authors show photorealistic videos under starlight for the first time in this article. They create a Generative Adversarial Network physics-based noise model that can better describe camera noise at low light levels. They used a combination of generated noisy video clips and actual noisy still photos to train a film denoiser using this noise model. The authors use no active lighting to record a 5–10 fps video dataset with substantial motion at 0.6–0.7 millilux. As a result, they produce increased video quality at lower light levels when compared to alternative approaches, exhibiting realistic video denoising in starlight for the first time.
The method
🔵 They train the noise generator with a discriminator to tell the difference between actual and synthetic noise. The authors employ a small dataset of non-moving picture pairings with long exposure/low gain and shorter exposure/high gain during the training phase. The noise generator will synthesize realistic noise after some training.
🔵 After that, they train the denoiser with a mix of artificial clean/noisy video clips generated with our noise generator and still clips from the camera. This eliminates the requirement for experiment motion-aligned video pairs when training a video denoiser.
Denoising of images and videos
Several approaches for image/video denoising have been suggested and investigated throughout the years. However, many traditional denoising approaches, such as sparsity, smoothness, and Gaussian mixture models, rely on specific picture priors. Others use a non-local method to denoise comparable regions throughout a picture collectively. Finally, deep learning-based techniques to image denoising have lately been used, in which a picture prior is learned from the data rather than assumed directly. These approaches have demonstrated considerable increases in picture quality over traditional methods. However, they frequently make naive assumptions about noise statistics, such as i.i.d Gaussian.
Conclusion
For the first time, the authors have shown photorealistic video denoising at sub-milli-lux levels of light. They were able to do so thanks to a combination of suitable camera hardware, a physics-inspired noise generator that generated realistic noisy video clips, and a trained video denoiser using authentic still images and synthetic noisy video clips. They’ve demonstrated the effectiveness of deep-learning-based denoising in extremely low-light situations. Future scientific breakthroughs at extremely low light levels are expected due to this effort. In addition, it can assist in pushing the boundaries of robot vision in highly dim environments.
Nighttime monitoring or usage combined with weapons systems are two examples of possible misuses of this work. This method has drawbacks. First, the noise generator can only generate noise that resembles a single gain level (in this case, the highest gain). The noise method might be improved to cope with varied camera gains/ISOs. Due to the dominance of NIR over RGB at night, denoised night films feature muted hues.
By improving embedded color cues or synthesizing realistic-looking colors, work in style transfer and recoloring might improve the visual appearance of denoised video footage. Finally, class-aware denoising and combined denoising/segmentation may improve the denoiser’s performance in the future.
Title: Dancing under the stars: video denoising in starlight
The Authors: Kristina Monakhova, Stephan R. Richter, Laura Waller, Vladlen Koltun
Project Page:
https://arxiv.org/pdf/2204.04210.pdf
Keywords: Computer Vision, Denoising videos, Artificial Intelligence, Graphics, Machine Learning, AI art, art, digital art, robot vision, Generative Adversarial Network, GAN,
I invite you to explore the concept of “AI creativity” by reading and learning from the many articles found on 🔵 MLearning.ai 🟠
- Check out my instagram with new material every week
- If you enjoyed this, follow me on Medium for more
- Want to collaborate? Let’s connect on LinkedIn
- https://linktr.ee/evartology
Data Scientists must think like an artist when finding a solution when creating a piece of code. Artists enjoy working on interesting problems, even if there is no obvious answer.
All our writers (members) receive the opportunity to be promoted on our social media, which increases the popularity of articles published on MLearning.ai
