Machine Learning Art
Your favorite sound as a pleasant picture
Cross Modal Image Synthesis
Can you envision the image by the sound? The answer is Yes! Learning Visual Styles from Audio-Visual Associations. This is an experiment in computer vision to test the extent to which sight and sound are intertwined.
- May 2022 — AI art tools update can be found ➡️ HERE ⬅️
Sound can be used as a creative, complementary tool to visualize abstract ideas, feelings, and memories. By assigning various sounds to different shapes, colors, and tones and then guiding listeners through this sonic space, it becomes possible for them to see images and envisage moving images — or “aural cinema.”
Project page (scroll down)
The noises we hear convey the visual textures inside a picture, from the pattern of rain to the crunch of snow. The authors provide a method for learning visual styles using matched audio-visual input. The model changes a scene’s texture to fit a sound, which we call audio-driven picture stylization. The model learns to edit input pictures to be more likely to co-occur with other input sounds after modification, given a dataset of paired, unlabeled audio-visual data. Their sound-based model surpasses label-based techniques in both quantitative and qualitative evaluations. They demonstrate that changing audio, for as, by adjusting its level, leads to predictable visual changes by using audio as an understandable embedding space for picture alteration.
Audio to Image
The authors propose that unlabeled audio-visual data be used to learn stylization to solve these challenges. Many scene characteristics, such as weather, provide extremely unique images and sounds. In order to train a model to predict visual information from audio, it must first recognize scene structures and then learn which visual textures are connected with each sound. They present a model for doing audio-driven picture stylization based on this concept. Given an input picture and a target sound, their model adjusts the image’s textures to better match the sound while maintaining the image’s structural content. The model learns a range of visual styles, each of which is defined by a sound — for example, bird chirps, blue sky, and rain.
Following training, the model can change pictures to meet a range of visual styles, each of which is defined by sound. They show that the model’s capacity to stylize pictures is successful through quantitative assessments and human perception experiments. The authors also present qualitative data demonstrating:
🔵 How simply mixing or adjusting the level of the audio causes matching changes in visual style
🔵 Unlabeled audio provides supervision for learning visual styles.
🔵 The model learns to conduct audio-driven stylization using in-the-wild audio-visual data.
Model architecture
The scene structure is preserved using the multi-scale patch-wise structure discriminator, while the scene texture is converted using the audio-visual texture discriminator. (Above) This is an example of a bright woodland that has been transformed into a wintry equivalent. In compared to other random patches, the created snow patch should mirror its matching input dirt patch.
Cross-Modal Image Synthesis
Audio-Driven Picture Stylization is a unique challenge that seeks to learn visual styles from linked audio-visual data. The authors offer a contrastive-based audio-visual GAN model and an unlabeled egocentric hiking dataset called Into the Wild to investigate this topic. According to the results, the novel model beats the label and image conditioned baselines in both quantitative and qualitative evaluations. They discovered that altering the audio level and mix causes predictable graphic changes.
I believe that this groundbreaking research will throw fresh insight into cross-modal picture synthesis.
Title: Learning Visual Styles from Audio-Visual Associations
Authors: Tingle Li1, , Yichen Liu , Andrew Owens , and Hang Zhao Tsinghua University, University of Michigan Shanghai Qi Zhi Institute
Project Page:
https://tinglok.netlify.app/files/avstyle/resources/preprint.pdf
Keywords: computer vision, Artificial Intelligence, Machine Learning, AI art, art, digital art, Image Synthesis, Audio-Driven Picture Stylization, GAN, Audio to Image
I invite you to explore the concept of “AI creativity” by reading and learning from the many articles found on 🔵 MLearning.ai 🟠
- Check out my instagram with new material every week
- If you enjoyed this, follow me on Medium for more
- Want to collaborate? Let’s connect on LinkedIn
- https://linktr.ee/evartology
Data Scientists must think like an artist when finding a solution when creating a piece of code. Artists enjoy working on interesting problems, even if there is no obvious answer.
All our writers (members) receive the opportunity to be promoted on our social media, which increases the popularity of articles published on MLearning.ai
