Machine Learning Art

Turn TEXT to 3D AI art

Open-World Scene: The Future of 3D DIGITAL ART [update Aug 2023]

3D vision-language models

I see very early but impressive demos of how language interfaces can be used to control AI art generators.

• July 2022 — AI art tools update can be found ➡️ HERE ⬅️

These tools will significantly impact artists’ skills and creativity in the next ten years. It will be a joy to behold and a lot of fun to work with, as it will allow us to create 3D AI art with the text.

Artificial Intelligence in Creative Media

Most existing tools would only be useful for experts, who could use them to generate great results quickly. I see a need for the least amount of effort when working with these tools; this is where NLP interfaces should play an important role.

Since AI ART is becoming increasingly expensive, it’s worth knowing free and available substitutes.

Most advanced text-to-image models are not publicly available. (Imagen, Parti). But many open-source models are accessible and affordable to use. HERE

Three dimensional digital art

Present AI art generators have been limited to the 2D world. However, there are attempts to apply textures and shapes to an open 3D world. Below is an exciting example of using SOTA models to understand 3D space using image language models

Open-world 3D Scene

This is an example of an open-world 3D scene understanding task, which is a family of 3D vision-language tasks that includes open-set classification with extra generalization requirements to novel vocabulary (e.g. synonyms of seen vocabulary), visual properties (e.g. lighting, textures), and domains (e.g. sim v.s. real). The main problem with these kinds of tasks is that there isn’t enough data. Existing 3D datasets don’t have as much variety or size as their 2D counterparts on the internet, so training robots doesn’t prepare them for the open 3D world.

Project Page (scroll down)

Semantic Abstraction (SemAbs) is a framework for understanding open-world 3D scenes using 2D VLMs and visual-semantic reasoning. While open-world visual-semantic reasoning needs to be exposed to internet-scale datasets, 3D spatial and geometric reasoning can be done with a small set of synthetic data. It could generalize better if learned in a way that isn’t tied to any particular meaning. For example, the 3D localization model doesn’t need to understand the idea of “behind the Harry Potter book.” It only needs to learn the concept of “behind that object.”

Overview of Semantic Abstraction. Using the SemAbs module, the framework can be used to understand (a) and (b) open-world 3D scenes ©. It has a semantic-aware wrapper (green background) that abstracts the input image and semantic label into a relevancy map and a semantic-abstracted 3D module (grey background) that completes the projected relevancy map into a 3D occupancy. This abstraction lets our method work for long-tail semantic labels like “CoRL ticket on top of the fireplace” that weren’t seen (in bold) during 3D training.

towards open-world AI art robotics

The above example shows the direction in which not only robotics is moving but also the art generators used to create 3D art. Video could be the closest application.

Keywords: computer vision, Artificial Intelligence, Machine Learning, AI art, art, wombo dream, digital art, Dalle 2, Imagen, wombo ai, Parti, text-to-image, diffusion models, generative art, wombo art, photographic quality, img by AI system, AI art generator, text to art generator, free ai art generator, 3D ai art

I invite you to explore the concept of “AI creativity” by reading and learningfrom the many articles found on 🔵 MLearning.ai 🟠

Check out my instagram with new material every week

• If you enjoyed this, follow me on Medium for more
• Want to collaborate? Let’s connect on LinkedIn
• https://linktr.ee/datasculptor
• 3D Machine Learning generated model on sketchfab

Data Scientists must think like an artist when finding a solution when creating a piece of code. Artists enjoy working on interesting problems, even if there is no obvious answer.

All our writers (members) receive the opportunity to be promoted on our social media, which increases the popularity of articles published on MLearning.ai

1. Linkedin (19.1K+ ML-professionals)
2. Twitter (5.1K+ followers)
3. Instagram (2.2K + followers )
4. Sketchfab * — individual vRooML!
5. Facebook
6. Youtube
7. Apple Podcasts
8. Substack

🔵 Submission Suggestions

Project Page:

https://arxiv.org/pdf/2207.11514.pdf

@inproceedings{ha2021semabs,  title={Semantic Abstraction: Open-World 3{D} Scene Understanding from 2{D} Vision-Language Models},  author={Ha, Huy and Song, Shuran},  journal = {CoRR},  volume = {abs/2105.03655},  year = {2021},  url = {https://arxiv.org/abs/2105.03655},  eprinttype = {arXiv},  eprint = {2105.03655}, }