Prompt Engineering via Prompt Patterns — Visualization Generator Pattern

The article is part of series: Prompt Engineering via Prompt Patterns

You can switch to video version of this article

Pictures are worth a thousand words. You must have heard this phrase thousands of times. But can a thousand words take the shape of a picture? Yes there are a bunch of options with which this is absolutely possible. You want to see Mona Lisa wearing a cowboy hat and riding a bike sitting behind a biker dog wearing a leather jacket? Boom and you get the picture created for you by AI tools. I haven’t tried this yet but you are welcome to give it a shot. How and where exactly would be your next question I assume. Lets explore the available options.

Lets start with simpler visualization tools before moving to advanced ones. There are two most popular text to visualization softwares available that take instructions as text and come up with visualizations. One is Graphviz Dot. Reserved for more technical users (and non technical ones are free to skip to next chapter in article), this is a tool to visualize graphs and networks, while accepting the instructions in text. Most entities and their relationships in real world can be visualized as a graph, tree or network.

Examples being family trees, mind maps, data structures, computer networks, databases, Facebook friends network etc. Now by text based input, I don’t mean natural language instructions here, this is a specific syntax reserved for the tool, but nonetheless the important part here is it is a text based input unlike other similar tools that take binary files as input to generate similar visualizations. Why are we even discussing this then? Just hang on and I’ll show you how you should be able to use such tool with natural language instructions.

The second more interesting option, and an offering from the same organization that powers ChatGPT, is Dall E. Truly deserving its own dedicated article, I’ll just briefly tell that it is a text to image model, where you describe your imagination of an image like Mona Lisa riding a bike one we discussed earlier, and it is able to generate a photorealistic image, painting or emoji. It can manipulate and rearrange objects in images so you can create variations of existing images.

Since pictures are still worth a thousand words, OpenAI has provided a public domain example of a prompt to generate an illustration like the one below created with prompt.

“An illustration of an avocado sitting in a therapist’s chair, saying ‘I just feel so empty inside’ with a pit-sized hole in its center. The therapist, a spoon, scribbles notes”

So you get the idea 😊. Note this option is not available with free version as of now and is available with plus version only.

So where does the visualization generator pattern fit in all of this? Well the intent of pattern is to create a pathway for the tool to produce imagery, by creating input for visualization tools that use text as input (like Graphviz Dot or Dall E). Since LLMs like ChatGPT are not able to create visualization by themselves, this pattern can be used to instruct the model to create the textual instructions that can be used to generate visualizations using the tools we discussed.

The pattern uses key contextual statements like ‘Generate an X that I can provide to tool Y to visualize it’. Now LLM cannot generate an image, so we clarify that LLM is not expected to generate an image, but should generate a prompt or textual input that we can provide to tools like Dall E or Graphviz dot to create the image or visualization.

An example of this can be an example prompt like

“Whenever I ask you to visualize something, please create either a Graphviz Dot file or DALL-E prompt that I can use to create the visualization. Choose the appropriate tools based on what needs to be visualized.”

Awesome right? We went a step further in letting ChatGPT or LLM figure out the tool as well for the topic under discussion. If we want to generate a bar chart, UML diagram, family tree or social media friends network, it would most probably pick Graphviz Dot since that is a more suitable tool for the type i.e. a rigid exactly defined structure.

However, if a more appropriate visualization can be a realistic image, coz there is no well defined structure, it would choose Dall E itself based on its own understanding of both the tools. These are not the only two options and you can add more tools you know about to the list.

This article described both the tools and how to use the natural language abilities of LLMs like ChatGPT to create images using the visualization generation pattern.If you found this informative, please clap/share the article. You can also consider subscribing to our YouTube channel as well. Thank you!!!.

Next article: Prompt Engineering via Prompt Pattern — Game Play Pattern