Generative AI - Best Practices for LLM Prompt Engineering
This article walks you through the process of creating robust prompts for LLMs. Learn how slight variations in your prompt can shape your model response.
What is Prompt Engineering?
Prompt Engineering refers to crafting effective prompts that can efficiently instruct LLMs that power Bard or ChatGPT to perform desired tasks. Those tasks can be applied to many different use cases.
It involves the precise wording, formatting, and structuring of a prompt to optimize the model’s output, enhancing relevance and accuracy.
The practice of prompt engineering is critical due to the sensitivity of LLMs to input. The slight variation in how a query is presented can lead to dramatically different responses, affecting the overall utility of the model. It’s more like art that follows some specific patterns.
Prompt Engineering is no exact science. My best practices are just a few of many. Experiment with it try different structures, wording, be creative. For some it is fun and interesting other might call it annoying. I just call it powerful.
Through this article, we use Google’s Generative AI Studio, which provides access to the PaLM 2 API. Check out my Getting Started with PaLM 2 article if this is new to you.
Why is Prompt Engineering So Important?
Effective, prompt engineering rather than fine-tuning is a good alternative for directing an LLM’s response.
A well-crafted prompt can dramatically enhance the model’s performance and efficiency by providing clear instructions, reducing ambiguity, and leading the model’s understanding in the desired direction.
Different Prompting Types
- Input Output Prompting
- Chain of Thought Prompting
- Self Consistency with CoT
- Tree of Thoughts
For this article, we cover the simplest type Input Output Prompting. Let me know in the comments if you’re interested in the other types. Tree of Thoughts seems very promising.
Generally, I would recommend the following approaches from top to down until you’re satisfied.
- Prompting
- Few shot Prompting
- Prompting + Retrieval (if applicable)
- Chain of Thought
- Self Consistency with Chain of Though
- Tree of Thoughts
- Fine-tuning
Best Practices
Enhancing Model Safety
Prompt engineering plays an important role in AI safety. Through well-structured prompts, we can implement a level of control over the model’s behavior, reducing the likelihood of generating biased, inappropriate, or harmful content.
Clarity in Prompts
Write your prompts as clearly as possible. This helps the model understand your intention and deliver better results. Avoid ambiguous language and instructions.
Few-shot prompting
With a few shot prompting, we provide examples of the same task we are trying to solve. This gives the LLM additional context and a clearer understanding of what’s expected.
A well-crafted prompt can outperform few-shot prompts. Try to iterate a couple of times over your prompt before introducing few-shot.
The Few-shot technique is particularly useful when the desired output is complex or creative.
Reducing Model Misinterpretations and Errors
As sophisticated as LLMs are, they can misinterpret prompts or produce errors if the instructions are not clearly articulated. Effective, prompt engineering mitigates this risk, leading to more accurate and reliable outputs.
The prompt on the left results in a vague answer, not exactly to the point and even not entirely correct. Hamburg and Boston have the lowest sales. The incorrect compelling answer is called hallucination. We cover that in a dedicated article. But for now, be aware of that.
Reproducibility
To get reproducible responses, we can reduce the temperature (model parameter). Or put guide rails using ordinary prompt engineering. This depends on the use case. A classification use case will work best if the temperature is very low, as it does not involve a lot of creativity.
We cover the model parameters like Temperatur, Top-K, and Top-P in a dedicated article.
Ensuring Format and Structured Response
Be specific, descriptive, and detailed about the desired response length, format, and style.
For example, we can instruct the model to return the answer as a JSON object. This can be particularly useful if you want to consume the model’s response in a programmatic way. This particular tweak also helps the model to return more precise and correct answers.
Be aware that even if instructed, the model might return non-valid JSON or even text. Reducing the temperature can help again. Give it a try and instruct the model to return the format you need. Run your prompt multiple times and check the error rate. If still too high, optimize your prompt.
We can combine this with an example to increase further the model's likelihood of returning the same JSON. This is a simpler form of few-shot learning as we only instruct based on a return example.
Iterative Prompting
It’s not always necessary or advantageous to encapsulate all your instructions in one single prompt. Break down complex tasks into a sequence of simpler prompts in an interactive conversation. This iterative prompting approach can often lead to higher-quality outputs, as it allows for mid-course corrections, refining the model’s understanding of the task at hand over several exchanges.
The question: Who was the father of Gwilym Lloyd George and where was he born?
Requires two steps to answer: first, identify the father of Gwilym Lloyd George, and second, determine where this person was born. In our example, the model fails to find the correct answer (Manchester, England) with the single-question approach.
The iterative prompting method might work as follows:
- First Iteration: The model is prompted with “Who is the father of Gwilym Lloyd George?” and responds with “David Lloyd George”.
- Second Iteration: The model is prompted with “Where was David Lloyd George born?” and responds with “Manchester, England”.
In each iteration, the model is prompted to recall a piece of knowledge relevant to the overall question, building the answer step by step.
Conclusion
Finally, a key practice in prompt engineering is constant experimentation and refinement. Language models are complex, and their responses can sometimes be unpredictable. Crafting the perfect prompt can be iterative, requiring several attempts. Regular testing and adjustment of your prompts will help you better understand the model’s behavior and improve its performance over time.
The effectiveness of prompt engineering cannot be understated. It’s a skill that can dramatically increase the quality of large language models' output. By following these best practices, you can design prompts that leverage the full power of these models and unlock their potential.
As the development and application of LLMs continue to expand, the skill of prompt engineering will only become more significant, forming an integral part of the future of artificial intelligence (If not replaced by yet another model).
We already seeing a lot of research papers around prompt engineering. Exiting to see what comes next.
Generative AI Series
I’ve written a series of articles and there’s more to come. Stay tuned by following me.
- Generative AI — The Evolution of Machine Learning Engineering
- Generative AI — Getting Started with PaLM 2
- Generative AI — Best Practices for LLM Prompt Engineering
- Generative AI — Document Retrieval and Question Answering with LLMs
- Generative AI — Mastering the Language Model Parameters for Better Outputs
- Generative AI — Understand and Mitigate Hallucinations in LLMs
- Generative AI — Learn the LangChain Basics by Building a Berlin Travel Guide
- Generative AI — Image Generation using Vertex AI Imagen
- Generative AI — Protect your LLM against Prompt Injection in Production
- Generative AI — AWS Bedrock Approach vs. Google & OpenAI
- Generative AI — How to Fine Tune LLMs
- more to come over the next weeks
Thanks for reading
Your feedback and questions are highly appreciated. You can find me on LinkedIn or connect with me via Twitter @HeyerSascha. Even better, subscribe to my YouTube channel ❤️.
