Summary

The article discusses the significant impact of instruction placement within prompts on the performance of large language models (LLMs), with a focus on the Gemini 1.5 model.

Abstract

The article titled "LLM Instruction Placement in Prompts — It Matters a Lot! ⭐" explores the nuanced effect of where instructions are placed within prompts given to LLMs, particularly Gemini 1.5. It presents an analysis demonstrating that the placement of instructions can greatly influence the quality of responses from LLMs. The study uses a "Needle in the Haystack" evaluation framework to assess the impact of instruction placement on the LLM's ability to extract specific information from a large body of text. The findings reveal that for prompts exceeding 2000 characters, placing instructions at the end rather than the beginning leads to substantially better performance. This insight is crucial for prompt engineering, which has become an essential skill in maximizing the effectiveness of LLMs in various tasks, including Retrieval Augmented Generation (RAG). The article also references a Stanford University paper that shows LLMs tend to forget information placed in the middle of long prompts, exhibiting a U-shaped performance pattern. The author, Lars Christian Wiik, encourages further investigation by providing a reproduce dataset and invites readers to engage with similar content by following his work.

Opinions

The author believes that the precise placement of instructions within prompts is a critical factor in the performance of LLMs, especially for tasks involving long contexts.
The study's results are described as "shocking," emphasizing the author's view on the importance of instruction placement.
The author suggests that the skill of prompt engineering will become increasingly necessary as the use of LLMs grows, indicating a belief in the ongoing relevance and development of this field.
The author values the community's input, as the idea for evaluating instruction placement came from a follower's suggestion, and encourages further engagement and discussion in the comment section.
By providing a reproduce dataset, the author shows a commitment to transparency and reproducibility in research, inviting others to validate and build upon the findings.
The author expresses a personal fascination with LLM performance analysis and positions their content as valuable for both professionals and enthusiasts in the AI field.

LLM Instruction Placement in Prompts — It Matters a Lot! ⭐

Gemini 1.5 — Evaluation of how instruction placement in large prompts affects quality

Understanding the intricate nuances of LLMs is critical when maximizing quality responses.

One such nuance is the position of instructions within the prompt.

The placement of instructions within prompts can significantly influence the performance of large language models (LLMs).

In this article, I will present my analysis measuring how instruction placement within prompts impacts the response quality.

The results were shocking! — It matters a lot.

Prompt Engineering 📝

After the release of ChatGPT in late 2022, a new term known as “Prompt Engineering” has emerged.

Prompt Engineering involves crafting and optimizing instructions for language models, which is crucial for enhancing accuracy, increasing efficiency, customizing outputs, and facilitating ethical use.

Google Trend graph of “Prompt Engineering” worldwide in the last 5 years

Prompt engineering can be challenging due to the precise nature of language required to guide AI responses effectively.

The need for comprehensive guidance and training in this specialized skill will likely increase with time. Unless we eventually enter into a world where language models start evolving their own prompts.

Long Context Prompts 💡

In certain tasks such as Retrieval Augmented Generation (RAG), the prompt will contain external context for the LLM to read and understand before applying your instructions.

There is an interesting paper “Lost in the Middle: How Language Models Use Long Contexts” (Nov 2023) published by researchers from Stanford University.

The study investigates how well language models utilize long contexts for tasks such as question answering and information retrieval.

Their findings were particularly interesting. They observed a U-shaped performance pattern — meaning the LLM remembers the beginning and the end of the prompt best while forgetting what was in the middle!

Here is a diagram from the paper:

Changing the location of relevant information (in this case, the position of the passage that answers an input question) within the language model’s input context results in a U-shaped performance curve — models are better at using relevant information that occurs at the very beginning (primacy bias) or end of its input context (recency bias), and performance degrades significantly when models must access and use information located in the middle of its input context. [source]

This study clearly shows that some language models have difficulties managing and applying information across longer text spans.

Quote from the paper:

“We observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.” [source]

Ok, so we should place the most relevant information at the start or the end of the prompt for the model to most accurately remember it — this should include the prompt instructions as well.

But does it matter whether we put our instruction FIRST or LAST?

Let’s find out!

Evaluation Strategy 🧠

I developed a specialized evaluation framework to measure the impact of instructional placement within prompts.

The study utilizes the “Needle in the Haystack” test, a well-established method for evaluating the quality of LLMs replies.

In this test, a specific piece of information (the “needle”) is placed within an extensive chunk of unrelated text (the “haystack”). The LLM is then tasked to respond to a query that requires extracting the needle from the haystack.

For more information regarding how this dataset was created, see my previous article “OpenAI’s GPT-4o vs. Gemini 1.5 ⭐ Context Memory Evaluation”.

I conducted a comprehensive assessment by compiling multiple datasets with varying prompt lengths including 1,000, 1,500, 2,000, 3,000, 4,000, 8,000, 12,000, and 16,000 characters.

100 datasets were created for each unique context length — where the needle was randomly placed between the 20th and 80th percentile of the haystack contexts.

I designed two distinct prompt formats for the experiment. One where the instructions were placed at the beginning of the prompt, and another where the prompt was placed at the end.

Below, you can see the two prompts I used for this evaluation:

The two prompts used for the needle-in-the-haystack evaluation

Results 📊

Two configurations of Gemini 1.5 were included, one where the instruction is placed at the start of the prompt (instr_first), and another where the instruction is placed at the end of the prompt (instr_last).

Both configurations were tasked to extract the hidden information within the large context. The LLMs were evaluated using 100 distinct needle-in-the-haystack datasets for each unique context length. The average accuracy per context length was then calculated.

The graph below shows the accuracy of different context lengths in this needle-in-the-haystack evaluation conducted using Gemini-1.5.

Note: Gemini 1.5 Pro was released as generally available at May 24, 2024, which is the version I used for this evaluation.

Diagram showcasing performance differences when placing the instructions and beginning vs. end of prompt for Gemini 1.5.

As we can derive from the graph above, there is a clear distinction between the two configurations.

The immediate observation is shocking. Placing the instruction at the end of the prompt results in significantly better performance for large prompts compared to placing the instructions at the beginning.

This seems true for prompts with at least 2000 characters and more.

An additional observation is that for shorter prompts, specifically those with 1000 characters, positioning the instruction at the beginning of the prompt appears to enhance accuracy in this test.

Reproduce Dataset

I created a reproduce dataset in case you wish to investigate it further. I stored the full prompt and the result of 20 examples from the 4k character dataset.

To reproduce, copy the full prompt into whatever tool you use to run Gemini 1.5.

See incorrect results for Instruction First: Link.
See incorrect results for Instruction Last: Link.

Here is an example of reproducing the results using Vertex AI. Make sure to put the temperature at 0 for near-deterministic replies.

Note: The idea of evaluating instruction placement came from a follower suggeting this could be an interesting investigation. Please do not hesitate to suggest ideas in the comment section!

Conclusion ⭐

The positioning of instructions within prompts plays a crucial role in optimizing the performance of large language models like Gemini 1.5.

This study highlights a clear distinction between placing the instruction at the beginning vs. the end of the prompt and reveals significant improvements when instructions are placed at the end for long context tasks.

This insight not only highlights the importance of Prompt Engineering as an essential skill in the era of advanced AI but also brings value for developers and researchers when fine-tuning language models.

Thanks for reading!

Follow to receive similar content in the future!

And do not hesitate to reach out if you have any questions!

Through my articles, I share cutting-edge insights into LLMs and AI, offer practical tips and tricks, and provide in-depth analyses based on my real-world experience. Additionally, I do custom LLM performance analyses, a topic I find extremely fascinating and important in this day and age.

My content is for anyone interested in AI and LLMs — Whether you’re a professional or an enthusiast!

Follow me if this sounds interesting!

Connect with me:

🌐 LinkedIn: Link
𝕏: https://x.com/LarsChrWiik (I just now created an account and will start posting here as well)
Medium: https://medium.com/@lars.chr.wiik/about