Prompt Engineering via Prompt Patterns — Cognitive Verifier Pattern

Summary

The cognitive verifier pattern is an advanced prompt engineering technique that enhances the reasoning of large language models by decomposing user questions into multiple, more specific questions, the answers to which are then synthesized into a comprehensive response to the original query.

Abstract

The cognitive verifier pattern is a method in prompt engineering that improves the ability of large language models to reason and provide accurate answers. It does so by instructing the model to generate additional questions that help clarify and refine the original question. This approach acknowledges that complex questions often have hidden dimensions and assumptions that need to be addressed. By breaking down a high-level question into smaller, more precise questions, the model can leverage its knowledge across specific topics. Once the additional questions are answered, the model integrates this information to provide a more concrete and nuanced response to the initial query. This pattern is particularly useful for intricate tasks where the original question may be too broad or ambiguous. It allows the model to identify missing information or unclear aspects of the prompt. The technique can be customized to guide the model in asking questions that align with the user's objectives, such as considering tools needed for a DIY project. The effectiveness of this pattern is exemplified in scenarios like deciding between buying an apartment or a house, where additional context such as location, taxes, and maintenance costs is crucial for a meaningful answer.

Opinions

The article posits that large language models can significantly improve their reasoning capabilities when questions are broken down into sub-questions.
It is suggested that a single, broad question is often too complex for a model to answer accurately without further clarification.
The author believes that the cognitive verifier pattern aids in uncovering the implicit dimensions and assumptions within a question.
The technique is described as beneficial for both the model and the user, as it leads to answers with greater depth and reasoning.
The author encourages readers to engage with the content by subscribing to the associated YouTube channel, indicating a belief in the value of the shared knowledge.

Prompt Engineering via Prompt Patterns — Cognitive Verifier Pattern

You can switch to video version of this article

There is research suggesting that large language models are able to reason much better if a question from user is subdivided into additional questions, that provide answers combined into the overall answer to the original question. Don’t worry if it didn’t make sense as it will in a few minutes. There is no magic to it, unless the question is very precise, to the point and really well scoped, there are hidden dimensions and assumptions involved in answering it. This is true for humans and models alike. A very high level question is difficult if not impossible to answer without followup questions to clarify certain aspects. A broad question combined with answers to additional, more specific questions helps the model combine information and knowledge it has for the topics associated with specific questions to come up with a much better and concrete answer.

So the cognitive verifier pattern works by instructing large language model to generate a number of additional questions with the goal of coming up with a more accurate answer than with the original question alone. In the same breath, you can instruct it to combine answers of these additional questions with the original question to produce the final answer(s) to the original question.

“When I ask you a question, generate three additional questions that would help you give a more accurate answer. When I have answered the three questions, combine the answers to produce the final answers to my original question”.

The pattern is best used for complicated tasks as breaking the question down into smaller component parts lets the model leverage its knowledge on specific topics to be combined to answer the top level question. By going step by step, it can also identify parts that are missing or where the prompt was unclear.

You can tweak the goal to nudge the questions in a certain way e.g. if you are trying to do a DIY project at home, you can ask the model to include questions to ensure you have the tool required to accomplish the task and take your answer into consideration while suggesting the steps for the project.

A very simple example can be a question like ‘Should I buy an apartment or a house?’, which by itself is difficult if not impossible to answer without additional information. Here the cognitive verifier pattern would help the model ask user additional questions like does he live in a city or a town, are there additional taxes on land ownership, or whether maintenance cost is high or low in the area. Similar question can be how much CO2 does a power plant produce. The answer would vary depending on the fuel type, size and many other factors. Factors which the model would try to gather through additional questions before trying to answer your question.

Hopefully that pattern would help you get answers with more reasoning and depth than before. If you liked, please give a clap/share. You can also consider subscribing to our YouTube channel as well. Thank you!!!.