Summary: Ambiguity-Aware In-Context Learning with Large Language Models

Sep 14 2023, Ambiguity-Aware In-Context Learning with Large Language Models — Lingyu Gao, Aditi Chaudhary, Krishna Srinivasan, Kazuma Hashimoto, Karthik Raman, Michael Bendersky

• This paper explores how to select the most effective demonstrations for in-context learning (ICL) with large language models (LLMs).
• LLMs are sensitive to the choice of prompts for ICL, so selecting good demonstrations is crucial. Using a text retriever to find semantically similar examples helps, but doesn’t consider the LLM’s existing knowledge about the task.
• The paper hypothesizes that considering the LLM’s existing knowledge about the task’s label space, especially ambiguity around labels, can help select better demonstrations.
• They propose constraints to select demonstrations that help resolve inherent label ambiguity for a test example. This includes using misclassified examples that fall on the decision boundary between the ambiguous labels.
• Experiments on text classification datasets show constraints based on label ambiguity improve over just using a retriever by +1.5–2.6% F1 macro. The improvements are higher for smaller LLM models.
• Analysis shows the ambiguous label set acts as a proxy for the true test label. Including more examples with these “likely” labels helps guide the model to the correct prediction.
• Overall, this work demonstrates that considering label space ambiguity and misclassified examples, in addition to semantic similarity, results in better ICL demonstration selection for text classification tasks.