Self-Rewarding Language Model

Current Language models are bottlenecked not only by the quantity of labeled data but also by the quality of labeled data. In our previous blog, we talked about DPO as a more stable and efficient way to optimize a model than RLHF. The problem is DPO requires that you have a substantial amount of labeled human preference data in order to train a model. This paper introduces agents that can both: Act as an instruction-following model, generate a response given a prompt, and Generate and evaluate new instruction-following examples to add to their own training set.

So, without further ado, let’s jump right into it.

Table of Contents

• Summary of LLM Training Pipeline
• The Problem of Labelled Data
• Self-Rewarding Language Model
• Result and Conclusion

Summary of LLM Training Pipeline

Before we understand Self-Reward we should take a quick look at the LLM training pipeline. Here are the different steps:

Step 1: Pre-training

1. Data Collection and Diversity: A vast dataset is gathered from the internet, ensuring a wide spectrum of text sources. The diversity of data is crucial for the model to understand various language styles, topics, and nuances.
2. Data Cleaning and Preprocessing: This involves removing irrelevant content, formatting issues, and noise. It’s a delicate process, as it needs to balance cleaning data while preserving meaningful context.
3. Tokenization: The text is broken down into manageable units (words, subwords) using techniques like Byte-Pair Encoding or WordPiece. This step is critical for handling the nuances of language, including morphology and syntax.
4. Use of Transformer Architecture: Transformers are chosen for their effectiveness in handling sequential data and their ability to capture long-range dependencies in text.
5. Training for Next-Word Prediction: The model is trained to predict the next word in a sequence. This step is foundational for the model to learn language structure and generate coherent text. However, at this stage, the model lacks the ability to understand context or user intentions deeply.

Output After Step 1:

• The model can generate text but may produce outputs that are contextually irrelevant or inappropriate, as it has yet to learn specific user interaction patterns or instructions.

Step 2: Supervised Fine-Tuning/Instruction Tuning

1. Learning from Targeted Inputs and Responses: The model is now trained with pairs of user inputs and appropriate responses. This training is crucial for the model to learn the context and relevance of responses.
2. Understanding Instructions and Knowledge Retrieval: The model starts to comprehend the meaning of instructions and learns how to retrieve and utilize its pre-trained knowledge effectively.

Output After Step 2:

• The model now can understand and respond to questions more accurately. It begins to establish a relation between questions and contextually appropriate responses.

Step 3: Reinforcement Learning from Human Feedback (RHFL)

1. Aligning with Human Preferences (Helpful, Honest, Harmless): The focus here is on refining the model’s outputs to align with ethical and practical human standards, encompassing helpfulness, honesty, and harmlessness.
2. Training Reward Model Using Human Feedback: Multiple outputs are generated for the same prompt, and human labelers rank them. This ranking data trains another neural network, the reward model, which understands human preferences in content.
3. Replacing Humans with Reward Model for Scalability: The reward model, once trained, takes over the role of humans in providing feedback, allowing for large-scale, efficient fine-tuning of the LLM.

Purpose of RHFL:

• This phase is critical for refining the model’s behavior. It ensures the model not only provides accurate and relevant responses but also does so in a manner that is ethically aligned with human values, avoiding misleading, harmful, or dangerous content.

In Summary:

• Pre-training lays the foundation for understanding language.
• Supervised Fine-Tuning/Instruction Tuning teaches the model to respond appropriately to specific instructions.
• RHFL aligns the model’s responses with ethical and human-centric criteria, enhancing its reliability and safety in real-world applications.

The Problem of Labeled Data

So, the idea is can we use synthetic data to train our language models? At first, this looks a bit counterintuitive, if the data that LLMs are going to generate is already in some way coming out of the data on the internet only, then how does it help? If during the pertaining of the LLMs, it learns the data distribution and then uses the same distribution to generate data, what more will we get? It is a perfectly valid thing to ask. Theoretically, a self-improving language model shouldn’t work, but we are starting to see synthetic data be a successful technique in new and upcoming papers.

The easiest way to think about this is that given that these models have, in essence, been exposed to the vast array of distributions (entire internet) available on the internet, the focus shifts towards strategically influencing the model’s output. This involves not just expanding the model’s exposure but directing its responses in specific ways that align with desired outcomes. The distinction is crucial: possessing a model capable of generating any response theoretically is one aspect, but steering it to produce specific, targeted responses is a far more nuanced challenge.

Being able to have the model generate it’s own synthetic data feels more like alignment or honing in skills than learning something new.

So, let’s keep this one question in our minds, can a language model create brand new text that adds to the vast corpora of knowledge? Or will it just saturate with ideas we already know, simply filtered and reformulated? It’s an interesting thought experiment that is good to keep in the back of your mind as we dive in.

Self-Rewarding Language Model

First, we start with a set of prompts. Then we seed a supervised fine-tuned (SFT) model to have two skills:

• Generate responses
• Evaluate responses.

Then we use the SFT model to generate data both in the form of prompts and responses as well as ranked preference pairs. There are multiple potential completions for each prompt, so we need to generate multiple responses and then use the same model to rank which response is better.

They use a clever “LLM-as-a-Judge” prompt to help generate and filter the data that is added back into the training dataset, in turn improving the model in a positive feedback loop.

Instruction Following

I hope you are familiar with supervised fine tuning to get your model to follow instructions, and align with preference pairs.

It is nice to separate an instruction or system message from the user prompt so that we can tailor the user experience or developer output. For example, we could also have an instruction that said “Always respond like a pirate”, and now every response has a little humor if that is what we are going for.

In the case of Self-Rewarding Language models, we will want our model to follow instructions to generate new data to be feeded back into the model.

LLM-as-a-Judge

The second step after getting a model to follow instructions, is to align the model to human preferences. Since there are multiple valid responses for each prompt, learning which responses we prefer takes a language model from a basic instruction following machine to a more robust and aligned model. Techniques like DPO or RLHF have been shown to improve the language model further than simply instruction fine-tuning.

In this case, the second preference optimization step uses the same LLM as a judge of its own outputs. For example, we may have the start of a sentence with two valid completions, where one is more correct than the other.

x: Iron man was ...

👍 the best MCU character of all time because of his character arc going 
from self obsession to sacrifice his own life
👎 ta good character

We would prefer the model generate the first completion rather than the second, so we rank it higher. We can have humans label these pairs that are generated from the model, but this is time-consuming and expensive. If the language model can judge its own outputs and give a quality score, then we could feed the data back in a self-improving loop.

Prompting as the Judge

The prompt is important to get working well for this entire pipeline to work. Later in the results section, they cite other prompts that have been tried that only have a 26% pairwise accuracy when compared to humans, whereas this particular prompt has 65.1% as a starting point.

This is not the first time this approach for using LLM as a judge has been proposed. The paper “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” looks at how well LLMs can judge the outputs of other LLMs their results are that:

Strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.

The interesting thing about this paper is that the initial LLM learns to be its own judge. They are not using a separate, smarter language model like GPT-4 as the judge. So not only does the base model get better at instruction following, but it gets better at judging its own outputs. This means as we iterate through training, we will each time we train a new model we will have a higher-quality preference dataset than the previous iteration.

A few key things to note here:

1. The initial model already needs to be able to generate reasonable responses for this to work
2. The reward prompt must do a decent job scoring the outputs to kick off the feedback loop

If we don’t meet these two conditions, we will be generating garbage, ranking garbage, and feeding garbage back into the model.

The model itself can then modify its own training set by:

1. Generate a new prompt, given a few show prompting
2. Generate N candidate responses, given each prompt
3. Evaluate each candidate's response, using the LLM-as-a-judge skill of the same instruct-tuned model to evaluate its own responses

Result and Conclusion

They show that the first iteration M1 with the EFT+IFT data does not outperform the IFT data by much (30.5 vs 30.9). They say this is good because this means that adding a self-reward skill does not impact the instruction following skill.

Then they show that the second iteration M2 provides superior instruction following iteration 1. This is promising that adding this new AIFT(M1) preference data is adding to performance. Iteration 3 also improves over iteration 2 when evaluated against itself by GPT-4.

Not only does it win against its baselines, but each iteration starts to get a higher win rate over GPT-4 Turbo as evaluated by GPT-4.

This is exciting because some of the models on the leaderboard here (Claude 2, Gemini Pro, GPT4 0314) typically contain a large dataset, sometimes over 1 million annotations, or use targets distilled from stronger models to generate the dataset. The self-improving technique starts with a much smaller dataset, and does not use any external models to generate or rank the responses.

Overall this paper is interesting because of the size of the dataset needed. They start with only a couple thousand examples and can outperform models that were trained on millions using this iterative process. The fact that they see improving performance over time without any human in the loop is also encouraging. This virtual cycle is promising in this small setting but is only a preliminary study.

A few things that would be interesting to see:

• When does this approach stop improving? Why did they stop at 3 steps?
• Would this method work with smaller (7B) models?
• What other prompts could we use in the LLM-as-a-judge step to direct the model in different directions? This has implications for safety as well as performance.

Writing such articles is very time-consuming; show some love and respect by clapping and sharing the article. Happy learning ❤

Please don’t forget to subscribe to AIGuys Digest Newsletter

References

[1] https://arxiv.org/pdf/2401.10020.pdf