Soft attention in Transformer-based Large Language Models (LLMs) is susceptible to incorporating irrelevant information from the context into its latent representations, which adversely affects next token generations. To help rectify these issues, we introduce System 2 Attention (S2A), which leverages the ability of LLMs to reason in natural language and follow instructions in order to decide what to attend to. S2A regenerates the input context to only include the relevant portions, before attending to the regenerated context to elicit the final response. In experiments, S2A outperforms standard attention-based LLMs on three tasks containing opinion or irrelevant information: QA, math word problems and longform generation, where S2A increases factuality and objectivity, and decreases sycophancy.
S2A has two steps. First, we want to regenerate the context to remove irrelevant or distracting information, filtering out the noise, and ensuring that only important details remain. This is done by prompting the model to rewrite the text and extract only the useful parts.
Then, we generate the response using only the regenerated context. This focuses attention on what’s relevant.
Instructed prompting Our S2A prompt given in Figure 2 encourages to remove opinionated text from the context, and uses instructions in step 2 (Figure 13) to ask for the response to be unopinionated. We can ablate the latter, and remove this instruction in the prompt template for step 2 of S2A. We can also compare this to a further baseline where we simply add the extra instruction request from Figure 13 to the original context (rather than performing S2A at all), termed instructed prompting in Shi et al. (2023).
Emphasize Relevance/Irrelevance The S2A implementations so far described have emphasized regenerating the context to increase objectivity and reduce ycophancy. However, there are other ways of emphasizing what context should be attended to. For example, one could emphasize relevance versus irrelevance instead. An instance of such an approach is given in the prompt variant in Figure 15, which we will also use in our experiments.
