We present STEP-BACK PROMPTING, a simple prompting technique that enables LLMs to do abstractions to derive high-level concepts and first principles from instances containing specific details. Using the concepts and principles to guide the reasoning steps, LLMs significantly improve their abilities in following a correct reasoning path towards the solution. We conduct experiments of STEP-BACK PROMPTING with PaLM-2L models and observe substantial performance gains on a wide range of challenging reasoning-intensive tasks including STEM, Knowledge QA, and Multi-Hop Reasoning. For instance, STEP-BACK PROMPTING improves PaLM-2L performance on MMLU Physics and Chemistry by 7% and 11%, TimeQA by 27%, and MuSiQue by 7%.
Here is a summary of the key points from the paper:
- The paper proposes STEP-BACK PROMPTING, a technique to improve reasoning capabilities in large language models (LLMs) like GPT-3 and PaLM by first prompting the model to “step back” and abstract key concepts before reasoning to the answer.
- STEP-BACK PROMPTING has two steps:
- Abstraction — Prompt the LLM to identify high-level concepts or principles relevant to answering the question. This acts as a “step back” from the specific details.
- Reasoning — Given the high-level abstraction, prompt the LLM to reason through the steps to arrive at an answer.
- Experiments across STEM, knowledge QA, and multi-hop reasoning tasks show STEP-BACK PROMPTING substantially improves performance of LLMs like PaLM-2L (up to 27% gains). It outperforms chain-of-thought prompting.
- Key advantage is reducing reasoning errors by grounding on high-level abstractions. Analysis shows abstraction is an easy skill to teach LLMs.
- Limitation is reasoning skills remain a bottleneck. But abstraction provides a way to simplify reasoning.
- Main takeaway is prompting LLMs to do abstraction before reasoning through complex questions is an effective technique to improve performance. abstraction provides a “step back” to ground reasoning.
