Knowledge/language maps // better answers for LLM/RAG
Naive RAG does not work well // a new (old) method using “language maps” is proposed to improve quality, but it looks like old good intents!
Mutable.ai has found that traditional vector-based Retrieval-Augmented Generation (RAG) methods are inadequate for real-world applications like codebase chats. Instead, they propose using “language maps” to enhance performance. Their long-term mission is to accelerate organizations with AI by simplifying software understanding and building.
Challenges with current Large Language Models (LLMs) include small context windows and the need for immediate reasoning, which hinder effective assistance with code. Mutable.ai initially used keyword retrieval and vector embeddings for their chat but faced poor performance, prompting a shift in strategy.
They developed a Wikipedia-style document, “Auto Wiki,” to understand codebases better. This document features diagrams and citations, providing a deeper understanding and improving context retrieval for LLMs. Using this approach, the answers are more accurate and relevant, demonstrating superior performance over traditional methods.
Mutable.ai’s new method, which you can test at wiki.mutable.ai, has shown consistently strong results across various codebases. They plan to release evaluations showcasing these improvements soon.
Why?
As we built RAG, we continued to notice hallucinations or incorrect answers. we realized three key issues:
There wasn’t enough data in the vector to provide a coherent answer. i.e. vector was 2 sentences, but the answer was the entire paragraph or multiple paragraphs.
LLM’s try to merge an answer from multiple different vectors which made an answer that looked right but wasn’t.
End users couldn’t figure out where the doc came from and if it was accurate.
We solved this problem by doing the following:
Figure out document layout (we posted about it a few days ago.) This will make issue one much less common.
Split each “chunk” into separate prompts (Agent approach) to find exact quotes that may be important to answering the question. This fixes issue 2.
Ask the LLM to only give direct quotes with references to the document it came from, both in step one and step two of the LLM answer generation. This solves issue 3.
