Halving Hallucinations: Jacek Cala on NIC-D’s Breakthrough Research into Graph-Based RAG and What It Means for Enterprise AI
Q1. Your team’s research shows that introducing graph-based tools into a RAG system can significantly increase the precision and recall of factual correctness, halve the number of hallucinated answers, and achieve the highest fine-grained truthfulness score among the evaluated scenarios — all with a modest increase in token usage. For an enterprise leader who is not a researcher, what is the single most important thing those numbers mean in practical terms — and why has this specific problem, the accuracy of RAG retrieval, turned out to be so consequential for whether AI deployments succeed or fail in production?
The single most important recommendation is “use the proposed graph-based RAG whenever possible”, that is, whenever the structure of your data and your application make this feasible. In many cases, constructing the simple graph required by our approach may be relatively easy, and performance gains are likely to be worth it. However, our research is by no means exhaustive, and there may be applications in which a simple graph does not provide measurable gains. An important direction for future work is, therefore, to validate whether our findings hold beyond the benchmark dataset and experiment we conducted.
To answer the other part of your question: “why (…) RAG retrieval turned out to be so consequential (…) in production”, we need to remember that LLM-based systems are known to be “confident and fluent liars”. If they lack the information to answer correctly, they can very effectively combine facts from their parametric memory and produce a plausible, eloquent answer. One that adds false claims, omits relevant ones, or does both.
This may not be a major problem when, for example, someone uses an AI system to choose a restaurant for dinner with friends. An inaccurate recommendation might cost some embarrassment and perhaps result in an underwhelming evening. In production, the situation is different, and consequences can be considerably more serious. Users expect reliability. They want to trust the systems and services they use.
Inaccurate answers are a critical concern in production for at least two reasons. First, when a system interacts directly with users, incorrect answers can undermine users’ trust, cause additional work to verify responses, and lead to overall dissatisfaction. Second, and perhaps more importantly, when an AI-based component is part of a larger system, its errors may propagate downstream. They may be hidden in logs and difficult to trace. And when an incorrect answer carries substantial cost, we have to minimise the risk of such errors.
By providing information relevant to a question, RAG can ground the model’s response in external evidence and so reduce the problem of hallucinations and omissions. RAG does not, however, eliminate these problems, and its benefits depend on both retrieval quality and the model’s ability to use it. What’s more, the situation is different when the retrieved information is irrelevant, misleading, or inconsistent with the parametric memory. Research shows that irrelevant passages, often described as distractors, can reduce QA accuracy. It is, therefore, crucial to improve RAG performance – its precision and recall – to build reliable and trustworthy AI systems.
Q2. One of the most striking aspects of your research is that these results were achieved using a lightweight graph structure with a relatively simple graph schema to support the RAG subsystem. That is a counterintuitive finding — most people assume that significant performance improvements require significant additional complexity.
What is it about the graph structure, specifically, even a simple one, that produces such a disproportionate improvement in retrieval accuracy — and what does that tell us about the fundamental weakness of conventional vector-only retrieval approaches for complex question answering?
Yes, when the project started, we were unsure which dataset and benchmark to use. Even midway through the project, after the dataset was selected, we still had doubts about whether simple links between documents, sections, and subsections would be helpful at all. It is now fairly clear that they are.
This is because they provide a reliable indication of semantic relationships between concepts. If two documents, or two sections within a document, are linked, they are likely to be related in some meaningful way. It is, therefore, worth exploring the adjacent nodes to see whether that relationship is relevant to the questions at hand.
For complex questions that involve multiple entities and documents and need several retrieval hops to find the answer, these well-grounded links are particularly valuable. But the most counterintuitive was that even a simple graph structure could help, provided that the links were reliable. By contrast, using only vector RAG, systems must generally do considerably more work to discover the relationships that are already represented in the graph.
Q3. Your study evaluated the system on MoNaCo, a challenging Wikipedia QA benchmark of complex query answering tasks, and found that answer rates increased from 29% to nearly 66% compared with conventional retrieval methods. Complex multi-hop questions — where the answer requires reasoning across multiple connected facts — are precisely the kinds of questions that enterprise AI agents are increasingly being asked to handle in real business workflows. What makes multi-hop reasoning so difficult for vector-only RAG systems, and why does the graph-based approach handle it more reliably?
Basic vector-based RAG compresses much of the information from a chunk into a single vector and then compares that vector with the representation of a question. This is an approximate method of measuring semantic similarity between the question and potentially relevant information. Among the chunks identified as semantically similar, however, many may be only loosely relevant.
If answering a question requires repeating this process several times, in other words, “digging deeper and deeper” for the answer, the errors accumulate. The system may quickly begin traversing a part of the knowledge base that is entirely irrelevant to the original question. Graph-based RAG, when implemented correctly, introduces reliable links between concepts or chunks. Using these links alone, or sometimes instead of, vector similarity can reduce the risk of retrieval errors at each traversal step and, ultimately, across the entire retrieval path.
Q4. Gartner has warned that unreliable outputs and failed RAG implementations contributed to the failure of half of generative AI projects last year. NIC-D sits at the intersection of academic research and enterprise applications. In your experience working with UK enterprises through NIC-D, what is the gap between how enterprises typically implement RAG today and what your research suggests they should be doing — and what are the practical barriers to organizations making that transition, beyond the technical ones?
Businesses vary considerably in their capabilities and needs. For some, even basic RAG can improve how they work with their data. For others, a more carefully designed approach may be required, especially when the data is less structured. The proposed graph-based RAG approach is certainly a valuable addition to the RAG toolkit, but whether or not it is suitable for a particular application and dataset depends on the type and structure of the data involved. The key challenge is determining which approach is most appropriate. There is no single, simple answer.
Q5. Your research used a curated subset of English Wikipedia articles as the structured dataset. Enterprise knowledge, by contrast, is typically fragmented across documents, databases, legacy systems, SharePoint sites, and unstructured sources — and it is rarely as clean or consistently structured as Wikipedia. How should enterprises think about the knowledge graph construction challenge — the process of building the graph from their own messy, heterogeneous data — and what is the minimum viable graph schema that your research suggests is sufficient to produce meaningful improvements in RAG accuracy?
That is partly what our paper tries to address. The messy structure of existing knowledge bases requires more than simply feeding random document chunks into a RAG-based AI system and hoping that it will make sense of them. Our proposition was to investigate whether simple, semi-structured data can be used to improve the performance of the end-to-end QA system.
The findings are very encouraging, at least for documents. We do not need dense, highly detailed knowledge graphs – even a simple graph representing document structure can improve performance. However, it is currently difficult to say whether these findings will generalise to repositories such as SharePoint or to other content formats, e.g., slide decks. That is an important direction for future work.
Q6. NIC-D’s mission is to accelerate data-driven innovation across the UK economy by connecting academic research with real enterprise application. This research represents exactly that kind of bridge — a rigorous benchmarking study with direct implications for how organizations deploy AI. What do you believe is the most important thing that UK enterprises, and enterprises more broadly, should do differently in the next twelve months as a result of these findings — and what is the research question your team is most excited to pursue next in this space?
It is important to understand that RAG means much more than vector-based similarity search. Vector-based similarity is a very useful approach – relatively easy to implement and can produce good results. However, there are many ways to extend it and build hybrid RAG-based AI systems. And graph-based retrieval should probably be among the first extensions to consider.
As noted in response to the first question, production use sets a much higher bar for accuracy and reliability. Our next goal is to understand whether the results presented in our research hold in production settings.
Resources
Graph-Powered AI Reduces Hallucinations and Boosts Answer Accuracy
…………………………………………….
Dr. Jacek Cała is the Principal Data Science Lead at the National Innovation Centre for Data (NICD). His background lies in scalable computing, with extensive expertise in machine learning, cloud computing, reinforcement learning, scientific workflows, and optimization.
He holds a PhD in Computing focusing on the adaptive deployment of component-based applications in distributed systems. Alongside his leadership role at NICD, he actively contributes to academic research as a Senior Research Associate in Computing Science at Newcastle University.