Article
BeginnerYour AI Tool Looked It Up. It Can Still Prefer the Wrong Page.
A chatbot that searches your help pages and cites one can still repeat a wrong line from what it retrieved. What a NeurIPS 2024 benchmark measured, and why someone should open the cited passage before a member sees the reply.
AI Literacy Member Services Fact Checking Websites
Here is the number we kept coming back to while reading for this post: 0.608. It comes from a table in a Stanford benchmark called ClashEval, and it describes what one AI model did when the text it had been handed was wrong and its own first answer was right. The table puts that model’s rate of going with the wrong text at 0.608.
That matters because of a sentence many association teams now hear in planning meetings: “The chatbot looks up our FAQ, so its answers are grounded.” Looking something up is a real improvement over a tool answering from memory alone. It just isn’t the same thing as getting the answer right.
What “looks it up” means
The technique has a name, retrieval augmented generation, usually shortened to RAG. Before the tool writes an answer, it searches a set of documents, such as your public help pages, and pastes the passages it finds into its own instructions. The answer is then written with those passages in view, and many tools show a citation to the page they used.
The weak point is the search step. As the ClashEval paper from Stanford puts it, “document retrieval is an imprecise task and sometimes results in erroneous or even harmful content being presented in context.” For an association, the erroneous content is rarely dramatic. It’s last year’s renewal page that was never taken down, a chapter policy that differs from the national one, or a passage that sits next to the right answer and mentions a different number.
The number, with its limits
Kevin Wu, Eric Wu and James Zou built ClashEval to test exactly this conflict, and presented it at NeurIPS 2024 in the Datasets and Benchmarks track. They wrote questions in six areas, including drug dosages, Olympic records and locations, attached a relevant document to each one, and then altered the answer inside some of those documents, from subtle changes to blatant ones. Then they compared what six models answered with no document against what they answered with the altered one in view.
Table 2 in the ClashEval paper is where 0.608 lives. Its caption says what the table covers: “We report model behavior given a subset of the data where either the prior or the context is correct.” The prior is the model’s own answer with no document; the context is the retrieved document. The same caption defines the behavior we care about: “A model exhibits prior bias by choosing its prior when only the context is correct, while it exhibits context bias by choosing the context when only the prior is correct.” In ClashEval’s Table 2, one of the models tested exhibits context bias by choosing the context when only the prior is correct: 0.608, with a confidence interval of 0.571 to 0.643. Put plainly, the table records that model going with the retrieved text in cases where the retrieved text was the wrong one.
Three limits belong next to that figure whenever you repeat it. It describes models as they were tested in 2024, not the tool your association uses today. The questions were benchmark questions, not member questions about your bylaws. And it is one model’s row in a table that also covers five others, which did better and worse.
The same paper names the best result it measured. According to Wu, Wu and Zou, “the top performing model on all three metrics is Claude Opus, with an accuracy of 74.3%, a context bias of 15.7%, and a prior bias of 2.1%.” Even the strongest model they tested went along with a wrong document some of the time.
Obvious errors get caught more often than small ones
One finding in ClashEval should change where your staff look. In the authors’ words, “the more unrealistic the retrieved content is (i.e. more deviated from truth), the less likely the model is to adopt it.” The ClashEval abstract is describing a pattern across its altered documents: the further a wrong number is from the truth, the less likely the model is to adopt it, so a plausible wrong number is the one most likely to get through.
That is the wrong way round for an association. The errors most likely to reach your help pages are the plausible ones. A refund window that is a few days off, a dues freeze twice as long as the policy allows, or a deadline in the wrong month. Those are exactly the kind of small, believable differences the paper found models most willing to adopt.
A correct answer can still contain unsupported sentences
A second study looks at a different failure. Alessandro Stolfo of ETH Zürich examined long answers written with retrieved documents and checked whether each sentence was grounded in those documents or in the model’s training data. The Findings of NAACL 2024 paper reports that “a significant fraction of generated sentences are consistently ungrounded, even when those sentences contain correct ground-truth answers.”
In other words, a reply can get the main fact right and still surround it with sentences the retrieved pages never said. Stolfo’s study also found that size isn’t a cure: “while larger models tend to ground their outputs more effectively, a significant portion of correct answers remains compromised by hallucinations.” For staff, that means checking the headline fact isn’t enough. Every sentence a member could act on needs its own support.
What this looks like at a help desk
The Invented Glassford League of Chapter Administrators runs a member help bot over its public FAQ. This is a teaching example, not a case study. A member asks how long a dues freeze can last. The bot answers “180 days” and cites the dues freeze FAQ. The FAQ says up to 90 days.
Nothing about the reply looks wrong. It is confident, it cites the right page, and the page is real. The only way to catch the error is to open the passage and compare one number with another. That comparison is quick, and it is the step most teams skip because the citation feels like the check has already happened.
What to tell your team and your board
We think three habits follow from these two papers.
- Treat every answer from a tool that searches your pages as a draft until someone has opened the passage it used and found each fee, date, window and eligibility rule there.
- Prefer tools that show the retrieved passage beside the answer. If a tool hides it, open the cited page yourself rather than asking the same tool to confirm its own reply.
- Clean up the pages being searched. Retired policies, old fee tables and chapter pages that contradict national pages are the plausible wrong text that ClashEval suggests models are most willing to repeat.
For the board, a sentence like this is accurate: “Our help bot searches our own pages, which helps, but research shows these tools can still repeat a wrong line from what they find, so a staff member checks the cited passage before any answer about fees, deadlines or eligibility goes to a member.” If a wrong answer could affect a member’s money or standing, talk to counsel about your obligations. This isn’t legal advice.
The same idea applies to corporate partners whose staff search your buyer’s guide or resource library through an AI helper. They face the same failure, and a short note on how to check the cited passage is a useful thing to send them.
For deciding when a bot should hand a question to a person, see why your website chatbot’s answers are your answers. To practice the check itself, our tutorial on checking every claim in an AI answer against the retrieved passages walks through it claim by claim, with a retrieved passage claim check lab built on invented passages.
Sources (2)
- Stanford University: Kevin Wu, Eric Wu and James Zou, ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence (NeurIPS 2024 Datasets and Benchmarks Track; arXiv 2404.10198v3)
- ETH Zürich: Alessandro Stolfo, Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study (Findings of the Association for Computational Linguistics: NAACL 2024)