AssociationAI / AI Literacy
AssociationAI team Published

Article

Beginner

Why an AI Tool Can Miss What Is in the Middle of a Long Document, and What That Means for Bylaws, Board Packets and Transcripts

A peer-reviewed 2024 paper found that the language models it tested in 2023 used information at the start and end of long inputs better than information in the middle. A 2024 benchmark found long-context models still struggled with inputs over 100K tokens. Here is what that means when association staff paste long documents into AI tools.

AI Literacy Governance Long Documents Board

Picture this illustrative case: a professional society is revising its bylaws, and the governance director has a 90-page redline that the board will vote on next month. She pastes the whole thing into an AI tool and asks for a summary of every change to member voting rights. The summary is tidy and confident. It covers the changes in the first article and the last few, and it says nothing about a change on page 47 that shortens the notice period for special meetings. This is a teaching example, not a case study, but it describes a real risk that research has measured, and it applies to anything long that association staff paste into a tool: bylaws, board packets, committee transcripts, accreditation standards, member survey comments.

A bigger window is not the same as reading all of it

Most AI tools now advertise how much text they can take in at once. It is natural to assume that if the document fits, the tool reads all of it equally. Research says that assumption is not safe.

The clearest evidence comes from a peer-reviewed paper, Lost in the Middle: How Language Models Use Long Contexts, by Nelson F. Liu and co-authors, published in the Transactions of the Association for Computational Linguistics in 2024. The models were tested in 2023. The ones the authors name in their introduction include MPT-30B-Instruct, LongChat-13B (16K), OpenAI’s GPT-3.5-Turbo and Anthropic’s Claude-1.3. Those are older models, and newer ones may behave differently. The finding still matters because of what it shows about how these systems can fail.

Their central result, from the abstract: “performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models”. In other words, the position of a fact inside a long input affected whether the models in the study used it.

The authors also tested whether models built for longer inputs did better. Their answer: “extended-context models are not necessarily better at using their input context”. And they drew a broad conclusion, “indicating that current language models do not robustly make use of information in long input contexts”.

Worse than having no document at all

One result from the paper is especially striking. In a test where a model had to answer a question using several documents, the paper reports that “when relevant information is placed in the middle of its input context, GPT-3.5-Turbo’s performance on the multi-document question task is lower than its performance when predicting without any documents”. That is one model, on one task, tested in 2023. But it shows that giving a tool the right document is not a guarantee that it will use it.

For the governance director, the parallel is direct. The page 47 change was in the document. Having it there did not mean the summary would reflect it.

Long inputs remain hard

A second peer-reviewed paper looked at much longer inputs. ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens, by Xinrong Zhang and co-authors at Tsinghua University, was published in the Proceedings of ACL 2024. The authors describe it as “the first LLM benchmark featuring an average data length surpassing 100K tokens”.

The benchmark was built to test real understanding rather than lookup. Its tasks were designed to “make simply retrieving a limited number of passages from contexts not sufficient for these tasks”. The result, in the authors’ words: “The experimental results indicate that existing long-context LLMs still require significant advancements to process 100K+ contexts effectively.”

Taken together, the two papers point the same way. Fitting a long document into a tool is a technical limit. Using everything in it is a separate ability, and the models these papers tested did not have it reliably.

What the papers tested, and what they did not

It helps to be precise about the evidence. The Lost in the Middle experiments used two kinds of task: answering a question from a set of documents, and finding a value by its key in a long list. The ∞Bench tasks covered several types of long input in English and Chinese, including summarizing a novel. Neither paper tested summarizing a document like a set of bylaws, and summarizing one is a different job from answering one question, so a tool may do better or worse at it.

Our view is that the lesson still carries over, because a summary is only as good as the parts of the document the tool actually used. But that is our inference, not a finding of either paper, and it is the reason this article recommends checking rather than a rule about how any tool behaves.

Why this matters more for associations than for most users

Association work runs on long documents where the important detail is often in the middle. A board packet puts the routine reports first and the contested item somewhere inside. A certification handbook puts eligibility up front and the appeal deadline in a later section. A committee transcript buries the decision in the second hour. A set of member survey comments has no order at all.

In each case the risk is the same as in the bylaws example. The summary will look complete. Nothing in it tells you what it skipped. And the people who rely on it, a board voting on a change or a member reading a summary of a policy, may never see the original.

What to do instead of trusting the summary

None of this means staff should stop using AI tools on long documents. It means a summary of a long document is a draft that needs checking against the document, with special attention to the middle.

Three habits follow from the research, and they are our recommendations rather than the papers’. First, ask about specific sections, not the whole document at once: “What changes does Article VI make to special meeting notice?” gets a more checkable answer than “summarize all changes.” Second, when something matters, find it in the original yourself, wherever it falls. Third, be most skeptical of a summary that is silent about the middle of the document, because silence is exactly what the position effect would produce.

Our same-day tutorial on checking whether an AI summary used the whole document turns that into a short test you can run on any long summary before it goes to a board or to members. For checking the facts inside a summary, our fact-check routine for member-facing AI still applies.

A caution about dates

Both papers tested models that are now older. The Lost in the Middle experiments ran on 2023 models, and the ∞Bench paper reports on models available when it was written. Tools have changed since, and some may handle long inputs better. The research does not tell you how today’s tool behaves. It tells you the failure is real, that it has been measured more than once, and that the only way to know whether your summary has it is to check.

Sources (2)