Tutorial
BeginnerCheck Whether an AI Summary Used the Whole Document Before It Goes to the Board
Before you trust an AI summary of a long document, split the document into beginning, middle and end, ask questions whose answers sit in the middle or need two distant passages, and compare the answers with the document yourself. The steps are our recommended practice; the sample document is invented.
Time needed: About 25 minutes for a 30 to 60 page document
Before you start:
- A long document you have an AI summary of, such as a board packet, bylaws redline or committee transcript, or use the sample
- The AI summary itself
- About 20 minutes with the original document open
AI Literacy Governance Long Documents Board
Picture this illustrative case: a trade association’s accreditation manager has a 40-page committee report on revised program standards and an AI summary of it that is due in the board packet tomorrow. The summary reads well. Before it goes out, she wants to know whether it reflects the whole report or mostly the first and last few pages. By the end of this tutorial you will have run a short coverage test on any AI summary of a long document and written down what it missed. The steps, the question types and the worksheet are our recommended practice; the research below is about testing models, and no source prescribes a staff procedure.
Try it now: the summary coverage lab runs in your browser with an invented committee report and summary already loaded. It splits the document into beginning, middle and end, picks sentences with dates and numbers from each part, and shows which parts the summary mentions. Nothing you paste leaves the page. For the printable worksheet, write to ai@trihelix.co.
Why fitting in the window is not enough
Researchers who build long-document tests do not trust automatic scores alone. The L-Eval benchmark, published at ACL 2024, was built with “more than 2,000 human-labeled query-response pairs including diverse task types, domains, and input length (3k∼200k tokens)”. Its authors note that earlier benchmarks “mostly employ n-gram metrics such as ROUGE”, which count overlapping words, and they report that “Length-instruction-enhanced (LIE) evaluation and LLM judges can better correlate with human judgments”. For a staff member, the takeaway is simple: a person checking against the document is the reference point.
The LooGLE benchmark, also at ACL 2024, separated questions by how far apart the needed information sits. Its authors “Manually designed both short and long dependency tasks.” Their finding about the models they tested is blunt: “most LLMs have shockingly bad long context ability and fail to capture long dependencies in the context, even when their context window size is enough to fit the entire document”. That is the authors’ own phrase about the models they tested. They argue for improving understanding “instead of merely enlarging their context window”.
Our same-day article on why AI can miss the middle of a long document covers the research on where in a document information gets lost.
The sample report
The report and summary loaded in the lab are invented for teaching; they describe no real association. The report has a fee change near the start, a new appeal deadline in the middle, a cross-reference that changes the meaning of an earlier section, and an effective date at the end. The summary mentions the fee change and the effective date only.
Six steps
- Split the document into three parts. Beginning, middle and end, by page count. The lab does this for pasted text.
- Pick two facts from the middle. Choose a date, a number, a deadline or a named change. These are your short dependency questions.
- Write one question that needs two distant passages. For example, a rule in one section that is modified by a definition in another. This is your long dependency question.
- Ask the tool those questions, or read the summary for them. Write down what it says, or that it says nothing.
- Check each answer in the document yourself. Mark it correct, wrong or missing.
- Fix or flag the summary. Add what was missed, or send it out labelled as covering only the parts you checked.
Before and after on the sample
| Question | Part | Summary before | After the check |
|---|---|---|---|
| What is the new appeal deadline? | Middle | Not mentioned | Added, checked against section 7 |
| Does section 9 change how section 3 applies? | Middle and beginning | Not mentioned | Added, checked against both sections |
| What is the fee change? | Beginning | Correct | No change |
| When does it take effect? | End | Correct | No change |
Check your result
- At least two questions came from the middle third.
- At least one question needed two passages far apart.
- Every answer was checked in the original, not in the summary.
- The summary that goes out either covers what you found or says what it does not cover.
Where a coverage check goes wrong
Asking only about the first page. That tests the part the tool was most likely to use.
Checking the summary against itself. Ask whether each fact is in the document, not whether the summary sounds consistent.
Trusting a bigger window. Fitting the document in is not the same as using all of it.
Testing once and moving on. A tool that covered one report well can still miss the middle of the next one. Run the check on every summary that goes to the board or to members.
Sources (2)
- Chenxin An and co-authors (Fudan University; The University of Hong Kong; University of Illinois Urbana-Champaign; Shanghai AI Lab): L-Eval: Instituting Standardized Evaluation for Long Context Language Models (Proceedings of ACL 2024, Long Papers; peer reviewed)
- Jiaqi Li, Mengmeng Wang, Zilong Zheng and Muhan Zhang (National Key Laboratory of General Artificial Intelligence, BIGAI; Institute for Artificial Intelligence, Peking University): LooGLE: Can Long-Context Language Models Understand Long Contexts? (Proceedings of ACL 2024, Long Papers; peer reviewed)