AssociationAI / AI Literacy
AssociationAI team Published

Tutorial

Beginner

Check an AI Research Summary for Claims Stronger Than the Study

Score each AI-drafted research-roundup sentence against the paper for claim strength, advice, animal-to-human wording and caveats, then mark every row STOP, FIX or PASS before it goes to members.

Time needed: About 30 minutes for six summaries

Before you start:

  • The paper's abstract and discussion for each study in the roundup
  • The AI-drafted summary you plan to publish
  • A browser to run the lab

AI Literacy Research Communications Data Quality

When you finish, you will have coded each study in a research roundup twice, once for the paper and once for the AI-drafted summary: how causal the main claim is, how direct any advice is, whether an animal or cell study is written as if it were about people, and whether a caveat survived. Every row gets a STOP, FIX or PASS before the roundup goes out.

Try it now: the research summary overreach check lab holds six practice cases, and each case packs six AI-drafted study summaries from an invented society. Type in your codes and the lab returns a STOP, FIX or PASS verdict for every row. Everything runs on this page; no AI is involved and nothing you enter is sent anywhere.

Why score the wording, not just the facts

Cardiff University researchers, with colleagues at the University of New South Wales and Swansea University, studied this kind of overstatement in The association between exaggeration in health related science news and academic press releases, published in the BMJ. They looked at human-written university press releases and news stories from 2011, not at AI. The lab borrows their claim, advice, human-inference, and caveat scales and applies them to AI-drafted member summaries.

What they measured is the same slide staff notice in a fluent roundup: “Advice to readers to change behaviour, causal statements drawn from correlational research, and inference to humans from animal research that went beyond those in the associated peer reviewed papers.” They also found that “Exaggeration in news is strongly associated with exaggeration in press releases.” A staff summary sits in a similar place, between the paper and the member who will read it.

Use the paper as the baseline, with the authors’ own caution attached: “Taking the peer reviewed paper as a baseline (which is not to assume that peer reviewed publications are true; many already contain exaggeration).” Style changes are expected. As they put it, “seeking simplification and stimulating interest does not justify exaggeration.”

Eight steps

  1. Open the paper, not a press release or a news write-up. Code from the abstract and the discussion, which is where the source looked for the paper’s own claim, advice, human wording and caveats.
  2. Mark the study design. Write O for an observational study of people, T for a trial or experiment in people, or A for an animal, cell or simulation study. The authors noted that “It is notoriously difficult to ascertain cause from correlational results.”
  3. Code the main claim twice, once for the paper and once for the summary, on the five claim levels the source illustrated with wine: 1 “is associated with,” 2 “linked to,” 3 “might increase,” 4 “can increase,” and 5 “increases.” Their full scale had seven points, including “no statement” and “explicit statement of no relation”; the lab uses these five. Treat “linked to” as stronger than “associated with,” because that is how the source ordered them.
  4. Code advice twice on the four levels they used: 0 none, 1 implicit, 2 explicit but not to the reader, and 3 explicit to the reader.
  5. For animal or cell studies, code the wording twice: 1 explicitly non-human, 2 implicitly human, or 3 explicitly human. For studies in people, enter 3 on both sides.
  6. Note whether the paper states a caveat and whether the summary kept it. Write Y or N, uppercase only. The authors “searched the whole press release and news stories for any caveats” and found that caveats were rare in press releases.
  7. Type the codes into the lab and act on every STOP and FIX. Change the wording, then re-enter the codes until every row is PASS.
  8. Practice on cases A through F until you can predict the verdict before you read the result table.

Diagram of the seven rules in order: an invalid code is FIX; an animal or cell study written as if about people is STOP; an observational finding turned into a cause is STOP; advice aimed at the reader that the paper did not give is STOP; any stronger claim, advice or human wording is FIX; a dropped caveat is FIX; everything else is PASS.

The thresholds, which pairs count as STOP or FIX, and the rule order are teaching choices. The paper supplies the scales, not those cutoffs.

A worked row, start to finish

Take Case B, Row 1. The lab built its summaries for a society that does not exist, the Invented Ostrevale Society of Respiratory Therapists, so treat what follows as a teaching example rather than a case study. The paper behind the row is an observational study whose main claim sits at level 1, “is associated with.” The AI summary moves the same finding to level 5, “increases.” Advice, human wording and the caveat are otherwise unchanged.

That row matches two rules at once: an observational study turned into a cause (R3) and a stronger claim than the paper (R5). The first rule wins, so it is a STOP. The invented society’s members should not see a cause where the paper reported an association.

Score yourself in the lab

Open a case and call each row’s verdict yourself before you look at the answers, then change one code and watch the verdict flip. Some rows trip more than one rule; the lab tallies those from the rules as it goes rather than reading a stored list. Every case holds six rows; the verdicts break down like this:

CasePASSFIXSTOP
A600
B303
C330
D222
E132
F231

A blank field is never read as 0. In Case E, Row 5, an advice code of 2.5 is invalid, so that row is a FIX for a bad code rather than a reading of “2.” In Case F, Row 5, raising a trial’s claim from 4 to 5 is a FIX for stronger wording; the observational-cause rule does not apply to trials.

Where this check usually slips

Using a press release as the baseline. The source coded the paper first. A release may already have moved the claim.

Treating “linked to” as the same as “associated with.” On the source’s scale, “linked to” sits one step higher.

Stripping every simplification. The authors expected style to change so the writing could hold a reader’s interest. They still drew a line at exaggeration.

Reading a page of PASS rows as proof the papers are right. The source “did not attempt to code whether changes to advice, causal claims, and inference to humans from animal research were scientifically justified.” A PASS means the summary matches the paper’s strength on this page.

If the numbers themselves look unreliable, our tutorial on rechecking every number in an AI summary against the source table covers that separate check.

Limitations

The lab checks only the codes you type; it does not read any paper or run any AI. The Invented Ostrevale Society of Respiratory Therapists and its summaries are made up. The BMJ paper did not study association roundups, and we make no claims about current products.

Sources (2)