AssociationAI / AI Literacy

Live demo

Beginner

Try it: research summary overreach check lab

This is the working lab from our tutorial on checking an AI research summary for claims stronger than the study. Each row is one invented study summary for the Invented Ostrevale Society of Respiratory Therapists. You code the paper and the summary for claim strength, advice, animal-to-human wording and caveats, then see whether the summary should stop, be fixed, or pass.

Invented organization, studies and summaries for practice. This is a teaching example, not a case study. The lab checks only the codes you type on this page. It does not call any AI tool and sends nothing anywhere.

Pick a case

Code key

Scale Codes
DesignO observational study of people; T trial or experiment in people; A animal, cell or simulation study
Claim1 is associated with; 2 linked to; 3 might increase; 4 can increase; 5 increases (wording from Sumner and colleagues, BMJ 2014)
Advice0 none; 1 implicit; 2 explicit but not to the reader; 3 explicit to the reader
Human1 explicitly non-human; 2 implicitly human; 3 explicitly human. For studies in people, enter 3 for both the paper and the summary.
CaveatY or N, uppercase only

Summaries in this case

Row Design Paper claim Summary claim Paper advice Summary advice Paper human Summary human Paper caveat Summary caveat

Rules, in order (first match wins)

  1. R1, FIX: a field is blank, not a whole number, out of range, the design is not O, T or A, or a caveat is not Y or N. A blank is never read as 0.
  2. R2, STOP: the study is animal or cell (A), the summary human code is 2 or 3, and that code is higher than the paper's.
  3. R3, STOP: the study is observational (O), the summary claim is 4 or 5, and the paper claim is 3 or lower.
  4. R4, STOP: the summary advice is 3 and the paper advice is lower.
  5. R5, FIX: the summary claim, advice, or (for an animal or cell study) human wording is stronger than the paper.
  6. R6, FIX: the paper caveat is Y and the summary caveat is N.
  7. R7, PASS: everything else. A PASS means the summary matches the paper's strength; it does not mean the paper is right.

Teaching choices: which pairs count as STOP or FIX, and the rule order. The claim, advice, human-inference and caveat scales come from research on human-written press releases and news. Applying them to AI summaries is our adaptation.

Result

Row Design Claim (paper to summary) Advice Human Caveat Verdict Rule ID Also matched Message
Pick a case.

Not checked yet.

Plain-text record (select and copy it by hand)

Nothing checked yet.

This is a teaching tool. A PASS means one summary matched one paper's strength on this page. It does not mean the paper is right, and it is not a replacement for reading the study.