Live demo
BeginnerTry it: research summary overreach check lab
This is the working lab from our tutorial on checking an AI research summary for claims stronger than the study. Each row is one invented study summary for the Invented Ostrevale Society of Respiratory Therapists. You code the paper and the summary for claim strength, advice, animal-to-human wording and caveats, then see whether the summary should stop, be fixed, or pass.
Invented organization, studies and summaries for practice. This is a teaching example, not a case study. The lab checks only the codes you type on this page. It does not call any AI tool and sends nothing anywhere.
Pick a case
Code key
| Scale | Codes |
|---|---|
| Design | O observational study of people; T trial or experiment in people; A animal, cell or simulation study |
| Claim | 1 is associated with; 2 linked to; 3 might increase; 4 can increase; 5 increases (wording from Sumner and colleagues, BMJ 2014) |
| Advice | 0 none; 1 implicit; 2 explicit but not to the reader; 3 explicit to the reader |
| Human | 1 explicitly non-human; 2 implicitly human; 3 explicitly human. For studies in people, enter 3 for both the paper and the summary. |
| Caveat | Y or N, uppercase only |
Summaries in this case
| Row | Design | Paper claim | Summary claim | Paper advice | Summary advice | Paper human | Summary human | Paper caveat | Summary caveat |
|---|
Rules, in order (first match wins)
- R1, FIX: a field is blank, not a whole number, out of range, the design is not O, T or A, or a caveat is not Y or N. A blank is never read as 0.
- R2, STOP: the study is animal or cell (A), the summary human code is 2 or 3, and that code is higher than the paper's.
- R3, STOP: the study is observational (O), the summary claim is 4 or 5, and the paper claim is 3 or lower.
- R4, STOP: the summary advice is 3 and the paper advice is lower.
- R5, FIX: the summary claim, advice, or (for an animal or cell study) human wording is stronger than the paper.
- R6, FIX: the paper caveat is Y and the summary caveat is N.
- R7, PASS: everything else. A PASS means the summary matches the paper's strength; it does not mean the paper is right.
Teaching choices: which pairs count as STOP or FIX, and the rule order. The claim, advice, human-inference and caveat scales come from research on human-written press releases and news. Applying them to AI summaries is our adaptation.
Result
| Row | Design | Claim (paper to summary) | Advice | Human | Caveat | Verdict | Rule ID | Also matched | Message |
|---|---|---|---|---|---|---|---|---|---|
| Pick a case. | |||||||||
Not checked yet.
Plain-text record (select and copy it by hand)
Nothing checked yet.
This is a teaching tool. A PASS means one summary matched one paper's strength on this page. It does not mean the paper is right, and it is not a replacement for reading the study.