AssociationAI / AI Literacy
AssociationAI team Published

Tutorial

Beginner

Ask the Same Question Several Times and Compare the Facts Before You Use Any of Them

Ask an AI tool one factual question three to five times in new chats, compare each fact across the answers, and check every fact you keep against the source. The steps are our recommended practice, adapted from a University of Cambridge method.

Time needed: About 15 minutes

Before you start:

  • One factual question that will reach a member or a decision-maker, such as a deadline, a requirement or a number
  • An AI tool your organization allows
  • The source document that holds the real answer, such as bylaws or a member handbook

AI Literacy Judgment Member Services Fact Checking

A member asks how long the late-renewal grace period is, and an AI tool answers in one tidy sentence. You could paste it into the reply. By the end of this tutorial you will have asked the same question several times, compared each fact across the answers, and checked the facts you keep against the document that governs them. The steps and the three-to-five count are our recommended practice; the research behind them automates the comparison and does not prescribe a routine for staff.

Try it now: the same-question repeat lab takes one question and up to five answers you got from separate chats. List the facts the answers state, type each answer’s value for each fact, and mark whether you checked the source. The lab gives every fact a PASS, FIX or STOP, totals them, and builds a plain-text record of what you compared. Invented examples are ready to load, including the one below.

Where the method comes from

Researchers at the University of Cambridge built a way to spot made-up facts by asking a model the same thing more than once. In SelfCheckGPT, they describe how the method “draws a further N stochastic LLM response samples” and compares them with the first answer. Their reasoning is the part worth borrowing: “if the information in a sentence appears in many drawn samples, one may assume that the information is factual, whereas if the statement appears in no other sample, it is likely a hallucination.”

They also tested how many samples it takes. In their words, “the performance of SelfCheckGPT increases smoothly as more samples are used, with diminishing gains as more samples are generated.” Three to five is our choice: enough to see a fact wobble, few enough to finish in a few minutes.

Our same-day article explains why the same question can get different answers from an AI tool, including why answers that agree can still be wrong.

A flow in four boxes: ask again in a new chat until you have three to five answers, list the facts, compare each fact's values, then check every fact you use against the source.

Eight steps

  1. Write the question once. Keep it word for word, so any change in the answers comes from the tool, not from you.
  2. Ask it in a new chat. We start fresh each time so earlier answers are not in view; that is our practice, not a finding.
  3. Repeat until you have three to five answers.
  4. List each fact the answers state. Look for who, what, when and how much. A deadline, a number of credits and a named approver are three separate facts.
  5. Enter each answer’s value for each fact. Type numbers as digits and dates as YYYY-MM-DD so the lab compares like with like. Leave a cell empty when that answer did not state the fact.
  6. Read the verdict for each fact. STOP means the answers disagree or the source says something else. FIX means too few answers stated the fact, or the answers agree but nobody has checked the source yet.
  7. Check every fact you plan to use against the source. Do this for the facts that agreed too.
  8. Record what you checked and where. A page number or section lets a colleague confirm it in a minute.

The worked example: a grace period that moved

This is case A in the lab, and it is invented. A coordinator at the Northgate Association of Dental Hygienists asked, “How long is the late-renewal grace period?” in four new chats. The grace period values below are in days.

A table for the invented Northgate example: the grace period reads 60, 60, 30 and 60 days and is marked STOP; who approves exceptions reads membership director in three answers, is confirmed in the bylaws, and is marked PASS.

FactAnswer 1Answer 2Answer 3Answer 4Source checkVerdict
Grace period60603060Not yet checkedSTOP: two different values
Who approves exceptionsmembership directormembership directormembership director(not stated)Bylaws 3.4 agreesPASS

Three answers out of four gave the same number, and a majority can feel like a result. It is not one. The lab stops on the grace period because the tool gave two values, and the reply waits until the coordinator reads the policy itself. The approver passes only because the bylaws confirm it and the note says where.

The overall line reads: “1 PASS, 0 FIX, 1 STOP. Do not send anything with a STOP fact.”

Check your result

  1. You have at least three answers, each from a new chat.
  2. Every fact in your reply has a value from each answer, or a blank where an answer left it out.
  3. Every fact you keep is marked as checked, with the place you found it.
  4. No fact marked STOP appears in what you send, unless you replaced it with the source’s own wording.

Where a repeat test goes wrong

Rewording the question between tries. A different question can earn a different answer, and then you cannot tell what moved.

Comparing sentences instead of facts. Two answers can be worded nothing alike and still agree on the date. Compare the values.

Trusting the facts that matched. Case C in the lab shows why: three answers agree on 24 credits, and the handbook says 30. A tool can repeat the same mistake every time, so a match only tells you the fact is steady, not that it is right.

Sources (1)