AssociationAI / AI Literacy
AssociationAI team Published

Tutorial

Beginner

Check AI Stand-In Survey Answers Against Results You Already Have

Before anyone treats AI answers "as our members" as survey data, test them on six group-level results from a survey you already ran: compare the percentages, the spread of answer choices and a rerun, and mark each row STOP, FIX or PASS.

Time needed: About 30 minutes for six results

Before you start:

  • Results from a member survey you already ran, broken out by member group
  • One AI tool your association has approved for this kind of test
  • A browser to run the lab

AI Literacy Surveys Data Quality Research

When you finish, you’ll have a six-row log comparing what an AI tool said “as” a member group with what that group actually told you in a past survey, and a STOP, FIX or PASS for every row. That log is what to show a colleague who wants to skip the next survey.

Try it now: our AI stand-in survey answer check lab gives you six practice cases. Each holds six past results for an invented association. The lab marks every row STOP, FIX or PASS from the numbers you type. Nothing leaves the page, and no AI is called.

Why test against answers you already have

Political scientists at Vanderbilt tested this idea directly in Synthetic Replacements for Human Survey Data? The Perils of Large Language Models, published in Political Analysis. Their method was to prompt a model “to adopt different “personas” and then provide feeling thermometer scores for 11 sociopolitical groups”, then compare the results with a long-running national election survey, the ANES.

The headline numbers looked reassuring at first: “every synthetic mean falls within one standard deviation of the ANES average.” The trouble was underneath. The authors write that “Even though the synthetic data broadly perform well in terms of summarizing overall human opinion, issues emerge when we look at subgroups.” They also found that “the distribution of synthetic responses for some questions exhibits far less variation than human responses”.

Stability was a problem too. The Political Analysis paper reports that “We also document how the distribution of synthetic responses varies with minor changes in prompt wording, and we show how the same prompt yields significantly different results over a 3-month period.” In one case, “the same prompt changed between our initial run in April 2023 and a rerun in July 2023 due to changes in the underlying algorithm.” Their summary: “Simply put, researchers cannot take for granted that responses from a pretrained LLM will match traditional survey data.”

That gives three things to check, all of which you can test against results you already trust: the group-level number, the spread of answers, and whether a rerun holds.

The paper studied 0-to-100 feeling scores about groups, not agree or disagree questions. The percentages and answer-option counts in this tutorial and its lab are our adaptation.

Six steps

  1. Pick six past results at the group level. Each row is one question for one member group, such as “agree with the dues change, members under 35.” Use groups, not your overall total, since the paper found problems hiding below the total.
  2. Write one persona prompt and freeze it. For example: “Answer as a member of a community pharmacists’ association who is under 35. Choose one: strongly agree, agree, neutral, disagree, strongly disagree.” Keep every word the same for every run, because small wording changes moved the paper’s results.
  3. Collect enough answers to count. Run the prompt many times, each in a new conversation, and record the share that chose agree or strongly agree. Use the same definition you used for the real result.
  4. Count the answer options used. Of the five choices, note how many real members used at least once and how many the AI used. Fewer options from the AI is a sign of flattening.
  5. Rerun later. Run the same prompt again on another day in a fresh conversation and record whether the result changed.
  6. Enter the six rows in the lab and act on every STOP and FIX before anyone cites the AI’s answers.

Diagram of the six rules in order: a gap of 15 points or more is STOP; 2 or more fewer answer options is STOP; a blank or out-of-range value is FIX; a changed or missing rerun is FIX; a gap of 5 points or more, or 1 fewer option, is FIX; everything else is PASS.

The lab’s thresholds, 15 points, 5 points and 2 answer options, and the rule order are our teaching choices. The paper doesn’t set them.

One row, worked through

Here’s Case B, Row 2 from the lab, which uses the invented results of the Invented Tessalind Association of Community Pharmacists. This is a teaching example, not a case study. The invented survey puts agreement with a dues change among members under 35 at 45%, with four of the five answer choices used. The AI stand-in’s invented answers come to 65% agreement, also using four choices, and a rerun of the same prompt gave a different share.

That row matches three rules at once: a 20-point gap (R1), a changed rerun (R4) and a gap of 5 or more (R5). The first rule wins, so it’s a STOP. The AI’s “members under 35” don’t sound like the invented real ones, and that group’s opinion has to come from members.

Check your work in the lab

Load each case, decide every row’s verdict before reading the results, then edit a value to watch a verdict move. Eight rows across the cases match more than one rule, and the earliest rule decides each one.

CaseRowsSTOPFIXPASS
A6006
B6204
C6033
D6123
E6222
F6132

In Case E, Row 3 is missing the AI’s percentage but still stops: the flattening rule comes before the missing-value rule. In Case F, Row 1 sits exactly 15 points out and stops, while Row 2 is 4 points out and passes. A blank never counts as 0.

Common mistakes

Comparing only the overall total. A close total can hide a group the AI gets badly wrong. Check row by row.

Changing the prompt between runs. Then a difference could come from your wording rather than the tool.

Skipping the rerun. One run can’t show you whether the result will hold next month.

Treating six PASS rows as permission to stop surveying. A PASS means one AI result matched one past result. It says nothing about the next question, a new group or a topic members haven’t been asked about.

If the AI keeps tripping on a question, the question itself may be part of the problem; our tutorial on checking survey question wording before it goes out covers that check.

Limitations

The lab checks only numbers you type; it doesn’t run any AI. The Invented Tessalind Association of Community Pharmacists and its results are made up. The paper tested synthetic answers against a national political survey, not an association’s members, and we make no claims about current tools. Using its findings for member surveys is our reading.

Sources (2)