Article
BeginnerAsking an AI to Answer "as Our Members" Is Not Member Research
A chatbot can produce fluent answers "as" a member group in seconds. A Stanford-led study published in Nature Machine Intelligence found those stand-in answers tend to sound like outsiders describing the group and spread across a narrower range than real people's. What that means for needs assessments, dues decisions and board decks.
AI Literacy Surveys Research Member Trust
The board meets Thursday, and the membership director at the Invented Tessalind Association of Community Pharmacists has no fresh survey on a proposed benefit for members who are losing their sight. A colleague suggests a shortcut: ask a chatbot to answer “as a member with impaired vision” and paste the replies into the deck. This is a teaching example, not a case study. The replies come back in seconds, they read smoothly, and nobody in the room could say what’s wrong with them.
A peer-reviewed study gives a clear reason to hesitate. Its answer isn’t that the chatbot writes badly. It’s that the chatbot tends to write like someone outside the group imagining what the group would say.
What the study did
The paper, Large language models that replace human participants can harmfully misportray and flatten identity groups, is a Stanford-led study published in Nature Machine Intelligence. The authors set out the idea in their abstract: “We argue analytically for why LLMs are likely to both misportray and flatten the representations of demographic groups, then empirically show this on 4 LLMs through a series of human studies with 3200 participants across 16 demographic identities.”
The human side of the comparison came from people who actually belong to each group. The paper compares model output with “responses from human participants who hold that demographic identity in the United States”, and it notes that “Participants are recruited on Prolific”. The researchers asked nine questions, and they sorted those questions by how much a person’s identity matters to the answer.
One of those sorts is the one that matters most for associations. The paper calls it R2-Relevant and defines it this way: “R2-Relevant: demographic identity is relevant but not contingent, e.g., political opinion polls, surveys on workplace harassment.” A question about whether a new benefit would help members who are losing their sight belongs in the same family. Identity doesn’t decide the answer, but it shapes it.
Finding 1: the stand-in sounds like an outsider
The paper’s first concern is misportrayal. The researchers compared each model’s answers with two kinds of human writing: what people in a group wrote about themselves, and what people outside the group wrote when asked to imitate them. A misportrayal, in the paper’s words, means the model’s responses came out “statistically significantly more like out-group imitations than in-group representations.”
For the R2-Relevant questions, the authors report this, in a sentence that covers all four models they tested: “For R2-Relevant (double the questions) we again see across all four LLMs there are misportrayals for non-binary person (32/48) and person with impaired vision (27/48), but not as much for White person (15/48); instead, we see a misportrayal for Gen Z (27/48) and woman (26/48).”
So for person with impaired vision, the paper prints 27/48. We give that figure exactly as printed and don’t recalculate it. What it tells a membership team is simple enough: on the kind of question a member survey asks, the AI stand-in for that group repeatedly read more like an outsider’s guess than like the group itself.
That pattern matters most for smaller groups. A membership team usually has the least data on exactly the members it most needs to hear from, and those are the people an AI stand-in is most tempting for.
Finding 2: the answers bunch together
The second concern is flattening. Real people in any group disagree with each other. Some love a proposal, some hate it, and plenty land in the middle. The study found the stand-ins didn’t show that spread. In the authors’ words: “We indeed find that all four models on all questions, and across nearly all four measures of diversity we use, generate responses that are flatter than that of humans (Results in Fig. 4).”
For a dues or benefits decision, a flattened answer is a quiet hazard. It can make a group look more united than it is, and it hides the members whose objection would have changed the plan. Picture a pharmacist who runs a small rural store and a pharmacist at a large hospital outpatient counter. They might answer a dues question in opposite ways, and both answers belong in the result. A stand-in that settles somewhere in between represents neither of them.
Finding 3: the fixes help, but not enough
The authors didn’t stop at the problem. They tested ways to prompt the models more carefully, and they report that they “provide inference-time techniques that we empirically demonstrate do reduce, but do not remove, these harms.” Better prompting narrows the gap. It doesn’t close it.
They also argue the limits come from how these tools are built, writing that “we show that there are two inherent limitations in the way current LLMs are trained that prevent this.” Their bottom line is short: “Overall, we urge caution in use cases where LLMs are intended to replace human participants whose identities are relevant to the task at hand.”
Why this lands on association work
Association research is full of questions where identity is relevant but not decisive: needs assessments, salary and workplace surveys, accessibility reviews, early-career programming, dues structures. That’s the R2-Relevant territory the study describes. Our reading is that an AI stand-in is weakest exactly where member research matters most, because the groups that most need a fair hearing are the ones it tends to misportray or flatten.
There’s also a trust problem. If a board deck says “what our members with impaired vision told us” and the words came from a chatbot, the deck is wrong about its own evidence. Members who later learn that will reasonably wonder what else was filled in.
Sponsors and exhibitors deserve the same honesty. If your association sells or shares “audience insight” about a professional segment, say plainly whether it came from members or from an AI imitating them.
What a stand-in can still do
None of this means the tool is useless around surveys. A chatbot can help you brainstorm questions to ask, spot confusing wording before a draft goes out, or suggest answer choices you forgot. That’s our reading, not a finding of the paper. The line to hold is between questions and answers: use the tool to sharpen what you ask members, and let members supply what they think.
Two neighboring problems are worth keeping separate. Bots that fill in a real survey link are a different risk, covered in our post on fake responses in open survey links. And when real members have written comments, summarizing them with AI raises its own questions, which we walked through in AI reading open-ended member comments.
What to tell your team
- Don’t report stand-in answers as member views. Not in a board deck, a chapter report or a sponsor prospectus.
- Label any AI persona draft. If a draft was written “as a member,” mark it as a brainstorm, not evidence.
- Protect the small groups. Where you have few responses from a group, go and ask more of those members rather than filling the gap with an imitation.
- Test before trusting. If someone wants to try stand-ins anyway, compare them against survey results you already have. Our companion tutorial shows how to check AI stand-in survey answers against results you already have.
Common mistakes
- Treating fluency as accuracy. A smooth answer can still be an outsider’s guess.
- Prompting harder and calling it fixed. The authors found better prompting reduced the harms without removing them.
- Reading a close overall result as fine for every group. Misportrayal showed up for particular groups, not uniformly.
Limitations
The study covered 3,200 US participants recruited on Prolific, 16 identities, nine questions and four models. It didn’t test association members, member surveys or any tool your team uses now, and we make no claims about current tools. Applying the findings to association work is our reading. The Invented Tessalind Association of Community Pharmacists is made up.
Sources (2)
- arXiv 2402.01908v3, the version our quotes were checked against: Angelina Wang, Jamie Morgenstern and John P. Dickerson, Large language models that replace human participants can harmfully misportray and flatten identity groups
- Nature Machine Intelligence 7 (2025), published version (DOI): Angelina Wang, Jamie Morgenstern and John P. Dickerson, Large language models that replace human participants can harmfully misportray and flatten identity groups