Tutorial
BeginnerCheck That a Person Really Reviews Every AI-Assisted Decision About a Member
List every place an AI tool scores, ranks or flags a member or applicant, mark which decisions matter, and check that a trained person reviews every case before it applies, can change the outcome, sees the data, and leaves a record.
Time needed: About 30 minutes for one program
Before you start:
- A list of the tools your association uses that score, rank or flag people, or the staff who know
- The written rules for the program the decision belongs to, such as scholarship or membership criteria
AI Literacy Governance Membership Member Services
“A person reviews it” is easy to write in a program policy and easy to hollow out in practice. The committee reads only the top ten of an AI ranking. Staff can see a flag but have no power to clear it. The check happens after the decline email has gone out. Here is a short check for each place an AI tool helps decide something about a member or applicant.
Try it now: the AI-assisted decision review lab takes each decision an AI tool helps make about a member or applicant and asks who reviews it, when, and whether they can change it. It marks each decision PASS, FIX or STOP and builds a plain-text record to keep with the program rules. The Riverbend Association of Dental Hygienists and its decisions are invented for the lab.
The model we borrow
The UK’s data regulator, the Information Commissioner’s Office, has draft guidance on automated decision-making, updated March 31, 2026. It applies UK law, not US law, and it is still a draft, so its wording may change. We use it here as a model of what a real review looks like, not as a rule for associations.
Its core test: “However, for human involvement to be “meaningful” in the context of the UK GDPR, you must ensure it is active and not just a token gesture.” It then lists what the reviewer needs. They should “have the ability to influence the outcome” and “have discretion and authority to alter the decision”. They should “be suitably trained and qualified to understand the system’s logic, outputs, limitations, and risks” and “take into account the relevant data and factors on which the decision was based.”
Two more lines close common loopholes. On sampling: “Using ad hoc spot-checking isn’t sufficient because some automated decisions won’t receive a check and therefore don’t have meaningful human involvement.” On who counts: “A human merely designing or building an automated system does not count as meaningful human involvement.”
Check each decision
- List every AI-assisted decision about a person. One line each, in plain words: what the tool does and what happens next. “Ranks scholarship applications for the committee” is a line. “Uses AI” is not. Check vendor features labeled scoring, auto-approve or flag.
- Mark each one significant or minor. In the lab, a decision is significant if it affects a person’s access to membership, money, a credential, an award or participation. That line is a teaching choice. Newsletter topic suggestions are minor; a membership decline is not.
- Ask who reviews, and when. For each significant decision, a named person should review every case, not a sample. The timing matters: the guidance says to “ensure that the involvement comes before you apply the decision to a person and at a time you can still change any recommendation”.
- Check the reviewer’s power and view. Can they change the outcome without asking permission? Do they see the application or record, not just the tool’s score? Have they been told what the tool tends to get wrong?
- Give the member a way to ask. The person affected should be able to ask for a human review. This step comes from the federal model in our article on AI-assisted decisions about members; including it here is a teaching choice.
- Keep a record. The guidance is direct: “You should keep a record of how the human was involved in the decision.” Note who reviewed each case, when, and whether they changed it.
- Run the lab and fix what it finds. Clear every STOP before the tool is used again, and schedule every FIX before the next cycle.
Worked example: five decisions at one association
Priya Natarajan, the invented membership director, lists five decisions with Tom Brennan, the scholarship committee chair.
The committee reads only the top ten scholarship applications, so the lab says: “Decision 1: spot-checks leave some people with no review. Review every case, or stop using the tool for this decision.” Membership declines go out with no one looking. Staff review every CE audit pick but cannot overrule it: “Decision 3: the reviewer cannot change the outcome, so the review is a token gesture. Give the reviewer authority to change it.” The award shortlist reviewer sees only scores, a FIX.
The lab loads six cases:
- A, clean: PASS.
- B, seeded gaps: the five decisions above. STOP.
- C, review after the fact: community posts hidden first, reviewed later. STOP.
- D, no way to ask for review: otherwise sound. FIX.
- E, untrained reviewer, no record: FIX, with the training gap reported first.
- F, empty: no decisions listed. STOP.
As the page notes, the way it splits gaps into STOP and FIX, and the order it checks them in, were chosen for teaching. Neither is a legal standard.
Check your result
- Every AI-assisted decision about a person is on the list.
- Every significant decision has a named reviewer who sees every case before it applies.
- Each reviewer can change the outcome and sees the underlying record.
- Members know how to ask for a human review.
- The record is kept with the program rules.
Common mistakes
Counting setup as review. Configuring the tool is not reviewing its answers.
Reviewing only the top of the list. The people most hurt by a bad ranking are the ones nobody reads.
Reviewing after the decision lands. By then you are undoing harm, not preventing it.