Article
BeginnerAn AI Tool Drafted the Exam Question: Taking One Item From First Draft to a Live Certification or CE Exam
One AI-drafted multiple-choice question followed in order, from whether AI drafting is allowed at all to the pretest and the final keep, revise or remove decision, using accreditation guidance, the 2014 testing Standards and two certifying boards' 2026 policies.
Certification Continuing Education Committee Trust Research
A volunteer item writer for a professional society’s certification exam has used an AI tool to draft a multiple-choice question, and the society’s education team wants to do the same for its continuing education quizzes. This is a teaching example, not a case study. Before that draft reaches a candidate, the exam manager has to settle whether AI drafting is allowed, which reviews the question must pass, and who makes the final call.
We read six documents on that path. One, a 2024 systematic review in BMC Medical Education, covered eight studies from 2023, and in every one the AI produced faulty questions judged inappropriate for medical exams. None of the six measures how associations use AI to draft certification or CE questions or how those questions fare on live association exams; the measured evidence is 2023 medical questions rated by experts and never answered by candidates.
1. Decide whether AI may draft items at all
Two certifying boards answered this in opposite ways. In a February 2026 column in the ABR’s newsletter, the American Board of Radiology’s leadership calls AI for writing assessment content very problematic, and its volunteers must agree not to use generative AI when they write exam questions or items for its Online Longitudinal Assessment. Literature searches stay allowed, to research or confirm the elements of the tested concepts.
The American Board of Genetic Counseling, a nonprofit certifying board, uses an AI-assisted drafting tool that, in its FAQ (undated, 2026), serves as a tool to assist subject matter experts. The same FAQ says examination questions are never authored solely by AI systems.
Decision point: the program picks one position and tells item writers in writing before the next drafting cycle.
2. Find which side of the firewall the question sits on
The NCCA guidance on AI in certification programs, version 1.1, released April 2, 2025, gives certification programs a framework for keeping NCCA accreditation; it does not reach an education team’s CE quiz. What matters to the education desk is the guidance’s statement that with or without AI, a firewall must be maintained between education/training and certification. Among the things AI used for training content must not reach, it lists confidential examination content, including non-public test items, scoring rubrics.
Decision point: the tool that drafts CE quiz questions never touches the certification item bank, and certification staff confirm it has not seen a live item.
3. Build the prompt from the content outline, not the item bank
The 2014 Standards say the content domain to be covered by a credentialing test should be defined clearly and justified. That outline, plus a public reference, is what a prompt can carry. The review counted three studies that did not base their prompt on a valid medical reference, such as an approved syllabus.
For accredited programs, NCCA’s confidentiality guidance asks for safeguards to prevent confidential information from being incorporated into AI training data, prompts, or external AI models, and ABGC’s FAQ states that its exam content remains protected and is not used to train publicly available or external AI models. Live items, answer keys and candidate records stay out, and the site’s guide to sorting member data before it goes into an AI tool covers the rest of that sorting.
4. Put the draft in front of subject-matter experts
NCCA’s guidance on exam development says SMEs must review all AI-generated content to ensure accuracy, fairness, and compliance with psychometric standards. The 2014 Standards describe a review built on empirical analyses and/or the use of expert judges.
The review’s own examples show what those experts catch: an AI-generated question in which two of the options were both correct, and another that a board-certified specialist deemed as low complexity for a standard exam. In one 2023 dermatology study the review summarizes, 22 percent of the generated questions were unclear.
The closest head-to-head test is a PLOS ONE study led by researchers at the University of Hong Kong, in which 50 AI-drafted and 50 staff-written questions from the same two textbooks were rated blind by five assessors in 2023. In that 2023 rating the AI questions scored lower only on relevance to clinical context, 7.56 against 7.88 on a scale of 0 to 10, with no significant difference in other domains of question quality and a wider range of scores for the AI drafts. The five assessors were also the study’s authors.
In the exam manager’s case, both reviewers find that two of the four options can be defended, so the draft goes back to the item writer with a note.
Decision point: a question with two defensible answers, a disputed key or a stem that misses practice is rewritten or dropped.
5. Read the options the way a test-wise candidate would
The review’s background section names common item-writing flaws, including the pattern where the longest answer is correct and the use of absolute terms, since students recognize that absolute terms usually render a statement false. Its rule for the wrong options is that the distractors should be plausible to those without detailed knowledge of the subject, which makes guessing harder. Any option a candidate could rule out on its form alone gets rewritten.
6. Run a sensitivity review before the item counts
The 2014 Standards say expert and sensitivity reviews can serve to guard against construct-irrelevant language and images, and that such reviews should be conducted before a test becomes operational. ABGC’s FAQ adds that AI technologies may introduce or amplify bias if not carefully managed.
7. Settle the intellectual property question
NCCA asks accredited programs for reasonable precautions against AI content that infringes others’ intellectual property or leaks into public view. In its 2026 column, the ABR’s board notes that many popular tools draw on sources that may be protected by copyright, and that questions and other material generated by AI are generally not protected by copyright. How a program handles ownership of AI-assisted items is a question for its own counsel.
8. Record what the AI did and who reviewed it
NCCA’s exam development section says programs must document AI’s role, establish oversight protocols, and accredited programs disclose AI use to the NCCA through initial applications and later filings. The 2014 Standards already expect that the procedures used to develop, review, and try out items and to select items from the item pool should be documented, including who the expert judges were and how they were trained.
Decision point: every item record gains two fields, one saying whether AI drafted it and one naming the reviewers who changed it.
9. Pretest, read the statistics, then let a person decide
Expert ratings cannot show how a question behaves with candidates. The systematic review found that none of the studies tested the AI generated MCQs in a real-world setting, and the Hong Kong team wrote in 2023 that, with no students sitting the test, real-world performance could not be judged. The 2014 Standards say that when sample size permits, empirical analyses are needed to check an item’s psychometric properties. A small CE quiz may never reach that size, which leaves more of the weight on step 4.
NCCA asks programs to regularly assess AI-assisted content and psychometric characteristics, and ABGC uses psychometric analyses to find questions that perform badly. Both keep the final call human: NCCA requires that final item retention, revision, or removal decisions must be made by qualified human personnel, and ABGC that final decisions regarding question content, accuracy, relevance, and quality always rest with ABGC subject matter experts.
In the exam manager’s case, the rewritten question goes onto the next form unscored, and the committee keeps, revises or retires it once its statistics are in.
What 2023 medical studies cannot settle for an association exam
Every measured result above comes from 2023 tools writing medical questions, and the Hong Kong team used the first-available question the tool returned, unpolished. Its authors note that real exam questions are reviewed and vetted by a panel of professoriate staff, a step their test skipped. The review’s authors stop at saying LLMs may serve as a supplementary tool for writing medical examinations. We read that as a case for the review steps, not a forecast of how an AI-drafted item will perform on an association’s exam.
Sources (6)
- Institute for Credentialing Excellence, National Commission for Certifying Agencies: NCCA Guidance Document, Use of Artificial Intelligence in Certification Programs (version 1.1, released April 2, 2025; applies to NCCA-accredited certification programs only)
- AERA, APA and NCME: Standards for Educational and Psychological Testing (2014 edition, open access; prepared by their Joint Committee)
- Artsi, Sorin, Konen, Glicksberg, Nadkarni and Klang: Large language models for generating medical examinations, systematic review (BMC Medical Education, March 29, 2024; peer-reviewed systematic review of eight studies published in 2023)
- Cheung and colleagues, University of Hong Kong with Edinburgh, Singapore and Galway: AI versus human authors of medical graduate exam multiple-choice questions, a multinational prospective study (PLOS ONE, August 2023; peer-reviewed, study run February 2023, assessors were also the authors)
- American Board of Radiology: From the Board of Governors, ABR Plans to Use AI Carefully (The Beam, February 2026; board column, policy statement not a study)
- American Board of Genetic Counseling: FAQ on ABGC's Use of AI-Assisted Exam Question Development (undated, 2026; the board's description of its own process, not a study)