AssociationAI / AI Literacy
AssociationAI team Published

Article

Beginner

A Fluent AI Draft Is Not a Finished Member Email: Reading Level, Headings, Link Text and Alt Text to Check First

One peer-reviewed study of ChatGPT answers found college-level reading scores on health questions in 2023. Federal plain language and email guidance shows what to check in an AI draft before members see it. No source measures how often AI gets headings or links wrong.

AI Literacy Accessibility Plain Language Communications

Picture this illustrative case: a communications coordinator asks an AI tool to turn a board decision about dues timing into a member email and a matching web page. The draft arrives smooth and confident, so it gets a quick skim and goes into the email platform. Members then meet long paragraphs written for a reader nobody on the list resembles, section titles that are only bold text, and links that say “here.” This is a teaching example, not a case study. The skill this article teaches is reading an AI draft as text written for no one in particular, and checking four things before it reaches a member.

What one study found about reading level

Only one source we found measures how AI output reads. Toiv, Saleh and their co-authors, at Henry Ford Hospital and the University of Pennsylvania, published a peer-reviewed study in August 2024 of ChatGPT answers to gastroenterology questions. The authors write: “We established a ChatGPT-3 account on November 5, 2023, and systematically input questions into separate queries.” They scored 29 answers with three readability formulas, and their summary is blunt: “In our study, all 3 readability formulas indicated that ChatGPT-generated responses were written at a college reading level or higher for all but 1 question.”

The numbers sit in the results. For 16 dialog-based questions, the study reports “the mean FRE score was 32.6 ± 11.4, the mean FKRGL grade was 13.4 ± 1.9, and the mean SMOG index was 15.5 ± 1.6”. For 13 definition-based questions it reports “the mean FRE score was 29.7 ± 6.6, the mean FKRGL grade was 14.0 ± 1.3, and the mean SMOG index was 15.7 ± 1.2”. Their conclusion ties this to a patient-information target: “Moreover, the readability of responses usually surpassed the sixth- to eighth-grade reading level recommended for healthcare information.”

Read that with its limits. The authors say “We evaluated AI-generated responses to medical questions using only one AI-powered LLM on a limited subset of questions, which may limit the generalizability of our findings to other available LLMs.” They add that “newer versions of ChatGPT and other LLMs may offer different capabilities and performance, which could influence the outcomes of similar studies in the future”. They also note that “even medical information created by human authors tends to be written at levels surpassing current recommended standards.” So we do not say that AI writes at a college level in general. The study measured one tool, answering health questions, in late 2023. It did not measure association emails.

What it does give a communications desk is a reason to stop assuming. A model answers the prompt in front of it, and it does not know who your members are unless you tell it. Whether the draft reads at the level of your list is a question you can only answer by checking it against that list.

Plain language is not a lower target

Checking reading level can sound like permission to flatten everything. The Federal Plain Language Guidelines reject that reading: “One of the most popular plain language myths is that you have to “dumb down” your content so that everyone can read it. That’s not true.” The question is fit with the audience, and a list of association executives may take a denser sentence than a list of first-year volunteers.

Digital.gov defines plain language as “content that is clear and easy to understand” and says it “is critical to helping the public to make sense of their obligations and benefits.” It also says “The Plain Writing Act of 2010 established the requirement that content for the public is written for its specific audience.” The Plain Writing Act, as Digital.gov describes it, concerns federal content, and we cite the page as a reference for good practice. If you want the ordinary plain-language check on a notice, our tutorial on the plain-language notice check walks through it.

Headings that only look like headings

A fluent draft usually arrives with a tidy shape, which makes it easy to assume the structure is real. The Federal Plain Language Guidelines’ page on useful headings makes the point from the reader’s side: “Well organized writing will still be difficult for users to follow if they can’t see how it’s organized.” On the question of structure itself, Digital.gov says “Use heading levels consistently and appropriately to create hierarchy.”

Here the check is concrete. Look at what the draft calls a heading and ask whether your page or email template treats it as one, and whether the levels step down in order instead of all sitting at one weight. In our observation, pasting a draft into a platform may turn a section title into ordinary bold text without any warning.

Digital.gov’s guidance on links is specific: “Link text should be descriptive and contain all the information the user needs to decide if they should click on it.” A draft that ends with “Read more here” fails that test as written, whoever wrote it. You can run the check in seconds by reading only the linked words, out of context, the way a person scanning a list of links would.

Alt text, in one line

A picture an AI tool describes for you still needs a person to read the description, because Section508.gov says “Manually check that all alt text is entered correctly. Microsoft auto-generates alt text for images, but it should not be used unless it has been verified to be correct.” Our posts on checking captions and alt text before publishing and on who checks AI captions and alt text cover how to do it.

What we can and cannot say about how often

Nothing we found measures how often an AI draft skips heading levels, writes vague link text or invents alt text. The reading-level study is the only measurement of AI output among our sources, and it covers one tool and health questions. What the sources do cover is how one tool’s answers scored on reading level and what good headings, links and alt text look like when a person checks them. The four checks apply to any draft, whoever or whatever wrote it, and the AI part is a reason to run them, not proof that they will fail.

The sources also come from a particular place. The Section508.gov page is guidance for U.S. federal staff, and it states that “Federal government electronic content must meet Section 508 Standards, which incorporates Web Content Accessibility Guidelines (WCAG) version 2.0 A and AA criteria.” That is a requirement for federal content. We use Section508.gov as a federal-agency reference for good habits, and it is not a requirement for readers of this article. The page also concerns federal email in Microsoft Outlook, so we do not generalize it to other tools.

A routine for the desk

Our recommended practice is to treat every AI draft as unfinished until four things have been looked at by a person: whether the reading level suits the actual list, whether the headings are real structure in the final template, whether each link makes sense when read alone, and whether every image has alt text someone has read. We think the order matters less than doing all four before the send, and by our estimate the check takes minutes on a member email. The companion tutorial, check an AI draft for reading level, headings and link text, has a browser tool that runs the mechanical parts on a pasted draft and a kit with a checklist you can adapt.

Sources (7)