Article
BeginnerBefore the Committee Runs Entries Through an AI Detector: Abstracts, Award Essays and Scholarship Applications
Why a detector score should not remove a member's abstract, award essay or scholarship application, what three published policies say, and a review process that holds up without one.
Committee Trust Awards Scholarships Peer Review
An awards committee is two weeks from scoring this year’s scholarship applications, award essays and abstracts for the annual meeting, and one volunteer proposes running every entry through an AI detector and setting aside anything that scores high. This is a teaching example, not a case study. The real decision is whether a number from a tool gets to remove a member’s entry before anyone reads it.
We read three academic tests of detectors and three published policies from a professional society, a member association’s journal and a research foundation. No source among them measures how often associations run detectors on submissions or how many members have been flagged; the false-flag evidence comes from tests on student essays, TOEFL essays, translated documents and journal abstracts.
Why a high score cannot carry the decision
The sharpest finding concerns members who write in English as a second language. An article in Patterns by five Stanford researchers summarising their tests describes running seven widely used GPT detectors on 91 TOEFL (Test of English as a Foreign Language) essays from a Chinese forum. In those 2023 tests the detectors labelled more than half of the TOEFL essays AI-generated, an average false-positive rate of 61.3 percent, even though they accurately classified the US student essays. 97.8 percent of the TOEFL essays were flagged by at least one of the seven detectors in that 2023 work. The authors explain that most GPT detectors use text perplexity to detect AI-generated text, which might inadvertently penalize non-native writers.
From there the authors argue that practitioners should exercise caution when using low perplexity as an indicator of AI-generated text. They also argue that academic conferences or journals prohibiting use of GPT may penalize researchers from non-English-speaking countries, and that detectors might compel non-native writers to use GPT more to evade detection.
The broader false-flag picture comes from two other teams. In a 2023 test by researchers at universities in Europe and Mexico, six of the 14 detection tools tested generated false positives, with the risk increasing dramatically for machine-translated texts. Fairness requires the counterweight: those same 2023 tests reached 96 percent overall accuracy on human-written English documents. The authors still warn that translation tools leave L2 students (and researchers) at risk of being falsely accused, and they note that the results from the tools can vary when the same material is tested at a different time.
The third team, from hospital neurosurgery departments in Türkiye and university medical faculties, ran three detectors in June 2024 on neurosurgery journal abstracts and introductions written before ChatGPT. In those 2024 tests one detector scored 30.4 percent of the human-written texts above 50 percent, while another did not score any original article above that line. The authors add that these detectors often produce differing results when analyzing the same content and that there is no clear consensus on a specific indicator value for detecting AI-generated text.
The committee’s worry is fair, too. The same neurosurgery paper warns that fake abstracts and publications can be presented at conferences, scientific meetings, or become published articles. Detectors also miss: the European team estimated in 2023 that approximately 50 percent of AI-generated texts that undergo some obfuscation would likely pass as human. A score can accuse an honest writer and clear machine text, and both studies reach the same rule: detector reports cannot be used as the only basis for an accusation, and should not be the sole basis for taking negative actions against academics.
What three published policies say
ASAE’s policy page for its Individual Honors applications (undated, read October 2026) prohibits AI tools in the preparation and submission of scholarship, grant, or award applications to ASAE, listing text generators, paraphrasers, summarizers, or translators. It says ASAE may use various methods to detect the use of AI tools, such as plagiarism checkers, linguistic analysis, and human review, and it requires applicants to sign a declaration stating that they have not used any AI tools in their applications.
The Journal of Singing, published by the National Association of Teachers of Singing, sets out its own approach. Because automated AI-detection methods are imperfect and can misclassify legitimate writing, the journal will not use them, and when a concern comes up in review the journal will request clarification about the author’s process. It may ask for version history, tracked changes, dated drafts, and it states that disclosure alone will never be cause for manuscript rejection.
On translation the two policies differ. ASAE’s page lists translators among the barred tools, while the journal says any translated submission must include the name of a human who vouches for the accuracy of the translation.
The Humboldt Foundation, which funds research fellowships, judges that preventing misuse by means of a detection of AI-generated texts is neither expedient nor feasible. Proposals sent to it are evaluated by subject specialists, especially concerning their originality and innovative strength. The Foundation’s view, based on its own tests, which are unpublished, is that current AI systems are not able to independently develop a convincing application proposal.
A review process that works without a score
The European team’s advice that a prevention-focused approach needs to be prioritised over a detection one points to the call itself, written before any entry arrives. Say which kinds of AI help are allowed, whether translation help counts, and what a submitter should disclose; the journal’s own shorthand, when in doubt, disclose use, fits on an entry form.
Then score entries on substance. Reviewers who know the field judge originality, evidence and fit, and the committee weighs what else it already collects, such as references or a record of service, much as the Humboldt Foundation counts other, equally important criteria (including career history and previous scientific performance) based on other documents.
When a reviewer has a real concern, the next step is a question rather than a removal. Ask the submitter how the piece was written and, where it helps, for an earlier draft. Frame the request the way the journal does, as one intended to clarify your process, then note the reply.
Finally, keep entries out of online checkers and chatbots. The journal flags that online AI checkers present legitimate data privacy concerns, and the site’s post on which member records can go into an AI tool at all covers the rest.
One scholarship essay, handled both ways
In the same scenario, a member drafts her scholarship essay in her first language, runs it through a translation tool and edits the English herself. Under the shortcut, a high detector score sets the essay aside and nobody reads it, and she never learns why her application went nowhere.
Under the process, the call allowed translation help with disclosure, so she wrote one line saying how she worked. Two reviewers scored the essay on its argument and her service record. One found the voice uneven in places, so the committee asked for the earlier draft; she sent the version in her first language, a committee member who reads that language compared the two, and the essay was scored like every other entry.
What the test results cannot tell a committee
Every figure above comes from tools tested in 2023 or 2024, on essays and abstracts rather than association entries, and none of the three teams tested today’s versions. The neurosurgery authors allow that detectors may serve as supplementary tools in peer review or abstract evaluation while warning that they often misclassify texts. A committee that keeps one in the room should let it prompt a question, never decide an entry.
Sources (6)
- Liang, Yuksekgonul, Mao, Wu and Zou, Stanford University: GPT detectors are biased against non-native English writers (Patterns, July 10, 2023; an article in Patterns by five Stanford researchers summarising their tests)
- Weber-Wulff and colleagues: Testing of detection tools for AI-generated text (International Journal for Educational Integrity, December 25, 2023; open-access original article, tests run March to May 2023, English-language texts only)
- Erol and colleagues: Can we trust academic AI detective? Accuracy and limitations of AI-output detectors (Acta Neurochirurgica, August 7, 2025; research article, detectors run in June 2024)
- ASAE: Use of AI Tools (ASAE's policy page for its Individual Honors applications, undated, read October 2026)
- Alexander von Humboldt Foundation: The use of generative AI in the Alexander von Humboldt Foundation's selection area (foundation statement, as of December 2025)
- National Association of Teachers of Singing, Journal of Singing: Generative AI and AI-Assisted Technologies Policy (the journal's own policy, revised January 22, 2026)