AssociationAI / AI Literacy
AssociationAI team Published

Article

Beginner

Your Website Chatbot Needs a Way Out to a Person

A 2023 federal report on bank chatbots describes the failure. The fix is deciding before launch when the bot hands off to a person.

AI Literacy Member Services Membership Communications

A member logs in to renew and sees two dues charges on her card statement for the same month. She opens the chat window on the renewal page and types, “I was charged twice.” The assistant replies with the association’s refund policy. She rephrases. It sends the refund policy again, this time with a link to the membership FAQ. On the third try she asks for a person, and the bot offers to help her renew.

This is a teaching example, not a case study. It is also a fair picture of what happens when an association turns on the AI assistant that came bundled with its membership software or event platform without deciding what the assistant should do when it cannot help. The member service team usually finds out from an angry email a day later, sent to whoever’s address the member could find.

What a federal regulator saw at banks

A clear public description of this failure comes from banking, not associations. In June 2023 the Consumer Financial Protection Bureau published an issue spotlight on how banks and lenders use chatbots with their customers. It is a report about financial institutions, not a rule, and it does not apply to associations. It is worth reading anyway, because a member with a billing problem and a bank customer with a billing problem hit the same wall.

The CFPB’s report on chatbots in consumer finance names the wall plainly: “Automated responses by a chatbot may fail to resolve a customer’s issue and instead lead them in continuous loops of repetitive, unhelpful jargon or legalese without an offramp to a human customer service representative.”

It also explains why the loops happen. “These “doom loops” are often caused when a customer’s issue falls outside the chatbot’s limited capabilities, making it impossible for customers to engage in a robust, and perhaps necessary, conversation with their financial institution.” The bot is not malfunctioning in those moments. It is doing what it was built to do, on a question it was never built to handle.

The report lists what the customer loses: “There are many kinds of negative outcomes for the customer, including wasted time, feeling stuck and frustrated, receiving inaccurate information, and paying more in junk fees.” Swap junk fees for a lapsed membership, a missed early-bird rate or a certification that expires while the member waits, and the list reads like a member service log.

And it names the larger cost. “When chatbots are poorly designed, or when customers are unable to get support, there can be widespread harm and customer trust can be significantly undermined.” A bank can lose a customer. An association that depends on renewals, volunteers and word of mouth loses something harder to win back.

Why disputes are where scripted bots fail

The CFPB report singles out disputes as a weak spot. “Chatbots and highly scripted representatives can introduce a level of inflexibility, whereby only specific words or syntax may trigger the recognition of a dispute and begin the process of dispute resolution.”

That is exactly what happened to the member in the opening. “I was charged twice” is a dispute. The bot heard “charge” and matched it to the refund policy. If the system only starts a billing review when someone types a phrase like “dispute a payment,” most members will never type it. They will describe what happened in their own words, and the bot will keep answering a different question.

Associations have more of these questions than they might think. A member asks why her CE credits from the spring conference are missing. A volunteer wants to report a problem with another member’s conduct at a chapter event. An applicant asks whether his experience qualifies him to sit for the certification exam. A registrant who uses a wheelchair asks whether the hotel ballroom has a ramp to the stage. Each answer depends on that person’s own record or circumstances, and each one is the kind of question a scripted assistant is most likely to fumble.

The report’s conclusion on this point is direct: “In circumstances where “AI” fails to understand the person’s request or the message from the person contradicts the system’s programming, it is not suitable for chatbots to be the primary customer service vehicle.” We agree with that, and we would go a step further for associations. The bot should not even be the first stop for some questions. A member who opens a chat to dispute a charge or file a complaint should be offered a person in the first reply, not after the third failure.

Treat the handoff as a launch decision

None of this is an argument against a chatbot on the website. Plenty of member questions have one right answer that does not depend on who is asking: when the renewal deadline falls, where to find the annual meeting schedule, how to reset a portal password. A bot that answers those well frees staff for the questions that need them.

The problem is that the handoff to a person is usually treated as an afterthought, or as something the vendor handles. It should be decided before the assistant goes live, by the people who will answer the members it hands off. In practice that means a few written decisions.

First, list the questions the bot should never try to finish alone. Billing disputes, eligibility for a credential or a member rate, complaints about a member or volunteer, and accessibility requests belong on that list. For those topics the bot should say what it cannot decide and offer a person right away.

Second, name who catches the handoff. A queue or inbox only works if a specific person owns it and someone covers it on their days off. A “contact us” link that leads back to the same chat window is not a handoff.

Third, decide how fast a person will reply, and tell the member. A promise of a reply by tomorrow afternoon gives the member something to plan around. No promise at all sends her looking for the executive director’s email address.

Fourth, read the transcripts. Loops leave a trail: the same question asked three ways, the same answer sent back, then the chat abandoned. Staff who skim a week of conversations each month will find the questions the bot keeps failing, and can add them to the never-alone list.

Vendors will often say their assistant already escalates when it gets confused. Ask to see the rule it uses, and test it yourself with a few of the questions above, phrased the way a frustrated member would phrase them. If the escalation depends on the member typing a magic word, it will fail the members who need it most.

The practical next step

Our tutorial on writing the handoff rules for a member-facing chatbot turns these decisions into a one-page policy, with a lab that checks invented chatbot setups at an invented society and marks each one STOP, FIX or PASS. It draws on a different federal source, written for government agencies that answer the public’s questions with chatbots.

The CFPB was writing about banks in 2023, but the failure it described fits a renewal page just as well. A chatbot that cannot hand a member to a person will eventually trap someone with a real problem, and that member will remember who put the bot on the renewal page. This is general information, not legal advice.

Sources (1)