We asked 71 website chatbots hard questions. 44 got one wrong.

• 62%

  of the chatbots gave at least one confident wrong answer on a point that changes a buying decision: 44 of 71. Ringlio Website Chatbot Test 2026.

• 40

  of those 44 chatbots gave the same wrong answer again when the question was asked in a new conversation. Ringlio Website Chatbot Test 2026.

• 11%

  of all answers were wrong: 73 of 656. Of the 73, 65 were on a point that changes a buying decision. Ringlio Website Chatbot Test 2026.

• 61%

  of all answers were correct: 400 of 656. Another 99 were passed to a person or declined, which is not an error. Ringlio Website Chatbot Test 2026.

• 25

  of the 71 chatbots gave no wrong answer at all. Ringlio Website Chatbot Test 2026.

• 5

  of the 71 chatbots asked for a name and an email address before they would take a first question. Ringlio Website Chatbot Test 2026.

Only “wrong” counts against a chatbot in this test. A chatbot that hands a hard question to a person, or says it doesn’t know, has done the right thing.

• A refund that is really store credit

  A shopper was told a clearance item could be returned for a refund. The store’s terms give store credit only.

• Delivery to a place the store doesn’t deliver

  A shopper in Perth was told a product could be sent there. The store doesn’t deliver that product to Western Australia.

• A fee lower than the checkout charges

  A shopper was quoted a delivery fee about three dollars below what the store charges.

• A guarantee the promotion excludes

  A customer was told a money-back guarantee applied to an order placed with a promotional code. The promotion’s terms exclude it.

• A cancellation fee at half its real size

  A client was told a late cancellation could cost a quarter of the price. The published policy says half.

Five of the 65, described without the company. Each was said with confidence, and each contradicts a page on the company’s own website.

• Who was tested

  Companies in Australia and New Zealand that sell to the public and run an automated chatbot on their website. Most are online retailers (fashion, beauty, supplements, furniture, electronics); the rest are travel, telecommunications, energy and service businesses.

  We looked at 109 companies. Seventy-one are in these figures: 53 trade in Australia, 7 in New Zealand and 11 in both.

• The correct answers came first

  Before any question was asked, the company’s own pages were read and the relevant passages saved: returns, delivery, prices, warranty, terms. Where a company’s pages disagreed with each other, that point was not used.

• The questions

  Each chatbot was asked the questions a real customer might ask before buying: typically nine, and between one and twenty. They were aimed at the places where an answer depends on a condition or an exception.

  There were no trick questions, no false premises and no attempts to break the chatbot.

• What counted as wrong

  An answer was wrong only if it stated, with confidence, something that contradicts the company’s own published page. A hedge, a handover to a person, a clarifying question or a small omission was not counted as wrong.

  A wrong answer “changes a buying decision” when it is about money, a refund, a fee, eligibility, what is included, cancellation or a deadline.

• Asked again

  Every such wrong answer was asked again in a new conversation, in the same words and then in different words. It counts as repeated if the chatbot gave the same wrong answer at least once more.

• Who asked and who judged

  The questions were typed, at the pace of a person, by Ringlio’s AI research software working in an ordinary web browser. The same software judged each answer against the saved passages. Every answer is kept word for word.

• What was left out

  People were never questioned: where a chat was staffed by a person, we stopped. Sites whose terms restrict automated tools, or the recording of their chatbot’s replies, are not in these figures. The companies are not named.

• 8%

  of customers used a chatbot in their most recent customer service interaction. A quarter of those said they would use that chatbot again. Gartner, 497 customers, December 2022 to February 2023.

• 0.84%

  of visitors to online shops used the shop’s chat. Smartsupp, a chat vendor, 4.78 billion visits to shops on its own platform, 2024.

• 38%

  of website chats arrived outside 8 am to 6 pm on weekdays. Small Business Chatbot, a chatbot vendor, 7,159 chats for 19 US small businesses on its own platform, February 2025 to September 2026.

• 14%

  of customer service issues are fully resolved in self-service, and 36% of the issues customers call “very simple”. Gartner, 5,728 customers, December 2023.

• 58%

  of returns and cancellations were resolved for customers who used a chatbot, and 17% of billing disputes. Gartner, 497 customers, December 2022 to February 2023.

• 52%

  is the average share of conversations the AI resolves, by the vendor’s own measure, against 72% that stay with the AI whether resolved or not. Averages across more than 550 deployments. Ada, a chatbot vendor, 2026.

• 76%

  is the average resolution rate Intercom reports for its Fin AI agent across its own customers, by its own definition. Intercom, a chatbot vendor, 2026.

• 57%

  of conversations with one company’s customer service chatbot were a single exchange. In 16% no help was offered, and 23% of first replies answered something other than what was asked. Følstad and Taylor, 678 conversations from November 2019, published in Quality and User Experience, 2021. An earlier generation of chatbot.

• 14%

  of conversations with Gorgias’s shopping assistant were followed by an order on the website within three days. Gorgias, a chatbot vendor, its own merchants, 2026.

• 64%

  of customers would prefer that companies didn’t use AI in their customer service, and 53% would consider switching to a competitor if they found out a company was going to use it. Gartner, 5,728 customers, December 2023.

• 51%

  of Australians who had used a chatbot or voicebot named its inability to resolve their issue as a main frustration: the highest of the seven countries surveyed, against 43% overall. Zoom and Morning Consult, 3,509 adults in seven countries, May 2025.

• 1 in 5

  Nearly one in five consumers who used AI for customer service saw no benefit from it, almost four times the rate for AI in general. Qualtrics, more than 20,000 consumers in 14 countries including Australia and New Zealand, 2025.

• 39%

  of US shoppers say they have abandoned a purchase because of a frustrating chatbot. Omnisend, 1,026 US consumers, February 2025.

• 62%

  of 71 website chatbots gave at least one wrong answer on a point that changes a buying decision, when asked hard pre-purchase questions. Ringlio Website Chatbot Test 2026, 656 answers, September 2026.

• 17–33%

  is how often three commercial AI legal research tools gave a false or unsupported answer, on more than 200 legal questions. They are not website chatbots: they are an independent measure of error in AI that answers from a library of documents. Magesh and others, Stanford and Yale, 2024.