It’s widely accepted that AI models often struggle with accuracy. Developers face ongoing challenges such as hallucinations and the tendency to stick to incorrect information.
Because usage varies greatly among individuals, it’s difficult to provide precise percentages regarding AI accuracy. However, a research team claims to have gathered specific statistics.
Recently, the Tow Center for Digital Journalism examined eight different AI search engines, including ChatGPT Search and Perplexity. They evaluated each engine’s accuracy and noted how often the tools failed to provide answers.
For their study, the researchers selected 200 news articles from 20 different publishers, ensuring that each article appeared in the top three results of a Google search when using a quoted excerpt.
They then conducted the same search using each AI tool and assessed how accurately each one cited the article, the news organization, and the URL.
After conducting their evaluations, the researchers categorized each search result by its accuracy, ranging from completely correct to completely incorrect.
The findings indicated that, aside from two versions of Perplexity, the AI tools generally did not perform well, with a 60% inaccuracy rate. Additionally, the AI’s confidence in these incorrect results further complicated the situation.
This research is interesting because it provides clear evidence of something we’ve recognized for some time: large language models (LLMs) can be very deceptive.
They often state their claims with strong confidence, insisting that what they say is accurate, even when it isn’t. This can lead to disputes or the creation of more false information when they are questioned.
In a 2023 article, Ted Gioia from The Honest Broker highlighted many instances where ChatGPT provided responses that were confidently incorrect. While some of these examples came from challenging questions, many were simply general inquiries.
Gioia humorously remarked that if he believed even half of what he heard about ChatGPT, he could let it run The Honest Broker while he relaxed on the beach with a drink, searching for his lost shaker of salt.
Even when ChatGPT acknowledged its mistakes, it often followed up with additional false information. It seems that the AI is designed to respond to every question, regardless of accuracy.
The researchers found that ChatGPT Search was the only AI tool that answered all 200 queries, but it only had a 28% accuracy rate, meaning it was completely wrong 57% of the time.
Surprisingly, ChatGPT isn’t the worst option available. Both versions of X’s Grok AI performed very poorly, with Grok-3 Search being 94% inaccurate. Microsoft’s Copilot didn’t fare much better, as it refused to answer 104 out of 200 questions.
Of the 96 queries that Copilot did respond to, only 16 were completely correct, while 14 were partially accurate. The remaining 66 responses were completely wrong, resulting in an overall inaccuracy rate of about 70%.
One of the most surprising aspects of this situation is that the companies behind these AI tools are not open about their accuracy issues while charging users between $20 and $200 per month for access.
Additionally, Perplexity Pro ($20/month) and Grok-3 Search ($40/month) had slightly better accuracy than their free versions, but they also had much higher error rates. This raises concerns about transparency.
On the other hand, not everyone shares this view. The reporter mentioned that after trying ChatGPT Search, he might stop using Google altogether. He praised the tool for being quick, knowledgeable, and accurate, along with having a clean interface without ads.
You can find more information in the Tow Center’s paper published in the Columbia Journalism Review, and we would love to hear your thoughts on the findings.
Posts You May Like