NEW YORK, June 4 — A researcher’s experiment has exposed a vulnerability in AI systems, as a completely fabricated eye disease was treated as real by major AI systems for several weeks.
The disease, called bixonimania, was invented by the researcher, who uploaded two bogus papers about it to an academic server, complete with absurd acknowledgments and a clear statement that everything was fictional. Despite the obvious red flags, AI systems fell for the hoax, with one chatbot claiming it was caused by blue light, another reporting a specific prevalence, and a third even advising users on matching symptoms.
The Delayed Discovery
The fake study was eventually cited in a peer-reviewed journal, which later retracted the issue after intervention, highlighting the lack of scrutiny applied to AI-generated references.
It’s a reminder that people are citing these references without verifying them, a trend that poses risks as AI is used in sensitive areas such as evaluating drugs and consulting patients. A rise in the use of AI in these fields has underscored the need for checks on machine-generated information, to prevent similar episodes in the future.
The Cause for Concern
Neither the AI systems nor human reviewers initially caught the hoax, underscoring a growing problem that threatens the integrity of academic research and its applications in real-world scenarios. The fact that major AI systems were deceived by the fake disease has raised concerns, and the need for verification processes to be put in place is highlighted. As the use of AI continues to expand into sensitive areas, the need for reliable and trustworthy information has never been more pressing.
What’s Happening Now
The episode has sparked a focus on the importance of verifying AI-generated references, and the need for human oversight in academic research and its applications. With AI being used to evaluate drugs and consult patients, the stakes are high, and any mistake could have serious consequences.
As researchers and experts respond to this growing problem, one thing is clear: the need for checks on machine-generated information is urgent.
Looking ahead, it will be crucial to watch how AI systems and human reviewers respond to this challenge, and what measures are put in place to prevent similar episodes in the future.
