A computer-assisted needle misses its target, puncturing the spine. A diabetic patient goes rapidly downhill after a computer recommends an incorrect insulin dosage. An ultrasound fails to diagnose an obvious heart condition that is ultimately fatal.
These are just a few examples of incidents reported to the United States’ Food and Drug Administration involving health technology assisted by artificial intelligence (AI), and Australian researchers say they are an “early warning sign” of what could happen if regulators, hospitals and patients don’t take safety seriously in the rapidly evolving field.
“This is essentially showing us that when we’re putting in AI systems, we just need to be taking the safety of these systems really seriously,” Professor Farah Magrabi said.
Her team at the Australian Institute of Health Innovation at Macquarie University this month published a review of 266 safety events involving AI-assisted technology reported to the US watchdog. The article appeared in the Journal of the American Medical Informatics Association.
Only 16 per cent actually led to patients being harmed, but two-thirds were found to have the potential to cause harm, and 4 per cent were categorised as “near-miss events” in which users intervened.
Co-author Dr David Lyell said issues arose most commonly when users failed to enter the correct data, leading to an incorrect result, or misunderstood what the AI was actually telling them when it produced a result.
For example, one patient suffering a heart attack delayed medical care because an over-the-counter electrocardiogram device – which is not capable of detecting a heart attack – told them they had “normal sinus rhythm”.
“AI isn’t the answer; it’s part of a system that needs to support the provision of healthcare. And we do need to make sure that we have the systems in place that supports its effective use to promote healthcare for people,” Lyell said.
The researchers chose to analyse cases in the US, where the implementation of AI-enabled health devices is more advanced than in Australia. The US regulator has, to date, approved 521 artificial intelligence and machine learning-enabled medical devices, with 178 of those added in 2022.
In Australia, the Therapeutic Goods Administration (TGA) does not collect data about the number of approved devices in Australia that have AI or machine-learning components, but Magrabi said the regulator was taking the issue “very, very seriously”.
“AI isn’t the answer, it’s part of a system that needs to support the provision of health care.”
David Lyell, Australian Institute of Health Innovation, Macquarie University
AI has been used in healthcare devices for decades, but an explosion in data collection, computing power and advanced algorithms has opened up new frontiers, said David Hansen, chief executive of the CSIRO’s Australian E-Health Research Centre.
Sydney-based medical device start-up EMVision is one Australian company taking advantage of these advancements to develop a portable device for diagnosing stroke without the need for an MRI.
In the development stage, the company is using an advanced algorithm and high-powered computers to simulate stroke in numerous places in the brain, building up a database of synthetic images similar to MRIs which are then compared with real-life MRI and CT results from hospital clinical trials at Royal Melbourne, Liverpool and Princess Alexandra hospitals.
“We wouldn’t be able to do what we’re doing today if we didn’t have the [high-powered computer] infrastructure for the simulation,” said head of product development Forough Khandan.
Co-founder Scott Kirkland said the intention was not to completely replace CT and MRI scans, but to diagnose stroke in the first hour when treatment is most effective. The bedside device, set to be launched in 2025, will use a “traffic light” system based on a probability algorithm to help determine what type of stroke might have occurred.
“It’s better for an algorithm to give an ‘I don’t know’ than an incorrect answer, and have the wrong treatment and or triage process followed,” Kirkland said.
Radiology is at the forefront of the rapid adoption of AI in healthcare, especially in breast cancer screening and analysis of chest X-rays.
“A couple of years ago, almost no radiologist would say they use it, now a fair percentage would say that they use it in their daily work,” said clinical radiologist and AI safety researcher Dr Lauren Oakden Rayner.
Rayner, a member of the college, said the technology had many potential benefits, but Australian regulators and clinicians needed to better understand the risks of fully autonomous systems before putting them into hospitals, clinics and homes.
“Humans are legally and morally responsible for decision-making, and it’s taking some of that out of human hands,” she said. “There’s no reason autonomous AI systems can’t exist … but they obviously have to be tested very, very tightly.”
News
False Memories Under Fire: Surprising Science Behind What We Really Recall
New research challenges the ease of implanting false memories, highlighting flaws in the influential “Lost in the Mall” study. By reexamining the data from a previous study, researchers found that many supposed false memories [...]
Born Different? Cambridge Scientists Uncover Innate Sex Differences in Brains
Cambridge researchers found that sex differences in brain structure exist from birth, with males having more white matter and females more grey matter, highlighting early neurodiversity. Research from the Autism Research Centre at the University [...]
New study shows risk factors for dementia – virus causes deposits in the brain
Research into the causes of Alzheimer's is not yet complete. Now a new study shows that head trauma can activate herpes viruses and promote the disease. Frankfurt am Main – As a neurodegenerative disease, [...]
Are Machines Truly Thinking? Modern AI Systems Have Finally Achieved Turing’s Vision
Modern AI systems have fulfilled Turing’s vision of machines that learn and converse like humans, but challenges remain. A new paper highlights concerns about energy consumption and societal inequality while calling for more robust [...]
The Surprising Link Between Smell, Sound, and Emotions
New research reveals how smell and hearing interact in the brain to drive social behavior, using mouse maternal instincts as a model. Imagine you’re at a dinner party, but you can’t smell the food [...]
Brain cells age at different rates
As our body ages, not only joints, bones and muscles wear out, but also our nervous system. Nerve cells die, are no longer fully replaced, and the brain shrinks. "Aging is the most important risk factor [...]
Long COVID Breakthrough: Spike Proteins Persist in Brain for Years
Researchers have discovered that the SARS-CoV-2 spike protein persists in the brain and skull bone marrow for years after infection, potentially leading to chronic inflammation and neurodegenerative diseases. Researchers from Helmholtz Munich and Ludwig-Maximilians-Universität (LMU) have [...]
Water-Resistant Paper Could Revolutionize Packaging and Replace Plastic
A groundbreaking study showcases the creation of sustainable hydrophobic paper, enhanced by cellulose nanofibres and peptides, presenting a biodegradable alternative to petroleum-based materials, with potential uses in packaging and biomedical devices. Researchers aimed to [...]
NIH Scientists Discover Game-Changing Antibodies Against Malaria
Novel antibodies have the potential to pave the way for the next generation of malaria interventions. Researchers at the National Institutes of Health (NIH) have identified a novel class of antibodies that target a previously unexplored region [...]
Surprising Discovery: What If Some Cancer Genes Are Actually Protecting You?
A surprising discovery reveals that a gene previously thought to accelerate esophageal cancer actually helps protect against it initially. This pivotal study could lead to better prediction and prevention strategies tailored to individual genetic [...]
The Cancer Test That Exposes What Conventional Scans Miss
Researchers at UCLA have unveiled startling findings using PSMA-PET imaging that reveal nearly half of patients diagnosed with high-risk prostate cancer might actually have metastases missed by traditional imaging methods. This revelation could profoundly affect future [...]
Pupil size in sleep reveals how memories are processed
Cornell University researchers have found that the pupil is key to understanding how, and when, the brain forms strong, long-lasting memories. By studying mice equipped with brain electrodes and tiny eye-tracking cameras, the researchers [...]
Stanford’s Vaccine Breakthrough Boosts Flu Protection Like Never Before
Stanford Medicine researchers have developed a new method for influenza vaccination that encourages a robust immune response to all four common flu subtypes, potentially increasing the vaccine’s efficacy. In laboratory tests using human tonsil [...]
Water’s Worst Nightmare: The Rise of Superhydrophobic Materials
New materials with near-perfect water repellency offer potential for self-cleaning surfaces in cars and buildings. Scientists from Karlsruhe Institute of Technology (KIT) and the Indian Institute of Technology Guwahati (IITG) have developed a surface [...]
Japanese dentists test drug to help people with missing teeth regrow new ones
Japanese dentists are testing a groundbreaking drug that could enable people with missing teeth to grow new ones, reducing the need for dentures and implants, AFP recently reported. Katsu Takahashi, head of oral surgery at [...]
An AI system has reached human level on a test for ‘general intelligence’
A new artificial intelligence (AI) model has just achieved human-level results on a test designed to measure "general intelligence." On December 20, OpenAI's o3 system scored 85% on the ARC-AGI benchmark, well above the previous AI best [...]
