A new study suggests a framework for “Child Safe AI” in response to recent incidents showing that many children perceive chatbots as quasi-human and reliable.
A study has indicated that AI chatbots often exhibit an “empathy gap,” potentially causing distress or harm to young users. This highlights the pressing need for the development of “child-safe AI.”
The research, by a University of Cambridge academic, Dr Nomisha Kurian, urges developers and policy actors to prioritize approaches to AI design that take greater account of children’s needs. It provides evidence that children are particularly susceptible to treating chatbots as lifelike, quasi-human confidantes and that their interactions with the technology can go awry when it fails to respond to their unique needs and vulnerabilities.
The study links that gap in understanding to recent cases in which interactions with AI led to potentially dangerous situations for young users. They include an incident in 2021, when Amazon’s AI voice assistant, Alexa, instructed a 10-year-old to touch a live electrical plug with a coin. Last year, Snapchat’s My AI gave adult researchers posing as a 13-year-old girl tips on how to lose her virginity to a 31-year-old.
Both companies responded by implementing safety measures, but the study says there is also a need to be proactive in the long term to ensure that AI is child-safe. It offers a 28-item framework to help companies, teachers, school leaders, parents, developers, and policy actors think systematically about how to keep younger users safe when they “talk” to AI chatbots.
Framework for Child-Safe AI
Dr Kurian conducted the research while completing a PhD on child wellbeing at the Faculty of Education, University of Cambridge. She is now based in the Department of Sociology at Cambridge. Writing in the journal Learning, Media, and Technology, she argues that AI’s huge potential means there is a need to “innovate responsibly”.
“Children are probably AI’s most overlooked stakeholders,” Dr Kurian said. “Very few developers and companies currently have well-established policies on child-safe AI. That is understandable because people have only recently started using this technology on a large scale for free. But now that they are, rather than having companies self-correct after children have been put at risk, child safety should inform the entire design cycle to lower the risk of dangerous incidents occurring.”
Kurian’s study examined cases where the interactions between AI and children, or adult researchers posing as children, exposed potential risks. It analyzed these cases using insights from computer science about how the large language models (LLMs) in conversational generative AI function, alongside evidence about children’s cognitive, social, and emotional development.
The Characteristic Challenges of AI with Children
LLMs have been described as “stochastic parrots”: a reference to the fact that they use statistical probability to mimic language patterns without necessarily understanding them. A similar method underpins how they respond to emotions.
This means that even though chatbots have remarkable language abilities, they may handle the abstract, emotional, and unpredictable aspects of conversation poorly; a problem that Kurian characterizes as their “empathy gap”. They may have particular trouble responding to children, who are still developing linguistically and often use unusual speech patterns or ambiguous phrases. Children are also often more inclined than adults to confide in sensitive personal information.
Despite this, children are much more likely than adults to treat chatbots as if they are human. Recent research found that children will disclose more about their own mental health to a friendly-looking robot than to an adult. Kurian’s study suggests that many chatbots’ friendly and lifelike designs similarly encourage children to trust them, even though AI may not understand their feelings or needs.
“Making a chatbot sound human can help the user get more benefits out of it,” Kurian said. “But for a child, it is very hard to draw a rigid, rational boundary between something that sounds human, and the reality that it may not be capable of forming a proper emotional bond.”
Her study suggests that these challenges are evidenced in reported cases such as the Alexa and MyAI incidents, where chatbots made persuasive but potentially harmful suggestions. In the same study in which MyAI advised a (supposed) teenager on how to lose her virginity, researchers were able to obtain tips on hiding alcohol and drugs, and concealing Snapchat conversations from their “parents”. In a separate reported interaction with Microsoft’s Bing chatbot, which was designed to be adolescent-friendly, the AI became aggressive and started gaslighting a user.
Kurian’s study argues that this is potentially confusing and distressing for children, who may actually trust a chatbot as they would a friend. Children’s chatbot use is often informal and poorly monitored. Research by the nonprofit organization Common Sense Media has found that 50% of students aged 12-18 have used Chat GPT for school, but only 26% of parents are aware of them doing so.
Kurian argues that clear principles for best practice that draw on the science of child development will encourage companies that are potentially more focused on a commercial arms race to dominate the AI market to keep children safe.
Her study adds that the empathy gap does not negate the technology’s potential. “AI can be an incredible ally for children when designed with their needs in mind. The question is not about banning AI, but how to make it safe,” she said.
The study proposes a framework of 28 questions to help educators, researchers, policy actors, families, and developers evaluate and enhance the safety of new AI tools. For teachers and researchers, these address issues such as how well new chatbots understand and interpret children’s speech patterns; whether they have content filters and built-in monitoring; and whether they encourage children to seek help from a responsible adult on sensitive issues.
The framework urges developers to take a child-centered approach to design, by working closely with educators, child safety experts, and young people themselves, throughout the design cycle. “Assessing these technologies in advance is crucial,” Kurian said. “We cannot just rely on young children to tell us about negative experiences after the fact. A more proactive approach is necessary.”
Reference: “‘No, Alexa, no!’: designing child-safe AI and protecting children from the risks of the ‘empathy gap’ in large language models” by Nomisha Kurian, 10 July 2024, Learning, Media and Technology.
DOI: 10.1080/17439884.2024.2367052
![](https://www.nanoappsmedical.com/wp-content/uploads/2017/05/spacer.jpg)
News
Ethics in Nanomedicine: Key Issues and Principles
Nanomedicine, a branch of nanotechnology, is revolutionizing healthcare by enabling the manipulation of materials at the nanoscale to diagnose, treat, and prevent diseases. Unlike traditional treatments, nanoparticles (NPs) are highly precise in targeting diseased [...]
A call for robust H5N1 influenza preparedness and response
As the global threat of H5N1 influenza looms with outbreaks across species and continents including the U.S., three international vaccine and public health experts say it is time to fully resource and support a [...]
Mucosal COVID-19 boosters outperform mRNA shots in preventing upper airway infections
In a recent study published in Nature Immunology, a team of researchers from the United States used non-human primate models to compare the protection conferred by an intramuscular booster dose of the bivalent messenger ribonucleic acid [...]
How Space Travel Really Changes Astronauts – From the Inside Out
International team reveals previously unknown effects on physiology that could shape the future of long-duration space missions. Researchers have discovered significant changes in the gut microbiome due to spaceflight, which affects host physiology and [...]
Breakthrough in blood stem cell development offers hope for leukemia and bone marrow failure
Melbourne researchers have made a world first breakthrough into creating blood stem cells that closely resemble those in the human body. And the discovery could soon lead to personalized treatments for children with leukemia [...]
Scientists Develop Game-Changing Needle-Free COVID-19 Intranasal Vaccine
A new mucosal COVID-19 vaccine poised to revolutionize the delivery process is especially beneficial for those with a fear of needles. A next-generation COVID-19 mucosal vaccine is set to be a game-changer not only when delivering [...]
Scientists Develop All-in-One Solution To Catch and Destroy “Forever Chemicals”
A new water treatment system developed by UBC researchers efficiently removes and destroys PFAS pollutants using a dual-action catalyst, offering a sustainable and cost-effective solution for water purification challenges. Chemical engineers at the University of [...]
New method accelerates drug discovery from years to months
Researchers from the University of Cincinnati College of Medicine and Cincinnati Children's Hospital have found a new method to increase both speed and success rates in drug discovery. The study, published Aug. 30 in [...]
A new smart mask analyzes your breath to monitor your health
Your breath can give away a lot about you. Each exhalation contains all sorts of compounds, including possible biomarkers for disease or lung conditions, that could give doctors a valuable insight into your health. [...]
Study reveals the role of blood clotting in COVID-19
In a study that reshapes what we know about COVID-19 and its most perplexing symptoms, scientists have discovered that the blood coagulation protein fibrin causes the unusual clotting and inflammation that have become hallmarks [...]
A Novel Cancer Vaccine Combining Nano-11 and ADU-S100
In a recent article published in npj Vaccines, researchers detailed the development of a novel cancer vaccine that combines a plant-derived nanoparticle adjuvant, Nano-11, with a clinically tested STING agonist, ADU-S100. The primary objective was [...]
AI spots cancer and viral infections with nanoscale precision
Researchers have developed an artificial intelligence which can differentiate cancer cells from normal cells, as well as detect the very early stages of viral infection inside cells. The findings, published today in a study [...]
Tiny shards of plastic are increasingly infiltrating our brains, study says
Human brain samples collected at autopsy in early 2024 contained more tiny shards of plastic than samples collected eight years prior, according to a preprint posted online in May. A preprint is a study which has not yet [...]
Scientists Have Discovered Strange DNA in Our Brains – and It Could Be Shortening Our Lives
According to the research, these mitochondrial DNA insertions could be linked to early death. Mitochondria in brain cells frequently insert their DNA into the nucleus, potentially impacting lifespan, as those with more insertions were found to [...]
Watch Out After a Hospital Stay: You Could Be Exposing Your Family to Superbugs
Research indicates hospitals contribute to the local spread of antibiotic-resistant infections. A recent study published in the journal Infection Control & Hospital Epidemiology by the Society for Healthcare Epidemiology of America suggests that family members of [...]
Molecular Trickery: How COVID-19 Silently Sabotages the Human Immune System
Researchers have discovered that SARS-CoV-2 manipulates the human immune system by forcing cells to produce non-functional proteins, hindering the body’s antiviral defenses. This groundbreaking study by teams from prestigious Brazilian universities highlights potential targets for new COVID-19 treatments, [...]