One question has relentlessly followed ChatGPT in its trajectory to superstar status in the field of artificial intelligence: Has it met the Turing test of generating output indistinguishable from human response?
ChatGPT may be smart, quick and impressive. It does a good job at exhibiting apparent intelligence. It sounds humanlike in conversations with people and can even display humor, emulate the phraseology of teenagers, and pass exams for law school.
But on occasion, it has been found to serve up totally false information. It hallucinates. It does not reflect on its own output.
Cameron Jones, who specializes in language, semantics and machine learning, and Benjamin Bergen, professor of cognitive science, drew upon the work of Alan Turing, who 70 years ago devised a process to determine whether a machine could reach a point of intelligence and conversational prowess at which it could fool someone into thinking it was human.
Their report titled “Does GPT-4 Pass the Turing Test?” is available on the arXiv preprint server.
They rounded up 650 participants and generated 1,400 “games” in which brief conversations were conducted between participants and either another human or a GPT model. Participants were asked to determine who they were conversing with.
The researchers found that GPT-4 models fooled participants 41% of the time, while GPT-3.5 fooled them only 5% to 14% of the time. Interestingly, humans succeeded in convincing participants they were not machines in only 63% of the trials.
The researchers concluded, “We do not find evidence that GPT-4 passes the Turing Test.”
They noted, however, that the Turing test still retains value as a measure of the effectiveness of machine dialogue.
“The test has ongoing relevance as a framework to measure fluent social interaction and deception, and for understanding human strategies to adapt to these devices,” they said.
They warned that in many instances, chatbots can still communicate convincingly enough to fool users in many instances.
“A success rate of 41% suggests that deception by AI models may already be likely, especially in contexts where human interlocutors are less alert to the possibility they are not speaking to a human,” they said. “AI models that can robustly impersonate people could have could have widespread social and economic consequences.”
The researchers observed that participants making correct identifications focused on several factors.
Models that were too formal or too informal raised red flags for participants. If they were too wordy or too brief, if their grammar or use of punctuation was exceptionally good or “unconvincingly” bad, their usage became key factors in determining whether participants were dealing with humans or machines.
Test takers also were sensitive to generic-sounding responses.
“LLMs learn to produce highly likely completions and are fine-tuned to avoid controversial opinions. These processes might encourage generic responses that are typical overall, but lack the idiosyncrasy typical of an individual: a sort of ecological fallacy,” the researchers said.
The researchers have suggested that it will be important to track AI models as they gain more fluidity and absorb more humanlike quirks in conversation.
“It will become increasingly important to identify factors that lead to deception and strategies to mitigate it,” they said.
News
The Surprising Link Between Smell, Sound, and Emotions
New research reveals how smell and hearing interact in the brain to drive social behavior, using mouse maternal instincts as a model. Imagine you’re at a dinner party, but you can’t smell the food [...]
Brain cells age at different rates
As our body ages, not only joints, bones and muscles wear out, but also our nervous system. Nerve cells die, are no longer fully replaced, and the brain shrinks. "Aging is the most important risk factor [...]
Long COVID Breakthrough: Spike Proteins Persist in Brain for Years
Researchers have discovered that the SARS-CoV-2 spike protein persists in the brain and skull bone marrow for years after infection, potentially leading to chronic inflammation and neurodegenerative diseases. Researchers from Helmholtz Munich and Ludwig-Maximilians-Universität (LMU) have [...]
Water-Resistant Paper Could Revolutionize Packaging and Replace Plastic
A groundbreaking study showcases the creation of sustainable hydrophobic paper, enhanced by cellulose nanofibres and peptides, presenting a biodegradable alternative to petroleum-based materials, with potential uses in packaging and biomedical devices. Researchers aimed to [...]
NIH Scientists Discover Game-Changing Antibodies Against Malaria
Novel antibodies have the potential to pave the way for the next generation of malaria interventions. Researchers at the National Institutes of Health (NIH) have identified a novel class of antibodies that target a previously unexplored region [...]
Surprising Discovery: What If Some Cancer Genes Are Actually Protecting You?
A surprising discovery reveals that a gene previously thought to accelerate esophageal cancer actually helps protect against it initially. This pivotal study could lead to better prediction and prevention strategies tailored to individual genetic [...]
The Cancer Test That Exposes What Conventional Scans Miss
Researchers at UCLA have unveiled startling findings using PSMA-PET imaging that reveal nearly half of patients diagnosed with high-risk prostate cancer might actually have metastases missed by traditional imaging methods. This revelation could profoundly affect future [...]
Pupil size in sleep reveals how memories are processed
Cornell University researchers have found that the pupil is key to understanding how, and when, the brain forms strong, long-lasting memories. By studying mice equipped with brain electrodes and tiny eye-tracking cameras, the researchers [...]
Stanford’s Vaccine Breakthrough Boosts Flu Protection Like Never Before
Stanford Medicine researchers have developed a new method for influenza vaccination that encourages a robust immune response to all four common flu subtypes, potentially increasing the vaccine’s efficacy. In laboratory tests using human tonsil [...]
Water’s Worst Nightmare: The Rise of Superhydrophobic Materials
New materials with near-perfect water repellency offer potential for self-cleaning surfaces in cars and buildings. Scientists from Karlsruhe Institute of Technology (KIT) and the Indian Institute of Technology Guwahati (IITG) have developed a surface [...]
Japanese dentists test drug to help people with missing teeth regrow new ones
Japanese dentists are testing a groundbreaking drug that could enable people with missing teeth to grow new ones, reducing the need for dentures and implants, AFP recently reported. Katsu Takahashi, head of oral surgery at [...]
An AI system has reached human level on a test for ‘general intelligence’
A new artificial intelligence (AI) model has just achieved human-level results on a test designed to measure "general intelligence." On December 20, OpenAI's o3 system scored 85% on the ARC-AGI benchmark, well above the previous AI best [...]
According to Researchers, Your Breathing Patterns Could Hold the Key to Better Memory
Breathing synchronizes brain waves that support memory consolidation. A new study from Northwestern Medicine reports that, much like a conductor harmonizes various instruments in an orchestra to create a symphony, breathing synchronizes hippocampal brain waves to [...]
The Hidden Culprit Behind Alzheimer’s Revealed: Microglia Under the Microscope
Researchers at the CUNY Graduate Center have made a groundbreaking discovery in Alzheimer’s disease research, identifying a critical link between cellular stress in the brain and disease progression. Their study focuses on microglia, the brain’s immune [...]
“Mirror Bacteria” Warning: A New Kind of Life Could Pose a Global Threat
Mirror life, a concept involving synthetic organisms with reversed molecular structures, carries significant risks despite its potential for medical advancements. Experts warn that mirror bacteria could escape natural biological controls, potentially evolving to exploit [...]
Lingering Viral Fragments: The Hidden Cause of Long COVID
Long COVID, affecting 5-10% of COVID-19 patients, might be caused by the enduring presence of the virus in the body. Research suggests that viral fragments, possibly live, linger and lead to symptoms. Addressing this involves antiviral treatments, enhanced [...]