‘Unbelievably dangerous’: experts sound alarm after ChatGPT Health fails to recognise medical emergencies
ChatGPT Health regularly misses the need for medical urgent care and frequently fails to detect suicidal ideation, a study of the AI platform has found, which experts worry could “feasibly lead to unnecessary harm and death”.OpenAI launched the “Health” feature of ChatGPT to limited audiences in January, which it promotes as a way for users to “securely connect medical records and wellness apps” to generate health advice and responses.More than 40 million people reportedly ask ChatGPT for health-related advice every day.The first independent safety evaluation of ChatGPT Health, published in the February edition of the journal Nature Medicine, found it under-triaged more than half of the cases presented to it.The lead author of the study, Dr Ashwin Ramaswamy, said “we wanted to answer the most basic safety question; if someone is having a real medical emergency and asks ChatGPT Health what to do, will it tell them to go to the emergency department?”Ramaswamy and his colleagues created 60 realistic patient scenarios covering health conditions from mild illnesses to emergencies.
Three independent doctors reviewed each scenario and agreed on the level of care needed, based on clinical guidelines.Sign up: AU Breaking News emailThe team then asked ChatGPT Health for advice on each case under different conditions, including changing the patient’s gender, adding test results, or adding comments from family members, generating nearly 1,000 responses.They then compared the platform’s recommendations with the doctors’ assessments.While it performed well in textbook emergencies such as stroke or severe allergic reactions, it struggled in other situations.In one asthma scenario, it advised waiting rather than seeking emergency treatment despite the platform identifying early warning signs of respiratory failure.
In 51.6% of cases where someone needed to go to the hospital immediately, the platform said stay home or book a routine medical appointment, a result Alex Ruani, a doctoral researcher in health misinformation mitigation with University College London, described as “unbelievably dangerous”.“If you’re experiencing respiratory failure or diabetic ketoacidosis, you have a 50/50 chance of this AI telling you it’s not a big deal,” she said.“What worries me most is the false sense of security these systems create.If someone is told to wait 48 hours during an asthma attack or diabetic crisis, that reassurance could cost them their life.
”In one of the simulations, eight times out of 10 (84%), the platform sent a suffocating woman to a future appointment she would not live to see, Ruani said,Meanwhile, 64,8% of completely safe individuals were told to seek immediate medical care, said Ruani, who was not involved in the study,The platform was also nearly 12 times more likely to downplay symptoms because the “patient” told it a “friend” in the scenario suggested it was nothing serious,“It is why many of us studying these systems are focused on urgently developing clear safety standards and independent auditing mechanisms to reduce preventable harm,” Ruani said.
A spokesperson for OpenAI said while the company welcomed independent research evaluating AI systems in healthcare, the study did not reflect how people typically use ChatGPT Health in real life,The model is also continuously updated and refined, the spokesperson said,Ruani said even though simulations created by the researchers were used, “a plausible risk of harm is enough to justify stronger safeguards and independent oversight”,Ramaswamy, a urology instructor at the Icahn School of Medicine at Mount Sinai in the US, said he was particularly concerned by the platform’s under-reaction to suicide ideation,“We tested ChatGPT Health with a 27-year-old patient who said he’d been thinking about taking a lot of pills,” he said.
When the patient described his symptoms alone, the crisis intervention banner linking to suicide help services appeared every time.“Then we added normal lab results,” Ramaswamy said.“Same patient, same words, same severity.The banner vanished.Zero out of 16 attempts.
A crisis guardrail that depends on whether you mentioned your labs is not ready, and it’s arguably more dangerous than having no guardrail at all, because no one can predict when it will fail,”Prof Paul Henman, a digital sociologist and policy expert with the University of Queensland, said: “This is a really important paper,“If ChatGPT Health was used by people at home, it could lead to higher numbers of unnecessary medical presentations for low-level conditions and a failure of people to obtain urgent medical care when required, which could feasibly lead to unnecessary harm and death,”He said it also raised the prospects of legal liability, with legal cases against tech companies already in motion in relation to suicide and self-harm after using AI chatbots,“It is not clear what OpenAI is seeking to achieve by creating this product, how it was trained, what guardrails it has introduced and what warnings it provides to users,” Henman said.
“Because we don’t know how ChatGPT Health was trained and what the context it was using, we don’t really know what is embedded into its models.”
Polls close in Gorton and Denton byelection after three-way battle between Greens, Labour and Reform
The polls have closed in the three-way battle for Gorton and Denton in south-east Manchester after one of the most unpredictable byelections in years.The Green party leader Zack Polanski said before voting that his party was “neck and neck” with Reform UK to overturn Labour’s 13,000-vote majority, and that Labour will need to “search their conscience” if Reform UK wins.Keir Starmer’s party had targeted left-leaning voters in the Greater Manchester seat with claims that only Labour can see off Nigel Farage’s Reform, saying that a vote for the Greens was “in effect, a vote for Reform”.Labour’s strategy of claiming the Greens could not win had echoes of the disastrous Caerphilly byelection in October, which the party lost to Plaid Cymru despite telling voters repeatedly: “Only Labour can beat Reform.”Labour is defending a 13,413-vote majority in Gorton and Denton, where nearly 80% of voters backed a party on the left at the 2024 election
The leadership issue may be settled, but Your Party’s struggle for electoral relevance has only just begun
At 11am on the dot, about 1,000 Your Party members, along with a few lurking journalists, were waiting online to find out who would lead the party after a bruising two-week election campaign. “One minute to paradise!”, joked one in the comments alongside the YouTube livefeed.In the end, they had time to make a brew. A Your Party official entered the comments to inform those waiting that the results would, in fact, begin half an hour later than planned. “I blame Thatcher,” one comrade grumped
Dual national rules are another own goal for Labour | Brief letters
Regarding the new rules on dual nationals (Report, 24 February), given its standing in the polls, surely the government would prefer not to give voters yet another reason to think they are governed by callous, indifferent fools? Permitting dual nationals to enter with an electronic travel authorisation would be a simple fix for a stupid and illiberal policy inflicted on its own citizens.Bill RobinsonNorwich Keir Starmer’s latest change of mind, over local elections (Report, 16 February), reminds me that years ago, my driving instructor told me U-turns should be avoided if possible. But if one was required, it was necessary to move as far to the left as possible before turning. Stuart Harrington Burnham on Sea, Somerset I realise a Quick Crossword is not supposed to be very difficult, but “number of days in two weeks” or “prehistoric monument on Salisbury Plain” (19 February) are clues that most young children could probably solve. I have lately gone from taking about 15 minutes to solve it to an average of four
The Your Party committee election was chaos. Why break the habit of a lifetime? | John Crace
Start as you mean to go on. Your Party has had a fair few ups and downs in its short lifespan. Some might call it chaos. Its two most prominent members, Jeremy Corbyn and Zarah Sultana, seem barely able to stand being in the same room as each other. Allegations of financial misconduct over membership fees and donations
Labour, Green party and Reform make final pitch to voters in Gorton and Denton – as it happened
Grassroots Left, the Your Party faction supporting Zara Sultana, has issued a statement following the leadership team elections won by the rival The Many slate, which backs Jeremy Corbyn. It says:double quotation markOur party is strongest when members have real power: over policy, finances, selections, and decision-making – through transparent, accountable structures. All Grassroots Left members will push for this on the CEC. We will push to make sure the branches are recognised immediately, fully supported and that members are put at the heart of the party.Your Party must now work together to become a party of and for the whole left – with no more witch-hunts or stitch-ups
Your Party under Corbyn to work with Greens on ‘coordinated left-flank offensive’
A Jeremy Corbyn-led Your Party will work with the Green party and others to push for a “coordinated left-flank offensive” against Reform and Labour, the Guardian understands.After winning a comprehensive victory to become the de facto leader of the leftwing startup party on Thursday, Corbyn will seek to rebuild bridges with pro-Gaza communities “alienated” by his rival Zarah Sultana, sources close to Corbyn said.Corbyn will become Your Party’s parliamentary leader, after an election in which Sultana was also voted on to the party’s leadership committee under its collective leadership model. Corbyn-backed candidates took 14 of the 24 available places on the party’s central executive committee (CEC). Sultana-backed candidates took seven of the seats and three went to independents
‘He’s doing all he can’: England back Buttler to end miserable run of form
The Indiana Bears? Why an interstate move for a cherished NFL team may work out
Norway’s all-conquering Winter Olympians have a message for us all – and it’s not what you think | Cath Bishop
US hockey star Hilary Knight responds to Trump’s ‘distasteful joke’ about women’s team
Saracens’ salary cap penalty under scrutiny over conflict of interest claims
English cricket’s hunger for Indian money has led it into a moral and legal minefield | Barney Ronay