A.I. Chatbots Defeated Doctors at Diagnosing Illness

NY Times ^ | 2024/11/17/ | Gina Kolata

[dennisw]

A small study found ChatGPT outdid human physicians when assessing medical case histories, even when those doctors were using a chatbot.

Dr. Adam Rodman, an expert in internal medicine at Beth Israel Deaconess Medical Center in Boston, confidently expected that chatbots built to use artificial intelligence would help doctors diagnose illnesses.

He was wrong.

Instead, in a study Dr. Rodman helped design, doctors who were given ChatGPT-4 along with conventional resources did only slightly better than doctors who did not have access to the bot. And, to the researchers’ surprise, ChatGPT alone outperformed the doctors.

“I was shocked,” Dr. Rodman said.

The chatbot, from the company OpenAI, scored an average of 90 percent when diagnosing a medical condition from a case report and explaining its reasoning. Doctors randomly assigned to use the chatbot got an average score of 76 percent. Those randomly assigned not to use it had an average score of 74 percent.

The study showed more than just the chatbot’s superior performance.

It unveiled doctors’ sometimes unwavering belief in a diagnosis they made, even when a chatbot potentially suggests a better one.

And the study illustrated that while doctors are being exposed to the tools of artificial intelligence for their work, few know how to exploit the abilities of chatbots. As a result, they failed to take advantage of A.I. systems’ ability to solve complex diagnostic problems and offer explanations for their diagnoses.

A.I. systems should be “doctor extenders,” Dr. Rodman said, offering valuable second opinions on diagnoses.

But it looks as if there is a way to go before that potential is realized.

[ladyjane]

Beth Israel Hospital - AKA The House of God

[zeestephen]

Re: "What many ordinary Americans don't realize is that Boston is, truly, the world capital of medicine..."

I think that title might still be open to some debate...

In terms of funding, metro Washington D.C. has Johns Hopkins Med (Baltimore), and, the HQ for the National Institutes of Health, in Bethesda, MD, which probably has the largest concentration of medical related PhDs in the world.

Also in the running, Stanford Med and UCSF Med (metro San Francisco), which have inherited tens of billions of dollars in medical endowments from regional Tech Titans.

Also interesting - northern New Jersey (metro New York City) probably has the highest concentration of Pharmaceutical talent in the world.

[Gay State Conservative]

I never said that there weren't *outstanding* hospitals in other US cities. Of course NYC...Baltimore...Rochester,MN...San Francisco...LA...Cleveland...and other cities have world class hospitals. In fact you can even find at least one or two in Britain...and maybe even Canada! :-)

All I know is that walking down the corridors of at least one or two Boston hospitals you're tripping over Nobel Prize winning physicians and surgeons. And I don't mean Nobel Peace Prize!

Just sayin'...

[zeestephen]

I was speaking strictly in terms of medical R&D dollars spent per year.

Bethesda, MD (NIH), and Johns Hopkins, and hundreds of local start ups are ahead of Boston.

The Palo Alto to SF corridor has more medical endowment cash than Boston.

I mentioned northern New Jersey because the huge Pharma budget clearly has an impact on medical science.

Metro Boston has a very significant private Life Sciences business budget, but less than metro D.C.

[Texan4Life]

https://apnews.com/article/artificial-intelligence-chatbot-jesus-lucerne-catholic-66268027fbcf4b48972d1d62541f0b16

"AI Jesus" avatar tests man’s faith in machines and the divine

After the two-month run of the “Deus in Machina” exhibit at Peter’s Chapel starting in late August, some 900 conversations from visitors –- some came more than once –- were transcribed anonymously. Those behind the project said it was largely a success: Visitors often came out moved or deep in thought, and found it easy to use.

A small sign invited visitors to enter a confessional -– chosen for its intimacy –- and below a lattice screen across which penitent believers would usually speak with a priest, a green light signaled the visitor’s turn to speak, and a red one came on when “AI Jesus” on a computer screen on the other side was responding.