fbpx

Navigating the Risks of Artificial Intelligence in Healthcare

Unlocking Safe and Effective Innovation

Before we fully embrace the use of AI in our work, it’s crucial to balance our enthusiasm with knowledge of the risks of artificial intelligence in healthcare, and how to mitigate them. After all, one wouldn’t use a defibrillator unless one fully understood how to do so safely, right? Using AI is no different. Like many powerful tools, its incredible potential comes with challenges that we need to navigate carefully, to ensure the benefits outweigh potential harms. 

There are certain things you’re likely to hear people talk about when they mention disadvantages of AI in healthcare. We’ll not only explore what those are, but share strategies to navigate those risks effectively so that you can confidently—and safely—harness the full benefits of AI.

Before we dive in, want to know how you can safely start using AI today to save time and improve efficiency? Medmastery’s free course, ChatGPT Essentials, will show you how! 

Misuse of AI tools

The issue of misuse can largely be eliminated by following the instructions for use. Granted, that seems so obvious on the surface. Of course you’ll follow instructions, right? 

But have you ever come across a new tool or device that seemed so intuitive, you barely glanced at the instructions? Or maybe skipped them altogether?

Resist the urge to dive into a new AI tool without first mastering its best practices and being aware of the strengths and weaknesses of the tool.

To be fair, sometimes the documentation we need is difficult to find or seemingly absent. Other times—like when using AI to make a logo—the stakes are low, so it’s not a big deal to wing it.  Nonetheless, if you understand the common weaknesses of AI tools, you’ll be able to more effectively guard against them. And that brings us to our next point…

“Black box” reasoning

The truth is that AI algorithms don’t reason like we do. For example, an algorithm was published that could apparently review lung photos and correctly identify COVID-19. This was undoubtedly exciting, until scientists realised the algorithm was using the presence of “R” (for “right lung”) on the photos to help it draw conclusions.

But it’s not uncommon for machine learning algorithms to reason in a way that’s a complete mystery to us mere mortals. Hence, the phrase “black box” reasoning. 

This issue lies at the root of many of the risks of artificial intelligence in healthcare. For starters, our inability to understand why and how an AI system comes to a given conclusion means we can’t fully trust it. Additionally, when an AI system makes a mistake, our lack of understanding of why it did so means we don’t typically have a clear roadmap to prevent the same mistake in future. 

The solution is to treat any advice you get from an AI as a suggestion, and always verify the reasonableness of the suggestion yourself. But, when assessing the reasonableness of an AI’s suggestion, it’s crucial to be aware of automation bias.

Automation bias

Automation bias can cause clinicians to place too much weight on an AI tool’s recommendation or diagnosis, and miss situations where an AI tool is wrong. Research shows that it can cause us to stop looking for confirmatory evidence, and instead, just trust the automated AI tool.

As an example of this in action, these researchers found that medical students using a clinical decision support tool for e-prescribing were more likely to make a mistake when the tool was wrong than when it was right. In other words, they had too much trust in the tool. That caused them not to take enough time to confirm whether or not it was correct.

Other research showed that people tend to be more likely to make an incorrect decision when a clinical decision support tool is wrong, compared to situations where they have no guidance. It’s important to note that not all the news is bad though. When the tool was correct, it improved accuracy compared to situations where users had to make decisions without its help.

On the one hand, it can be tempting to dismiss concerns about automation bias since surely we all have the common sense to understand the need to verify any recommendations given to us by an AI system, right?

But we’re only human, after all. And our psychology makes us vulnerable to automation bias—albeit to varying degrees. If we’re using a system that appears trustworthy, we will be even more likely to have faith in its recommendations.

So, is this one of the risks of artificial intelligence in healthcare that we can mitigate?

Potential solutions for automation bias

Well, one line of thought is that if we avoid over-automating tasks, we will reduce automation bias.

Other suggestions include simply raising awareness of automation bias, so we can do a better job of guarding against it. 

However, research suggests that an even more effective way to reduce the incidence of automation bias is to deliberately expose users to rare errors during training. This significantly reduced complacency and increased the amount of verification users performed to verify the automated recommendations. 

Privacy

Unless you know an AI system is compliant with your local patient privacy laws, the safest approach is to assume any information you put into an AI system will not be private and take precautions accordingly. This is not only a good idea from the standpoint of ensuring legal compliance, but it’s also in line with patient preferences. Survey results showed that while 72% of patients were comfortable sharing health data with their physician, only 11% were comfortable sharing it with tech companies.

So, remove patient-identifying information from prompts that you provide to an AI. In some cases, simply removing obvious things like the patient’s name and contact info may be sufficient. However, as reported by BMC Medical Ethics, in some circumstances, AI systems have proven themselves to be capable of reidentifying the patient! So, especially in very unique cases, you may have to remove more information than usual. 

Constant change—a blessing and a curse

Updates to popular AIs are celebrated for the advancements they bring. The AIs gain new skills and often produce higher quality output (definite blessings!). However, for reasons not fully understood yet, sometimes these updates cause the AI to regress in certain areas (the curse!). As this NEJM article on the pros and cons of ChatGPT-4 as a medical chatbot rightfully points out, AIs are in a constant state of change. So, their responses can improve—or even worsen—over time. 

If you notice it stops making a particular mistake, that doesn’t mean it won’t unexpectedly start doing it again after a future update. 

Hallucinations

The AIs of today are often like a brilliant mad scientist who sometimes hallucinates. Useful applications of AI in healthcare include helping us think through certain issues, brainstorming, and as a tool for learning. But the same creativity that allows the AI to provide us with bright ideas sometimes runs wild, leading to unpredictable missteps. 

Say you wanted to use an AI to produce notes for a patient’s record. The AI would likely save you time, but you’d definitely have to be on the lookout for blatantly wrong information. 

For example, this NEJM paper reported an instance where they had the AI use a transcript of a conversation between the physician and patient to produce a note for the patient’s medical record. The AI fabricated a fake BMI for the patient, providing us with a perfect example of how the risks of artificial intelligence in healthcare can manifest in real life.

One way to reduce such errors, as that paper pointed out, is to ask the AI to review its own work to double check for mistakes. Doing so does indeed tend to reduce the AI’s errors (and sometimes eliminate them altogether). And of course, there’s no substitute for your personal review of the AI’s work. 

Keep those tips in mind when you’re using AI, whether to write a discharge summary for a patient, to write a research paper, or to assist you with learning.

Want to know best practices that experienced users of AIs like ChatGPT implement for their writing, learning, and more? Take Medmastery’s free course, ChatGPT Essentials!

Flaws in the AI’s assessment of data quality

Currently available large language models aren’t always able to accurately assess the quality of the information within their training data. There are many potential reasons for this, including the following:

  • Errors can occur if the AI can’t fully understand the formatting of the information provided. 
  • Gaps in the AI’s training data can cause it to draw inaccurate conclusions from the information that it does have.

As you can imagine, one of the dangers of AI in medicine is that if the AI isn’t able to accurately assess the quality of the information it has access to, it’s only a matter of time before it comes to an incorrect conclusion.

Bias

When an AI’s training data contains biassed information, the AI may not have enough background knowledge to recognize it. That, along with other factors, can cause the AI to share biassed information and misinformation with users.

For example, researchers from Stanford and other institutions checked 4 commercially available large language models (Bard, Claude, ChatGPT, and ChatGPT-4) to see if the AIs would spread inaccurate race-based information. Every single one of the models tested was susceptible to sharing incorrect race-based information (e.g., using debunked race-based calculations for lung and kidney function). 

Another research paper detailed how an AI algorithm underestimated the severity of health issues in Black patients. The AI’s algorithm used health care costs as a way of estimating a patient’s degree of illness. But within the patient data used to train the AI, even when the level of need was equivalent, less money was spent on Black patients compared to White patients. Clearly, using costs to estimate severity of health issues did not lead to accurate conclusions.

Another issue is that often the contents of an AI system’s training dataset aren’t public, so you may not be able to predict when bias is likely to creep into an AI’s output. 

AIs may not be up to date

The best AIs will be very upfront about telling you their knowledge cut-off date. For example, in a recent chat with OpenAI’s ChatGPT-4o, it replied, “I have been trained on a diverse and extensive dataset that includes medical and pharmaceutical information up to the cutoff date in 2023.” That being said, sometimes popular AIs like ChatGPT don’t mention their training cut-off date at all. So, it’s left up to the user to educate themselves on such things.

So, be on the lookout for odd answers that aren’t in sync with current treatment guidelines and other best practices!

Managing risks of artificial intelligence in healthcare

Whether you’re a sceptic disappointed by AI’s hype train too many times or an AI enthusiast brimming with excitement about its potential, your viewpoint is valid. AI certainly isn’t a panacea that’ll fix everything—there are some cons of AI in healthcare. But it’s also undeniable that AI offers tangible solutions to many challenges. The key lies in maintaining a balanced perspective, appreciating both the current realities of AI and potential innovations on the horizon.

Want to broaden your expertise on how AI fits into healthcare and the practice of medicine? Be sure to browse our list of the top AI courses for clinicians!

References

Educational Resources

References

AI in HEALTHCARE

Want to become a pro at prompting, and consistently get usable results? Be sure to check out Medmastery’s AI prompting course. Learn techniques to apply to the plethora of AI resources in constant development.

Sheralyn Guilleminot LITFL Author

BSc.Pharm (University of Manitoba), Pharmacist and Medical Writer

BA MA (Oxon) MBChB (Edin) FACEM FFSEM. Emergency physician, Sir Charles Gairdner Hospital.  Passion for rugby; medical history; medical education; and asynchronous learning #FOAMed evangelist. Co-founder and CTO of Life in the Fast lane | Eponyms | Books | Twitter |

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.