The issues with AI medical advice

Not long ago, when people had health questions, they turned to their doctor first. Then came social media. And now? For many, AI is the new first stop.

In KFF’s poll on health information and trust, 32% of adults say they’ve turned to AI for health information and advice in the past year. 29% of respondents say they’ve used AI tools for physical health advice, while 16% say they’ve used them for mental health guidance.

Treating chatbots as clinicians is increasingly common. But relying on AI for medical advice comes with real risks for patients—and for the employers that cover their care.

Why are employees turning to AI with their health questions?

It’s tempting to chalk this up to laziness or misplaced trust in technology. But for many people, AI fills needs the healthcare system struggles (or flat-out fails) to meet. There are plenty of understandable reasons why people bring their health concerns and questions to a chatbot first.

Quick answers

According to the same KFF poll, 65% of respondents said immediate information was a major reason why they relied on AI. People can get answers right away, which is a lot faster than waiting to see a provider. In major U.S. metropolitan areas, the average wait time for a new patient doctor’s appointment is 31 days across a range of specialties.

Visit preparation

Nearly half of Americans say they feel anxious before a doctor’s appointment, and heading into that visit feeling more informed can help calm some of those nerves. It’s a common reason, with 41% of people in the KFF poll saying they used AI tools because they wanted to look up information before seeing a provider.

Comfort

Chatting with an AI bot feels relatively low-stakes and confidential (even when it isn’t), particularly when discussing sensitive medical details. In the KFF poll, 36% of respondents said they were more comfortable looking up their health questions privately.

And, interestingly, in some studies people rated AI responses as more empathetic than human-written ones, which amps up their comfort level when corresponding with AI. In comparison, almost 50% of Americans believe the healthcare system lacks compassion, and that compassion is present in less than 1% of routine office visits.

Access

Some people turn to AI for healthcare not because they believe it’s their best option, but because it often feels like their only option. In the KFF poll, one in five AI health users say they use AI because they don’t have a healthcare provider or are unable to get an appointment. That number rises even higher (38%) among younger users between the ages of 18 to 29.

Additionally, more than 30% of U.S. adults don’t have a regular primary care provider, so AI is often their most suitable starting point.

Equity

Similarly, unlike healthcare, pretty much anyone can use AI tools for answers. The KFF poll found that larger shares of younger adults, uninsured adults, Black adults, and Hispanic adults turned to chatbots for mental health advice. Many of these groups face more hurdles to traditional care, so a free, always-available chatbot can be easier to reach than a therapist or doctor.

Affordability

Money matters, too. In the KFF poll, one in five AI health users said worries over affording healthcare were another major factor in their decision to use AI.

28% of American adults have skipped or postponed necessary healthcare in the past year because of cost concerns. In comparison, many AI tools still have free versions users can leverage. Only 2% of U.S. households spend money on AI tools, despite the fact that 56% of American adults have used them.

What are the possible risks of AI medical advice?

Those are valid reasons for turning to AI, and it can be a helpful starting point for learning more about a symptom or prepping for an appointment. But chatbots aren’t doctors—and while their advice might be free and convenient, it can turn out to be costly and complicated.

Privacy

According to the KFF poll, 77% of the public say they’re concerned about the privacy of the personal medical information provided to AI tools. Yet, despite those fears, 41% of those who have used AI for healthcare advice say they’ve uploaded personal medical information to an AI tool or chatbot.

AI (along with its uses and availability) keeps changing, which means privacy regulations are still evolving too. Currently, there isn’t a single universal law that requires AI systems to keep medical information private; privacy rules largely depend on who’s using the AI tool and where the person is geographically when they enter the data. That means a consumer AI tool isn’t automatically subject to HIPAA rules.

So, that chatbot conversation about persistent heartburn that feels personalized and confidential? Depending on the AI tool and the user’s location, it could actually be stored, reviewed, used to train future AI models—and potentially exposed in a future data breach.

Bias

AI tools are programmed to please people by telling them exactly what they want to hear. In fact, research shows that chatbots are far more likely to agree with users who state their opinions upfront, rather than providing critical or neutral responses. If a user starts a conversation with, “I think I have [xyz diagnosis],” the model is inclined to provide evidence to support their original assumption instead of offering neutral information or encouraging them to think more critically.

Plus, AI tools don’t just have a confirmation bias—they also hold their own biases against certain groups. Some research has found that AI models often provide less accurate and less honest responses to certain users, including those who have lower English proficiency, obtained less formal education, or aren’t from the U.S.

While AI tools seem to offer equal access to everybody, they can actually perpetuate inequity in healthcare.

Accuracy

AI’s medical advice might be fast and easy, but it’s not always right. In a recent study, participants were instructed to consult with AI tools about medical scenarios they were provided. After conversing with the chatbot, participants correctly identified their hypothetical condition only about one-third of the time. Only 43% of participants made the correct decision about their next steps.

Doctors are trained to ask patients thoughtful questions to get a better grasp of their symptoms and medical history. But when conversing with a chatbot, users are left to their own devices. They decide what information to share (and what to leave out). AI doesn’t have the same level of training to dig deeper or avoid leading questions, so it might never learn a crucial detail a doctor would’ve caught early on.

Put simply, a lot of AI’s output depends on the input. This means people can get drastically different answers based on how they phrase their own symptoms and experiences. One person might end up in the ER or a specialist’s office for something minor, while others brush off symptoms that actually need immediate attention. The result is poorer health outcomes, wasted time, and higher costs for patients and employers.

Safety

Some of the riskiest AI advice isn’t wrong. It’s technically correct, but still inappropriate for the person asking. According to a report from Duke University School of Medicine, for example, one user even asked how to perform a medical procedure at home. While the chatbot gave a warning that it should only be performed by professionals, it still provided step-by-step instructions. A doctor would’ve (hopefully) stopped that conversation immediately.

Some of this is because AI relies so heavily on inputs. Patients tend to phrase questions like, “What dose of this drug should I take for my condition?” But if their self-diagnosis or medication choice is incorrect to begin with, the chatbot’s answer will just build on that mistake.

Again, clinicians are trained to read between the lines and redirect patients when they need to. Chatbots are designed to answer the question that’s asked—even when it’s the wrong question.

What employers can do to steer employees toward better care

Like it or not, employees will probably keep asking chatbots their health questions. You don’t need to stop them—but you can steer them in the right direction. Here’s how.

Address the barriers that push people to use AI medical advice

Many people use chatbots because more “traditional” care feels too slow, too hard to reach, or too expensive. Your benefits can address all three. You could:

• Offer virtual primary or same-day telehealth so employees can talk to a clinician in hours instead of waiting a month for an appointment.
• Reduce or waive cost-sharing for primary care, telehealth, and mental health visits.
• Consider a centers of excellence program that connects employees with high-quality specialty care with little or no out-of-pocket cost.

Emphasize AI as a starting point

Encourage them to use AI tools as an initial resource rather than a final decision-maker. Researching a symptom before an appointment can help employees ask better questions, but diagnosis and treatment decisions still belong with a clinician.

This is worth frequent reminders, as the KFF poll found that 42% of people who used AI for physical health questions didn’t follow up with a doctor, and 58% of those who used it for mental health didn’t follow up with a mental health professional.

Make other resources easy to find

Employees often turn to AI because they don’t know where else to go, so spell it out for them. For example:

• Create a one-page guide that lists your nurse line, telehealth service, employee assistance program (EAP), care navigation, and specialty care options, along with phone numbers and links.
• Name a specific HR contact for benefits questions, and share realistic response times so employees know when to expect an answer.
• If you provide employees with access to a specific AI tool, train that tool on your company’s HR policies and benefits so employees can ask the tool about what’s available to them and easily find answers.

You don’t have to overhaul your benefits overnight. Start with the barrier that’s most likely tripping up your employees (whether it’s long wait times, high costs, or confusion about where to go) and then build from there.

AI is quickly becoming a first stop for health questions, and that likely won’t change anytime soon. So, as an employer, your goal is to make sure it doesn’t become the last stop.

When quality care is fast, affordable, and easy to find, employees are far less likely to let a chatbot have the last word on their health.

That’s where Carrum Health comes in. Carrum connects employees with vetted centers of excellence, so they can get expert answers and high-quality specialty care—without all the prompts and problems.