Building a Healthier Future: Designing for AI Health Equity

In a typical scenario, a patient might input their symptoms or lab results in a chatbot to generate a diagnosis, a second opinion, or ideas for a treatment regimen. While some doctors note that this suggests a level of patient engagement in the management of their own health — an activity they welcome — they also explain how AI chatbots can promote confusion when the generated outputs are inaccurate or not applicable to the patient.

Users' engagement with AI chatbots are quite different than their engagement with more conventional informational tools like the Google search engine. A surging number of people are turning to AI chatbots for companionship and therapeutic relief. 

Safety-net providers identify AI as a potential remedy, noting that the technology's ability to help collect and analyze digital and medical biomarkers while the patient is away from the clinic could be transformational. Safety-net providers also express concerns about AI, including insufficient technical and financial resources to support AI training and the building of the technical infrastructure required to integrate AI into the clinical workflow.

Most clinicians understand the potential value of large language models (LLMs) in health care but voice concerns about the ability of these systems to generate outputs that are based on clinically validated evidence. 

Although LLMs are being used to support diagnostic reasoning and offer mental health support to those in need, there are risks associated with the use of LLMs in health care, including the tendency of these systems to "hallucinate," that is, to make things up and add in bias against populations that may not be adequately represented in the datasets and training models used to develop frontier LLMs.

Synthetic data is artificially generated data that mimics real-world data but is not collected from actual people or real environments. The core problem synthetic data solves is non-generalizability. AI models trained on narrow or biased datasets consistently underperform across demographic subgroups, care settings, and devices, which systematically amplifies existing health disparities (Gallon, 2024; Daneshjou et al., 2022)

While synthetic data offers powerful tools for enhancing equity and privacy, it is not immune to reproducing or even amplifying biases that exist in source datasets or in the design and deployment processes.

Pharmaceutical companies face mounting expectations to accelerate research and development timelines, diversify clinical trials, and ensure AI-enabled tools perform equitably across populations and geographies. Evidence-based practices can align innovation with equity imperatives across the development lifecycle.

One of the most enduring problems in the execution of clinical trials is the lack of diversity in participant populations (Oertelt-Prigione & Turner, 2024). Inadequate race and ethnic representation in clinical trials limits the ability to detect notable differences between demographic groups with regards to an intervention's safety or efficacy.