Ziad Obermeyer, cofounder and chief scientific officer of Dandelion Health.
Dandelion HealthWhile the Biden administration grapples with how to keep up with the fast-changing technology, Dandelion Health is trying to fill the void with a de-identified dataset for developers to build and test how their algorithms perform.
Heart failure can be tricky for doctors to diagnose. Some patients won’t find out their heart isn’t pumping enough blood until they come into the emergency room with a heart attack. And it’s more common for men to have the tell-tale chest-clutching pain, while women’s symptoms may be much more subtle, meaning they might go undetected for even longer.
In 2023, cardiologist David Ouyang developed an algorithm to try and catch heart failure earlier. It analyzes electrocardiograms or ECGs – recordings of electrical heartbeat signals – to predict if a patient has heart failure, flagging them to the doctor so they get further testing.
But Ouyang, who works at Cedars-Sinai in Los Angeles, needed to be sure that the algorithm would be accurate for both men and women — as well as people of different races and ethnicities or who live in different places, factors that can impact the symptoms of heart failure. In other words, he needed to make sure the algorithm wouldn’t have any bias, a long-standing issue for algorithms that are trained using historical data from a limited number of people. If that data is not representative of all types of patients, then the resulting algorithm won’t be accurate for everyone.
To get answers, he turned to Dandelion Health, a startup that’s working on a solution to help check for potential bias of this type in healthcare algorithms. Dandelion is creating a massive, de-identified dataset from millions of patient records so developers can build and test the performance of their algorithms across diverse types of patients. Ouyang said working with Dandelion gave him “more confidence” that his algorithm is “mature and ready to deploy.”
Opening up the black box of algorithmic decision-making and performance bias is an issue the Biden Administration, software developers and healthcare players are all grappling with, especially given the speed of advancements in generative AI over the past year. But it could take years before President Biden’s AI executive order translates into detailed regulation, creating an opportunity for companies like Dandelion to fill the void.
Dandelion’s founding team hopes they can help establish a framework for testing and validating healthcare AI while regulators play catchup. To do so, the startup just closed a $15 million seed round led by Primary along with Sharp HealthCare, Moxxie Ventures, Phoenix Venture Partners and Floating Point. The company declined to disclose the valuation.
“Over the last few years, billions and billions of dollars have gone into startups to build AI algorithms for clinical use,” Dandelion cofounder and CEO Elliott Green told Forbes. “And the problem they’re having now is actually selling them.”
The biggest thing that developers of health-related algorithms need is data on which to train and test their software to make sure it works. But getting that data is a challenge: hospitals, which are required to protect the privacy and security of their patient records by federal law, can’t go handing out data willy-nilly to every company that asks for it. This also means that many healthcare algorithms are being built on small and incomplete datasets, which means performance problems are being identified after they’re already in use.
We have full control over the data … That was attractive to us to ensure that we had a hand on the wheel.
Jared Antczak, chief digital officer, Sanford Health
So how big, exactly, does a dataset need to be to reduce bias? Dandelion cofounder and chief scientific officer Ziad Obermeyer, who researches bias in healthcare as a professor at the University of California Berkeley, admits that there’s no “magic number” of patients from different ethnicities or income levels or conditions when it comes to creating a representative training dataset for healthcare AI. But Dandelion’s goal, he said, is to be able to answer the question of “how good” one algorithm is compared to the others out there.
So far the company has struck deals with three hospitals to take the medical records of 10 million patients that appear in many different formats – electronic records, image files, waveforms, lab results – and de-identify and structure it so that researchers and software developers can train, test, validate and, eventually, apply for regulatory clearance. Dandelion is intentionally inking deals with health systems across the country, including Sanford Health in the rural midwest, Sharp HealthCare in urban California and Texas Health Resources in cities and suburbs in the southwest. Within the next year, Dandelion hopes to have partnerships with two more health systems for a total of 15 to 20 million patient records.
One of the features that makes Dandelion unique is that it leases data from its hospital system partners and they evaluate any data requests from algorithm developers. “We have full control over the data,” Sanford’s chief digital officer Jared Antczak told Forbes. “That was attractive to us to ensure that we had a hand on the wheel, so to speak, as we step into this space.” There is a revenue-sharing agreement between Dandelion and its hospital partners, though the company declined to specify details.
Sanford has a particularly rich dataset, since the health system has used the same electronic health record provider Epic Systems for nearly 20 years. That means its data can be used to examine what happened to a patient over time — very useful for determining if an algorithm is working as intended.
For example, say an algorithm is scanning chest images to look for lung tumors. Rather than being trained on just the report from the human radiologist, the algorithm can be trained on what actually happened – did the patient respond to a particular treatment? Was another tumor discovered later on? Could the cancer have been detected at an earlier stage? “It means you base the algorithm on what actually transpired over time and not just on what the radiologist thought was going to transpire,” said Green. “It turns out these often have significant differences.”
Sanford’s data is also valuable because it represents a rural population of patients; there’s not as much churn among patients in the upper rural midwest as in an urban setting, which means it includes multiple generations of families, said Antczak. By partnering with Dandelion and developers who use the data, “we can ensure that we mitigate the bias of only including urban populations” in algorithm development and “promote health equity more broadly,” he said.
Healthcare access can vary wildly “depending on where you live, who you are, the color of your skin, the language you speak.”
Ziad Obermeyer, cofounder, Dandelion Health
In 2019, Obermeyer and Sendhil Mullainathan, a professor of computation and behavior science at the University of Chicago Booth School of Business, co-authored a research paper on bias in healthcare algorithms that was published in Science. That paper’s findings would inspire them to start Dandelion along with Green, a former vice president of partnerships and strategy at health insurer Oscar Health, and Niyum Gandhi, the CFO of Massachusetts General Brigham, who now serves as Dandelion’s board chair.
The paper revealed how differences in access to healthcare services among Black and white patients could ultimately result in fewer Black patients being flagged by an algorithm that used overall healthcare costs as a proxy for which patients need extra care. The cost prediction algorithm was developed by Optum, a division of UnitedHealth Group, and widely deployed.
That’s because if you just consider the total cost of care – that the sickest patients would be the ones with the highest bills – the data will skew towards people who can afford to go to the doctor. The result was that only around half of the Black patients who should get extra services were identified.
Access can vary wildly “depending on where you live, who you are, the color of your skin, the language you speak,” Obermeyer told Forbes. In this case, white patients were more likely to go to clinics and get treatment or surgery and had higher costs, while Black patients were more likely to use the emergency room once their untreated conditions were spiraling out of control. The end result? “The bias just piles up.”
Optum spokesperson Tyler Mason said Obermeyer’s study “mischaracterized” the cost prediction algorithm based on the “incorrect use” by one health system. “The algorithm is designed to predict future costs that individual patients may incur based on past healthcare experiences and does not result in racial bias when used for that purpose – a fact with which the study authors agreed,” Mason said in a statement. (Disclosure: In a previous career, I worked at a PR firm that had UnitedHealth Group and Optum as one of its clients.)
Obermeyer said it’s true the algorithm excludes race as a data point and accurately predicts the future healthcare costs of individuals. But on average Black patients generated lower costs than white patients with the same chronic illnesses. This means an accurate cost prediction indicates the reality that Black patients are getting fewer services. “The tragedy of this whole thing is the algorithm is doing exactly what we told it to do,” said Obermeyer. “It’s just that we told it to do the wrong thing.”
This isn’t just about Optum, Obermeyer said. It’s an industry-wide problem that is affecting millions of patients: artificial intelligence algorithms may not perform as intended on real-world data, and, in many cases, problems aren’t identified until they’re deployed on a large scale.
The rapidly evolving nature of AI technology also means the rules are being developed on the fly. As of October 19, 2023, the FDA has authorized nearly 700 AI-enabled medical devices. The majority of them are for uses in radiology, such as assisting human doctors with reading medical scans – and there have been no authorizations for devices that use large language models, the technology behind ChatGPT. However, Troy Tazbaz, director of the FDA’s Digital Health Center of Excellence, acknowledged the agency will have to update its approach, especially as the AI models and inputs get more complex. “With generative AI, the technology is adapting without human intervention, so the guardrails have to be different from traditional software,” he told Forbes.
There is debate among policy experts as to whether the existing regulations that govern software-based medical devices are stringent enough for AI. “Regulation will be catching up to our knowledge of how this technology develops,” said Genevieve Kanter, a professor of public policy at the University of Southern California. Some AI health software developers aren’t required to submit clinical trial data to begin with, she said. Plus, they’re evaluated at a single point in time – even though the models and data inputs can drastically change. “We need some kind of continuous performance monitoring in order to identify the weaknesses that are going to emerge,” said Kanter.
Last week, 28 healthcare organizations signed a voluntary commitment with the Biden administration agreeing to focus on several core principles when it comes to deploying healthcare AI to ensure it is “fair, appropriate, valid, effective and safe.” But it does not prescribe the process for doing so.
In the meantime, Dandelion hopes to provide a solution. Eventually the plan is to create a marketplace where algorithms that have been tested and validated on its data will be connected with health systems and other customers. “It needs to be done really responsibly,” said Green. “Ultimately, the winner is the patient.”
MORE FROM FORBESMORE FROM FORBESBiden’s New Executive Order Will Regulate AI Models That Could Threaten National SecurityBy David JeansMORE FROM FORBESAn AI Doctor In A Box Coming To A Mall Near YouBy Katie JenningsMORE FROM FORBESFrom Google To Nvidia, Tech Giants Have Hired Red Team Hackers To Break Their AI ModelsBy Rashi ShrivastavaMORE FROM FORBESThis Startup Raised $30 Million To Take On Microsoft’s AI Notetaker For DoctorsBy Katie Jennings
{Categories} *ALL*,_Category: Implications{/Categories}
{URL}https://www.forbes.com/sites/katiejennings/2023/12/21/how-this-startup-is-using-10-million-patient-records-to-reduce-bias-in-healthcare-ai/{/URL}
{Author}Katie Jennings, Forbes Staff{/Author}
{Image}https://imageio.forbes.com/specials-images/imageserve/6583257217bfe9ddbb3bf88f/0x0.jpg?format=jpg&crop=1920,1080,×0,y587,safe&height=600&width=1200&fit=bounds{/Image}
{Keywords}Healthcare,/healthcare,Innovation,/innovation,Healthcare,/healthcare,Venture Capital,/venture-capital,Daily Cover,daily-cover,Editors’ Pick,editors-pick,premium{/Keywords}
{Source}Implications{/Source}
{Thumb}{/Thumb}