By Mengmeng Zhang (C’29)
Artificial intelligence is reshaping modern medicine. In emergency departments, where triage determines who gets seen first, a 2024 study found that a widely used algorithm undertriaged 5.5% of high-acuity patients, consistently ranking the sickest individuals as lower priority than a human nurse would have. As these systems take on greater clinical responsibility, they raise a question medicine has not fully answered: who must teach AI to care?
Medical ethics has always depended on empathy and human judgment. A physician’s reasoning involves emotion, uncertainty, and lived experience, and algorithms are built to replicate none of it. They learn from historical data, and when that history reflects who received good care and who did not, those patterns are encoded into outputs rather than interrogated. One review found that several demographic groups remain underrepresented in AI training datasets, meaning bias is embedded before a single patient is ever seen.
When an algorithm contradicts a patient’s experience, that decision is difficult to contest. One study found that one widely deployed algorithm consistently underestimated illness severity in Black patients because it used healthcare costs as a proxy for clinical need, a measure that reflects who could historically afford care rather than who actually needed it. This is not an isolated design flaw but the predictable consequence of building systems without accountability mechanisms capable of catching this kind of failure.
Bioethics must evolve alongside these technologies. Accountability and transparency must be embedded in algorithmic design from the outset, long before disparities surface in patient outcomes. Clinicians, developers, and regulators each bear a share of that responsibility, which in practice means institutional oversight at the point of deployment, mandatory bias audits against independently verified performance standards, and
meaningful inclusion of underrepresented communities in the datasets that train these systems. The measure of ethical AI is not whether machines can simulate compassion; it is whether the people designing and deploying them choose to act as though every patient’s life carries equal weight.