Researchers have revealed that AI may be able to determine race from X-ray scans, despite the fact that there is no evident difference between human specialists. The AI was able to detect race with about 90% accuracy using only X-ray and CT pictures, and experts are baffled as to how it accomplished this. The findings show that when using AI in medical technology, significant caution should be exercised, as AI may be able to classify photos that are assumed to be anonymous, hazy, or otherwise distorted.
The study appeared in the Lancet Digital Health journal. In a statement to the Boston Globe, Marzyeh Ghassemi, an MIT assistant professor of electrical engineering and computer science and coauthor of the work, said, “When my graduate students showed me some of the results in this article, I genuinely felt it must be a mistake.” “When my pupils informed me, I honestly believed they were insane.”
A deep-learning model was fed CT and X-ray scans of many different locations of the body on diverse groups of individuals, each image tagged with the person’s race, to test AI’s ability to detect images. The tags were then removed, along with any other distinguishing characteristics like as skin and hair color. The AI was asked to identify the individual’s race in each photograph. Even without any distinguishing traits, the AI was able to detect race with over 90% accuracy in all parts of the body.
To be sure, the researchers worried if the AI was being clever and making the assumption based on covariables like body mass index (BMI) or breast density, which may indicate one race over another. Even after removing all possible covariables and just showing AI datasets of persons with comparable BMI and body types, the AI was still able to determine race. How is that possible? To be honest, the researchers aren’t sure.
This isn’t the first time AI has startled scientists by correctly recognizing race under nearly impossible circumstances; earlier research has shown it can do it even when the image is highly distorted or manipulated. However, the results are perplexing since covariables were removed. The researchers believe the AI can detect differences in melanin between black and white people’s skin, which may show up in CT and X-ray scans yet go unnoticed by humans. But it is just one hypothesis, and the real job now begins with understanding the outcomes.
Regardless, the findings should raise severe concerns regarding the use of AI in hospitals and how it may behave differently to various races, as has been demonstrated in the deployment of AI on several occasions. The Boston Globe quoted Leo Anthony Celi, co-author and associate professor at Harvard Medical School, as saying, “We need to take a halt.” “We can’t rush the algorithms into hospitals and clinics until we’re convinced they’re not making racist or sexist choices,” she says.
