The DiagnosUs mobile app crowdsources the categorization and evaluation of medical data from students and practitioners of the medical field. By incentivizing correct diagnosis with real money, this technique helps medical AI firms refine their software. Centaur’s method consistently produces accurate results that can even outperform those of medical professionals. In his vision for the future, Duhaime’s organization integrates human knowledge with AI models through constant monitoring.
MIT alumnus’ platform taps the wisdom of crowds to label medical data for AI companies.
Centaur Labs developed a tool in which medical data classification is outsourced to professionals in exchange for token payments. The feedback is used to refine AI models that could save human lives.
While Erik Duhaime PhD ’19 was writing his thesis at MIT’s Center for Collective Intelligence, he saw his wife, a medical student at the time, spend countless hours using study applications with flashcards and quizzes. Medical students, as a group, were able to classify skin lesions with a higher degree of accuracy than trained dermatologists, according to his study. The key was to regularly assess each student’s performance on cases with known answers, eliminate the contributions of those who were poor at the task, and intelligently pool the contributions of those who were good at the task.
Duhaime started Centaur Labs, a firm that developed a smartphone app called DiagnosUs to gather the opinions of medical specialists on real-world scientific and biomedical data, thanks in large part to the study habits of his wife. Users can examine photos of skin lesions that could be malignant and listen to audio samples of heart and lung sounds that could suggest a problem, all within the app. Centaur incorporates user feedback and provides financial incentives for those who provide valuable insight. Companies that specialize in medical AI use patients’ feedback to enhance their own algorithms.
Companies using AI for biotech, developing drugs, or commercializing medical devices have an urgent need for well-labeled medical data, and our strategy blends that requirement with the desire of medical specialists to refine their skills.
“I realized my wife’s studying could be productive work for AI developers,” Duhaime reflects. About half of the tens of thousands of people using our app right now are medical students, who are really floored by the prospect of winning money while they study. To that end, we’ve created a gamified platform where users compete to categorize data for teams developing life-saving AI in exchange for rewards (such as money) based on their performance.
Gamifying medical labeling
Thomas Malone, the Patrick J. McGovern Professor of Management and the founding director of the Center for Collective Intelligence, advised Duhaime throughout his doctoral studies.
In his own words, “what interested me was the wisdom of crowds phenomenon,” Duhaime adds. If you poll a group of people on how many jelly beans are in a jar, you’ll get answers that are, on average, very close. In a task that calls for experience or skill, I was curious as to how you deal with such a challenge. It’s not a good idea to go around asking strangers if they think you have cancer, but we all know that second views in medicine are invaluable. Our service is essentially a turbocharged second opinion service.
Duhaime started looking into ways to use crowdsourcing to boost the accuracy of medical diagnoses. In one experiment, he found that by combining the opinions of the greatest performers, he could surpass professional dermatologists by training groups of lay people and medical school students that he characterizes as “semiexperts” to categorize skin disorders. He also discovered that he could achieve better results than with each method alone by combining computers trained to detect skin cancer with the opinions of experts.
According to Duhaime, “the central insight was that you do two things.” It may seem obvious, but measuring people’s performance isn’t done frequently, especially in the medical field. A dermatologist would respond, “Yeah, of course, I am good; I am a dermatologist.” They may not be aware of their abilities in certain areas. Second, when you obtain different perspectives, you need to figure out how they can compliment one another. Expertise is multifaceted, so assembling the finest team is more like assembling the best trivia team than selecting the five best at the same thing. Different dermatologists may have varying strengths; for instance, one may be more adept at diagnosing melanoma, while another may be more adept at grading the severity of psoriasis.
Duhaime started Centaur while he was still working on his PhD, taking advantage of the resources available to entrepreneurs at MIT. In 2017, he was awarded money from the Sandbox Innovation Fund at MIT, and in the summer of 2018, he took part in the delta v startup accelerator at the Martin Trust Center for Entrepreneurship at MIT. Later that year, he was accepted into the famed Y Combinator accelerator thanks to the experience he gained.
Duhaime, together with Centaur co-founders Zach Rausnitz and Tom Gellatly, created the DiagnosUs app for users to practice and hone their diagnostic abilities. According to Duhaime, over half of the site’s visitors are enrolled in a health professions school of some kind.
When compared to studying for tests, when multiple choice questions may be included, Duhaime claims, “it’s better.” They acquire hands-on experience through seeing real-life cases.
Centaur compiles weekly data from tens of thousands of people all across the world, totaling millions of opinions. According to Duhaime, the average user makes about as much as a barista would make in a week, while a doctor in Eastern Europe has made about $10,000.
People can do it while sitting on the couch or riding the subway, as Duhaime puts it. “It’s not like work; it’s enjoyable.”
This method is very different from the standard practice of outsourcing data labeling and AI content moderation to countries with fewer resources.
The methods used by Centaur are also reliable. Centaur published a report with researchers from Brigham and Women’s Hospital, MGH, and Eindhoven University of Technology demonstrating that its crowdsourced opinions accurately categorized lung ultrasounds. Crowdsourced classification of dermoscopic pictures was found to be more accurate than that of highly experienced dermatologists in a separate study conducted by researchers at Memorial Sloan Kettering. Centaur isn’t limited to just visual data; it can also process audio, text from sources like research articles or anonymous interactions between doctors and patients, and EEG and ECG waveforms.
Finding the experts
The top performers, Centaur has discovered, often come from unexpected backgrounds. At a conference in 2021 attended by around 50 epileptologists, each with more than 10 years of experience, researchers organized a contest using the DiagnosUs app to obtain expert comments on EEG patterns. The winner of the contest was expected to be present at the conference, so the organizers made them a special shirt.
But when the scores were tallied, medical students Jeffery Danquah and Andrews Gyabaah from Ghana surprised everyone by coming out on top. The top conference goer ended up placing seventh.
Gyabaah said to Centaur’s crew, “I started doing it for the money, but I realized it actually started helping me a lot.” at the words of one satisfied DiagnosUs user: “There were times in the clinic where I realized that I was doing better than others because of what I learned on the DiagnosUs app.”
Duhaime predicts that Centaur Labs will be utilized as a constant check on AI models as AI continues to alter the nature of work.
Duhaime says, “Right now, we’re basically just serving as the humans in the loop for a variety of tasks.” “Right now, we’re helping people train algorithms primarily,” he continues. One possible interpretation of our role is as “feedback on the outputs of the model or monitoring the model,” rather than as “a way to train AI.”
Centaur Labs, in Duhaime’s estimation, will play a significant role in the future of work integration between people and AI algorithms.
“It’s not just train algorithm, deploy algorithm,” Duhaime explains. “Instead, there will be these digital assembly lines all throughout the economy,” the author writes, “and you need on-demand expert human judgment infused at different places along the value chain.”
