Lessons Learned from the CMS Artificial Intelligence Health Outcomes Challenge

On April 30, CMS announced the winner and runner-up of the CMS Artificial Intelligence (AI) Health Outcomes Challenge (“AI Challenge”), a prize competition for innovators to demonstrate how artificial intelligence tools can be used to accelerate the development of AI solutions that predict patient health events for Medicare beneficiaries for potential use by the CMS Center for Medicare and Medicaid Innovation (Innovation Center) in testing innovative payment and service delivery models. This announcement, which marked the culmination of a two-year competition, was an exciting example of how public/private partnerships can drive innovation. The AI challenge was notable for several reasons...

Strong, diverse participant interest. The AI Challenge attracted applicants from across the health care and AI communities – small and large, for-profit and non-profit, traditional players and newcomers. CMS received more than 300 applications to participate in the competition. The 25 organizations who qualified to take part in the first round included data science specialists, health systems, diversified consultants, academics, and large U.S. multinational companies. The winner of the AI Challenge, selected out of the seven finalists in the second (final) round, was ClosedLoop.ai, a health care data sciences company located in Austin, TX. The runner-up was Geisinger, an integrated health system based in Danville, PA.

Impressive participant submissions. In the second round of the competition, the seven finalists submitted predictive algorithms for 12-month mortality of Medicare beneficiaries, and further developed their first-round forecasts of unplanned hospital and skilled nursing facility admissions, and adverse outcomes. All finalist submissions demonstrated a high-level of technical expertise and creativity. The top competitors generated predictive algorithms with outstanding accuracy scores, particularly considering the limited data set used to train and test their algorithms. Accuracy metrics included industry standard measures, such as area under the ROC curve, calibration, and precision. Among the submissions, there were several excellent dashboard designs to explain predictions to clinicians and patients in an understandable, useful, and trustworthy way. All finalists addressed the issue of implicit algorithmic bias in their submissions, and proposed creative approaches to mitigating potential bias.

Beneficial public-private partnerships. The AI Challenge is an example of how public-private partnerships can work to advance the state of knowledge in a developing area, and generate solutions that can benefit Medicare beneficiaries. CMS had two partners in the AI Competition – the American Academy of Family Physicians (AAFP) and Arnold Ventures – both of which contributed prize funding. In addition, AAFP provided clinical expertise to help assess the initial 300+ applications and, in the first and second rounds of competition, evaluate how effectively the AI predictions could be explained to clinicians (aka “explainability”). Arnold Ventures also contributed its experience in supporting evidence-based health care solutions, including competition expertise.

Finalist engagement in the AI Challenge. Following the competition, we received encouraging feedback from many of the finalists:

“The real challenge of bringing the power of this technology to every health care organization and every health care decision is only beginning.”

“Working on a problem as challenging as Medicare patient outcomes inspired my team. We are already benefiting from the ideas generated during that work.”

“It was a great learning experience for our team. The CMS AI Challenge already has had many positive effects in our organization.”

“Thank you for running such a wonderful AI Health Outcomes Challenge program.”

Lessons learned and opportunities. The AI Challenge was the first prize competition operated by the CMS Innovation Center, and the Innovation Center’s first initiative focused on AI-driven health care solutions. Through the competition, we deepened our understanding of AI and its uses in health care, and our appreciation of the talented AI scientists both in and outside of the health care sector. We see the potential to use AI to help us design and implement our models, including tools to help support clinicians. We also see potential to use challenge competitions in other ways in connection with our models, with or without an AI focus.

Visit here to learn more about the AI Health Care Outcomes Challenge.

###