SEATTLE, March 2, 2020 – In a study published today in the journal JAMA Network Open, researchers demonstrated that machine-learning algorithms could help improve the accuracy of breast cancer screenings when used in combination with assessments from radiologists. The study was based on results from the Digital Mammography (DM) DREAM Challenge, a crowd-sourced competition that kicked off in 2016 to engage a broad, international scientific community to assess whether artificial intelligence (AI) algorithms could meet or beat radiologist interpretive accuracy.

“This DREAM Challenge allowed for a rigorous, apples-to-apples assessment of dozens of state-of-the-art deep learning algorithms in two independent datasets,” said Dr. Justin Guinney, VP of Computational Oncology at Sage Bionetworks and Chair of DREAM Challenges. “This is a much-needed comparison effort given the importance and activity of AI research in this field.”

Conducted by IBM Research, Sage Bionetworks, Kaiser Permanente Washington Health Research Institute, and the University of Washington School of Medicine, the Digital Mammography DREAM Challenge determined that, while no single algorithm outperformed radiologists, a combination of methods in addition to assessments by radiologists helped improve overall accuracy of screenings. The research was conducted using hundreds of thousands of de-identified mammograms and clinical data from Kaiser Permanente Washington (KPWA) and the Karolinska Institute (KI) in Sweden, without releasing the data to participants.

“Our study suggests that an algorithmic combination of AI and radiologist interpretations could provide a mechanism for significantly reducing unnecessary diagnostic work-ups in the U.S. alone,” said Dr. Gustavo Stolovitzky, the Director of the IBM Translational Systems Biology and Nanobiotechnology Program at IBM’s Thomas J. Watson Research Center, and Founder of the DREAM Challenges.

To help protect data privacy and prevent participants from downloading sensitive mammography data, study organizers applied the model-to-data approach, which avoids the distribution of data to participants and mitigates the risk of sensitive patient data being released. Participants were invited to submit their algorithms to the study organizers who developed a system that automatically ran the models on the data.

“The concerns that patients feel about the use of medical images is always first in our minds. The novel model-to-data approach for data sharing is particularly innovative and essential to preserving privacy, because it allows participants to contribute innovations which might actually improve the standard of care, without receiving access to the underlying data,” said Dr. Diana Buist, of Kaiser Permanente Washington Health Research Institute and co-first author of the paper. “Also, the inclusion of data from two different countries with differing mammography screening practices highlights important translational differences in how AI could be used in different populations.”

Mammography screening is commonly used for early detection of breast cancer. While this detection tool has generally been effective, mammograms must be assessed and interpreted by a radiologist, using human visual perception to identify signs of cancer. This has led to false-positive results in an estimated 10 percent of the 40 million women who receive routine annual breast cancer screenings in the U.S.

“Based on our findings, adding AI to radiologists’ interpretation could potentially prevent hundreds of thousands of unnecessary diagnostic workups each year in the United States. Robust clinical validation is necessary, however, before any AI algorithm can be adopted broadly,” said Dr. Christoph Lee, professor of radiology at the University of Washington School of Medicine. He was the lead radiologist for the Challenge and co-first author of the paper.

Media Contacts:

RELATED CONTENT

Blog post from Dr. Diana Buist: https://www.kpwashingtonresearch.org/news-and-events/blog/2020/artificial-intelligence-aiding-not-replacing-radiologists

ABOUT IBM RESEARCH

For more than seven decades, IBM Research has defined the future of information technology, with more than 3,000 researchers in 12 labs located across six continents. Scientists from IBM Research have been awarded six Nobel prizes, a U.S. Presidential Medal of Freedom, ten U.S. National Medals of Technology, five U.S. National Medals of Science, and six Turing Awards. The teams have also included 19 inductees into the U.S. National Academy of Sciences and 20 inductees into the U.S. National Inventors Hall of Fame. For more information about IBM Research, visit www.ibm.com/research.

ABOUT SAGE BIONETWORKS

Sage Bionetworks is a nonprofit biomedical research and technology development organization that was founded in Seattle in 2009. Our focus is to develop and apply open practices to data-driven research for the advancement of human health. Our interdisciplinary team of scientists and engineers work together to provide researchers access to technology tools and scientific approaches to share data, benchmark methods, and explore collective insights, all backed by Sage’s gold-standard governance protocols and commitment to user-centered design. Sage is a 501c3 and is supported through a portfolio of competitive research grants, commercial partnerships, and philanthropic contributions.

ABOUT KAISER PERMANENTE WASHINGTON HEALTH RESEARCH INSTITUTE

Kaiser Permanente Washington Health Research Institute (KPWHRI) improves the health and health care of Kaiser Permanente members and the public. The Institute has conducted nonproprietary public-interest research on preventing, diagnosing, and treating major health problems since 1983. Government and private research grants provide our main funding.

ABOUT THE UNIVERSITY OF WASHINGTON SCHOOL OF MEDICINE

The University of Washington School of Medicine is part of the UW Medicine health system. The school educates the next generation of physicians and scientists, leads the community-based WWAMI Program that serves Washington, Wyoming, Alaska, Montana and Idaho, and was second in the nation (FY 2018) in biomedical research funding with $923.1 million in total revenue, according to the Association of American Medical Colleges.

ABOUT DREAM CHALLENGES

DREAM (Dialogue on Reverse Engineering and Assessment Methods) Challenges emerged in 2006 to leverage the wisdom of the multidisciplinary scientific community to solve fundamental and difficult questions in biomedical research. DREAM’s methodology is based on crowd-sourcing scientific Challenges, fostering open and collaborative research, and promoting data sharing. In 2013, DREAM partnered with Sage Bionetworks, which developed and administers the technology platform that underpins DREAM Challenges.