DREAM is a volunteer organization of more than 30,000 solvers focused on crowd-sourced challenges to benchmark informatic algorithms in biomedicine.
Crowd-sourced Challenges in Biomedicine
Crowd-sourced challenge-based benchmarking is an emerging paradigm for the development and unbiased assessment of tools and algorithms that aim to translate biological and biomedical big data into useful knowledge. By distributing problems to large communities of expert volunteers, complex biological, health care, and life sciences questions can be addressed efficiently and quickly, while incentivizing adoption of new standards. Challenges provide a successful resolution to the “self-assessment trap,” providing an objective framework for the evaluation of algorithms and methods. Today, there are a diverse set of Challenge organizations including the for-profit Kaggle and InnoCentive companies, and many academic and non-profit Challenge communities such as Critical Assessment of protein Function Annotation (CAFA), Critical Assessment of Genome Interpretation (CAGI), Critical Assessment of protein Structure Prediction (CASP) and the Dialogue on Reverse Engineering Assessment and Methods (DREAM).
About DREAM
The DREAM Challenges is a community-driven organization whose mission is to promote open, reproducible and collaborative scientific research in biomedicine. Founded in 2006 by Gustavo Stolovizky and Andrea Califano to advance methods in systems biology, DREAM today is recognized as a leader in biomedical challenges, with a proven methodology for motivating research teams to solve complex and ground-breaking problems in biology and biomedicine. Over 30,000 individuals from around the world – with vastly diverse backgrounds and multi-disciplinary interests – have participated in or benefited from our Challenges. While DREAM leverages the psychology of competition to incentivize groups, our mission is to collectively and collaboratively advance human health through a deeper understanding of biology and disease.
The outputs of our Challenges are publications, open source code, data, documentation, and benchmarks about the methods used to address the Challenge. All these materials are made open to the public and are free of intellectual property protection. The most salient lessons and insights garnered from our Challenges are published in open-access peer-reviewed journals. We emphasize biological insight and translatability as much as the performance of the algorithms used to address the questions posed in the Challenges. Elements of collaboration and education are also incorporated, and manifest as post hoc collaborative activities that include meta-analysis, integrative tool development, and co-authoring of manuscripts.
In summary, DREAM:
- addresses a broad set of questions in healthcare and the life sciences
- reaches a large, multi-disciplinary, and international community
- has a direct goal of opening data and setting field-defining standards
- ensures high-impact, collaborative, and peer-reviewed publication of results and data
- innovates in the science of benchmarking design and execution