Frequently Asked Questions
What does DREAM in “DREAM Challenges” mean?
DREAM is an acronym that stands for Dialogue for Reverse Engineering Assessments and Methods but over the years it has evolved into the “dream” of collaboration, sharing data, and open science being the norm in science not the exception.
Participating in DREAM Challenges
Who can participate in DREAM Challenges?
Participation in the DREAM Challenges is open to all individuals except for Challenge organizers, officers, directors, advisory board members of Challenge organizers, and individuals who report to Challenge organizers, officers, directors and advisory board members of Challenge organizers (as employees of their companies.)
Do you only do DREAM Challenges on healthcare or life sciences topics?
Our niche for DREAM Challenges is Translational Medicine and Systems Biology, but during the reorganization that DREAM has gone through, a third topic relates to “Emerging Data” and includes any kind of dataset that could be amenable to formulate a challenge. This is especially important due to the appearance of new experimental techniques but also due to the collection of large datasets. We have done challenges on many types of data including simulated networks and experimental datasets such as genomic sequences, RNA-seq datasets, phosphorylation measurements, measurement of genome-wide drug-induced effects, physiological variables, etc.
How do I find Challenge participants who have skill sets complementary to mine to form a team?
The Synapse platform provides a forum where participants can communicate to each other – this is a good way to find other participants. In addition, you can of course use any means to identify other participants.
How are the participants vetted?
The Challenge is open to anyone that is a community player. We have no restrictions on participation, but ask that participants respect the terms of data usage. If you want to help solve problems and want to work in a community of like-minded individuals, you are welcome.
How are DREAM Challenges Organized?
Dream Challenges are designed and run by a community of researchers from a variety of organizations. Do you know how many researchers and which organizations?
Many organizations that have supported the organization of Challenges in one way or another, amongst them we should highlight a few: IBM Research has supported the effort and contributed scientific leadership since the beginning of DREAM. Sage Bionetworks, which is our closest partner and besides their scientific expertise they lend IT support and share our open science philosophy. More than 50 organizations have contributed either data, funding or leadership contributing to the growth of the DREAM Challenges. Challenge organizers can be counted in the 100’s, Challenge participants are closing in the 10,000 in terms of registration and data downloads. About 30 to 50 teams submit to the final submissions for each Challenge.
The organizations that have participated in the DREAM Challenges include: Alzheimer’s Foundations, Astra Zeneca, Boston University, Broad Institute of MIT and Harvard, Columbia University, Dana Farber Cancer Institute, EPFL, European Bioinformatics Institute, Genome Institute of Singapore, Global CEO Initiative on Alzheimer’s disease, Harvard University, MD Anderson Cancer Center, Memorial Sloan Kettering Cancer Center, MIT, Mount Sinai Institute of Medicine, Nature Biotechnology, Nature Genetics, Nature Neurosciences, NCATS, NCI, NIEHS, Ontario Cancer Institute, Oregon Health and Sciences University, PLOS Computational Biology, PLOS One, Prize4Life, Project Data Sphere, Rice University, Sage Bionetworks, Sanofi Aventis, Stanford University, The Brain Tumor Society, TIGEM Telethon Institute of Genetics and Medicine, University of California, San Diego, University of Rostock, University of Toronto, University of Washington, University of Colorado, Virginia Bioinformatics Institute, and Weizmann Institute of Science
Who decides what Challenge questions to ask and how do they make those decisions?
Anyone with a good question and the data to answer that question can propose a DREAM Challenge. Ideas are vetted by a group of a group of veteran DREAM Challenge Organizers. We look for questions that are fundamental to modern biomedical and basic biology research. We also ensure that there is adequate data to address the challenge including a gold standard dataset that can be used for challenge evaluation. We also ask the participants and the community in general via forums, webinars, etc. While it may appear running a DREAM Challenge is relatively simple, there is a huge effort by many individuals to ensure the data is of high quality, we can score the Challenge, and there is interest to solve the problem from fellow DREAMers.
How are the Challenge questions publicized to the wider community in order to get to the right people?
Challenges are publicized through several outlets, including direct emails and social media notes. We post press releases and discuss the Challenges at academic conferences. Many times we partner with other institutions that also do recruitment of participants.
How long does it typically take to answer a Challenge Question?
Challenges usually from 3-6 months, but this does not always mean the Challenges are solved. We learn a great deal from the solutions to a question and the community answers to the question represent a performance benchmark that we hope will help in improving the answer to a question in the future.
Are the DREAM Challenges organized for profit?
No. DREAM Challenges are a completely not for profit effort. Commercial companies are involved in DREAM Challenges, but they are not making money, and most are making in-kind donations to the effort.
How is DREAM Challenge different from Kaggle or Innocentive?
Sage Bionetworks and DREAM Challenges are not-for-profit, while Kaggle and Innocentive are for-profit entities. Kaggle and Innocentive run challenges in which people compete to solve problems posed by industry sponsors, and in exchange for sizeable cash prizes, the solutions to the challenges are owned by the sponsoring company. The DREAM Challenges organizing team is an open community of data scientists spanning universities, researcher institutes and industry with a common goal of addressing fundamental research questions in computational/systems biology and biomedicine. The DREAM Challenges are a scientific research “competition” developed by scientists for scientists. This means that all aspects of a Challenge are fully engaged by the scientists in the research community; we go far beyond the “administration” of a challenge. We work with data providers to:
(i) develop base-line models to insure signal in the data and viability as a challenge
(ii) normalize and curate data
(iii) develop robust metrics and scoring statistics
(iv) assist with writing of manuscript from the challenge results for publication
(v) advise through all stages of the challenge, both scientific and logistical.
The ultimate goal of the DREAM Challenges is to develop a research community around challenges and advance science at a faster pace than can be done by any one individual or research group. The results of the DREAM Challenges represent a valuable benchmarked resource and the community continues to build on these resources. As reflected in article citations, there have been over 1000 publications that have been directly or indirectly spawned from the DREAM Challenges.
How can I contribute data to create a DREAM Challenge?
DREAM Challenges make new data immediately available for analysis by an extended community, and are powerful tools for assessing which data types are most relevant for the answering of specific scientific questions. Importantly, DREAM Challenges are structured to protect the right of the data producer to publish their data before others who accessed it through the Challenge. If you are interested to understand more about making data available for the running of a DREAM Challenge, please contact us here.
What is Challenge-assisted peer review?
The peer review process is the standard way to assess scientific papers however the generation of large amounts of scientific data presents a huge challenge to this approach. Indeed what is the best way to compare the many algorithmic approaches used to infer biological knowledge from these large datasets? Not all peer reviewers have the knowledge nor the access to a high performance computing infrastructure needed for a rigorous analysis. DREAM Challenges enable a more rigorous evaluation of scientific data and verification of the conclusions reached.
I am interested in becoming part of the DREAM Challenges Community, how do I get involved?
We welcome scientists to join our community and become part of our team. Contact us here with your interest and background.
I am thinking about participating
Why should I participate in a DREAM Challenge rather than one with Kaggle or Innocentive where they have “real” money I could win?
You are free to participate or not participate in any challenge-based competition you wish. DREAM Challenges aren’t about money, though occasionally cash rewards are offered. We are about building research communities and addressing fundamental research questions. By participating in DREAM Challenges you become part of a larger effort where researchers push each other to develop the optimal solution. The community is made up of researchers from around the world, thus you are provided opportunities to interact and develop relationships with world recognized scientists. Many collaborations, including jobs, have been started from DREAM Challenge participation. DREAM Challenges result in top tier publications and all community members are included as authors. DREAM Challenges also result in benchmarked methods and datasets that are continually being improved. Methods developed in challenges have real-world application, such as prediction of disease progression, prediction of drug response or development of community standards for mutation calling in cancer genomes. If would like to be part of a community that is providing service to the community via new tools, methods, and standards of performance we hope you will join us.
What do you mean by “objective scoring” and who decides how this is done?
The results from DREAM Challenges become community standards for benchmarking of both datasets and methods. In order for this to happen, the solutions to DREAM challenges must be rigorously and objectively scored. For any given computational task, there are many ways that the solution can be evaluated. The DREAM Challenges organizers take great efforts to develop the most robust and rigorous scoring approaches possible. This involves organizing a scoring committee for each challenge. This committee meets regularly via teleconference to discuss the advantages and disadvantages of an array of possible scoring metrics. The committee performs trial runs of different metrics for comparison and to ensure robustness has at least two members of the committee independently implement scoring methods. Scoring methods are developed for each DREAM Challenge independently and we solicit input on scoring from the community as a whole. The DREAM Challenges organizers want nothing more than for a challenge to be scored with the most appropriate and fair method possible. We are always open to suggestions.
Why are you asking participants to submit source code as a part of the final submission package? A central tenant of DREAM Challenges is that formalized comparison of methodologies requires transparency and reproducibility in analyses. Submission of source code allows the Challenge organizers to confirm that winning predictions can be replicated using the method described. It also allows Challenge participants the opportunity to review each other’s code.
You say that collaboration is a part of what DREAM Challenges are trying to achieve. What are the ways that this is taking place?
DREAM Challenges emphasizes collaboration to competition – our aim is to advance together science, and the challenge is an instrument to bring people together to look at important problems. Hence, while there are often rewards for best performers, we include all participants in publication and other means of acknowledgement. Furthermore, we encourage different participants to build teams, and there are no restrictions so that people that have not worked together and come from different places can come together to join a challenge.
Finally, we often propose collaborative rounds after the standard challenges, where teams can come together to solve the given problem.
What is Synapse and how does it support DREAM Challenges?
Synapse is a scientific data and analysis management platform under development by Sage Bionetworks that is provided as a free service to the scientific community for any collaborative data science project. The goal of the Synapse team is to support large scale collaborations in data-intensive science that demonstrate the power of open science to accelerate our understanding of biology. Each individual DREAM Challenge is given a working space within Synapse for its data to be securely housed, predictions to be submitted, and results provided back to the solvers. Additionally, we encourage each challenge team to create their own Synapse project to organize their work for the challenge, and to share their methods with other project teams.
What are the benefits for using leaderboards in DREAM Challenges?
The leaderboards are meant to provide participants feedback about how they are doing, and how the community is doing. The specifics of how and when leaderboards are updated are determined for each challenge, but in general we believe rapid feedback to be part of helping solvers find the best solutions to a challenge.
What happens to all of the predictions that participants submit? Are those just for DREAM or are they made available for the research community?
We score your predictions to judge your performance in the challenge as outlined in the challenge description and challenge rules. The goal of Synapse is faciitating sharing and distribution of ideas and code among participants and the research community. As described in the challenge rules, the DREAM Challenges and Sage Bionetworks make no claim to ownership of an Entry or any intellectual property that it may contain. When submitting model code, participants should provide it to under an open-source license of their choice. The license must permit the DREAM Challenges and Sage Bionetworks to distribute the code to the public for noncommercial research and development use via Synapse. Participants may keep copyright to their code submissions. If participants do not provide information on licensing, participant’s entry shall be under the FreeBSD license.
What motivates people to spend time answering a Challenge? Do they get any recognition?
Top performing teams are recognized through in several ways. There are cash incentives for some challenges, but the results are mostly academic in that the Challenges result in high impact papers. The most common answer to why researchers participate is because they are interested in addressing a fundamental challenge in the field, so participants are fantastic ambassadors of community, data-driven science.
How are their answers used? Do you have any examples?
Challenge results are always published for public consumption, including the underlying methods from participating teams. The Challenges establish a benchmark in the field that can be built on in the future. DREAM Challenges are moving toward helping in clinical practice. For example, the prognostic calculators developed for Prostate Cancer DREAM Challenge will be vetted for potential clinical use in metastatic, castrate resistance prostate cancer. Another example includes the ALS stratification challenge, where methods developed are being vetted to help in treatment for ALS patients.