DREAMTools is an open-source Python package for evaluating DREAM challenge scoring metrics. DREAMTools provides a command line interface that enables researchers to test new methods on past challenges, as well as a framework for scoring new challenges.
The tool set can be downloaded here.
We developed this toolset as the diversity of challenges done by the DREAM Challenges community and the plethora of languages that have been used in past challenges has led to a fragmentation of the software designed to score submissions. In order to tackle this problem, we chose Python as a glue language. In addition to a clear syntax and the ability to scale-up software, Python can include compiled codes (e.g., Fortran and C) or call other scripting languages (Perl, R). Besides, languages such as Ruby or MATLAB can also be easily translated to Python, which was an invaluable asset to incorporate many of the earlier challenges that were originally encoded in MATLAB.
DREAMTools library framework (figure from the DREAMTools paper). DREAM challenges are described in the DREAMChallenges website where researchers can get an overview of the past and current challenges. Each challenge has its own project page within the Synape framework where details about the challenge are available. The final leaderboard showing benchmarks achieved at the end of the challenge are also shown in the Synapse project. DREAMTools provides a Python library that allows researchers to retrieve a template for each closed challenge and to easily score a prediction/template against the gold standard. In a few lines of code, the score of a prediction can then be compared to the official leaderboard, as illustrated in the example in the green box on the right hand side of the figure.
Availability of the DREAM scoring functions within DREAMTools (table from the DREAMTools paper). The first column provides the nickname used in DREAMTools to refer to a challenge. We follow the convention of the DREAM Challenges website and use a nickname that takes the form DXCY, where X is set to the DREAM version and Y is set to the challenge number. For example, the HPN-DREAM Breast Cancer challenge will be referred to as D8C1. If a challenge has sub-challenges, we will also need to provide names to identify them.
The challenge’s title (second column) and its Synapse identifier (fourth column) can be used to retrieved all details about a challenge. The third column gives the challenge status within DREAMTools: most of the challenges scoring functions are implemented in DREAMTools (green boxes); open challenges are not yet available (blue boxes); a couple of challenges did not release the gold standard and may not be implemented (red boxes labelled ’No GS’ for no gold standard); some are planned to be implemented in future releases (orange boxes labelled ’TBD’ for to be done).
To read more about the tool go to the paper here