CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

All RNAs from the PDB dataset
extended base pair definition


  All RNAs from the PDB dataset were used to benchmark RNA secondary structure prediction methods.

The "extended" secondary structure definition includes all interacting bases using Watson-Crick, Hoogsteen or sugar edges, in both cis and trans orientations, including also base pairs conforming to the "standard" definition (the canonical A-U, G-C and Wobble G-U pairs that belong to the cis Watson-Crick/Watson-Crick geometry).

Input for comparative methods consisted of a sequence collection / an alignment containing the query sequence and all sequences from a given seed alignment for an Rfam family identified for the query.

 
   Updated: Oct. 2, 2012
   Base pair definition: extended
   Type of RNA structures: all (including pseudoknotted)
   RNA sequence length range: 20, 30000
   Number of RNA sequences: 121
   Robustness test: no

Important! This ranking will be updated in future, when predictions for new RNAs are generated by the methods being tested. This ranking shows the performance of methods as of Oct. 2, 2012.

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 21) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

54 methods predicting RNA secondary structure were compared with each other (thus each method has 53 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 32 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Tested since Wins Losses = ? Predictions attempted Predictions generated
1 MXScarna(seed) no 2009-06-15 38 2 1 12 44 44
2 CentroidFold yes 2009-08-01 37 7 1 8 106 106
3 CentroidAlifold(20) yes 2009-07-01 36 0 0 17 26 26
4 RNAalifold(20) no 2009-02-18 34 2 0 17 28 28
5 RNASampler(20) no 2009-02-18 32 3 1 17 28 28
6 Sfold no 2009-02-18 32 11 2 8 121 121
7 MXScarna(20) no 2009-02-18 31 4 1 17 28 28
8 Contrafold yes 2009-02-18 31 11 3 8 121 121
8 Fold no 2010-03-15 31 11 3 8 82 82
10 HotKnots no 2009-02-18 29 13 3 8 121 118
11 CentroidHomfold‑LAST yes 2011-05-11 28 1 3 21 45 44
12 IPknot yes 2011-07-01 28 2 2 21 38 38
13 UNAFold no 2009-02-18 28 14 3 8 121 121
13 MaxExpect yes 2009-10-15 28 14 3 8 100 99
15 ContextFold yes 2011-11-15 27 0 0 26 30 30
16 RNAshapes no 2009-02-18 26 16 3 8 121 118
17 RNAfold no 2009-02-18 25 17 3 8 121 121
18 PknotsRG no 2009-02-18 24 18 3 8 121 121
19 CentroidAlifold(seed) yes 2009-07-01 23 7 11 12 43 43
20 ProbKnot yes 2010-08-10 22 20 2 9 74 74
21 Murlet(20) yes 2009-02-18 19 12 5 17 28 28
22 Carnac(20) no 2009-02-18 19 13 4 17 28 28
23 Afold no 2009-02-18 19 20 4 10 121 95
24 RNAsubopt no 2009-02-18 18 24 3 8 121 118
25 Cylofold no 2010-06-08 15 24 3 11 76 63
26 McQFold yes 2009-02-18 15 26 4 8 121 120
27 RNAalifold(seed) no 2009-02-18 14 19 3 17 45 29
28 Pknots no 2009-02-18 14 28 3 8 121 116
29 Alterna no 2009-02-18 13 28 1 11 121 89
30 RNASampler(seed) no 2009-02-18 12 12 2 27 45 17
31 CRWrnafold yes 2011-08-23 11 12 4 26 36 32
32 RSpredict(20) no 2009-06-03 11 24 1 17 27 27
33 Mastr(20) yes 2009-02-18 10 24 2 17 28 28
34 Vsfold4 no 2009-02-18 9 35 1 8 121 118
35 RNASLOpt no 2011-11-01 8 16 3 26 30 30
36 Multilign(20) no 2011-03-01 6 20 2 25 10 10
37 RNAwolf yes 2011-07-01 6 26 0 21 38 38
38 Murlet(seed) yes 2009-02-18 6 27 3 17 45 26
39 MCFold yes 2009-02-18 5 36 2 10 121 109
40 Vsfold5 no 2009-02-18 5 39 1 8 121 117
41 RDfolder no 2009-02-18 4 37 1 11 121 92
42 RSpredict(seed) no 2009-06-03 3 38 0 12 44 44
43 PPfold(seed) yes 2011-04-18 1 22 0 30 12 11
44 NanoFolder yes 2011-11-17 1 24 0 28 20 17
45 Mastr(seed) yes 2009-02-18 1 40 0 12 45 45
46 Carnac(seed) no 2009-02-18 0 36 0 17 45 30
N/A Multilign(seed) no 2011-03-01 0 0 0 53 13 5
N/A TurboFold(seed) no 2011-04-20 0 0 0 53 12 9
N/A TurboFold(20) no 2011-04-20 0 0 0 53 9 9
N/A PPfold(20) yes 2011-04-18 0 0 0 53 9 9
N/A PETfold_pre2.0(seed) no 2011-11-03 0 0 0 53 8 8
N/A PETfold_pre2.0(20) no 2011-11-03 0 0 0 53 5 5
N/A CMfinder(seed) yes 2012-09-01 0 0 0 53 0 0
N/A CMfinder(20) yes 2012-09-01 0 0 0 53 0 0


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
MXScarna(seed)
CentroidFold
CentroidAlifold(20)
RNAalifold(20)
RNASampler(20)
Sfold
MXScarna(20)
Contrafold
Fold
HotKnots
CentroidHomfold‑LAST
IPknot
MaxExpect
UNAFold
ContextFold
RNAshapes
RNAfold
PknotsRG
CentroidAlifold(seed)
ProbKnot
Murlet(20)
Carnac(20)
Afold
RNAsubopt
Cylofold
McQFold
RNAalifold(seed)
Pknots
Alterna
RNASampler(seed)
CRWrnafold
RSpredict(20)
Mastr(20)
Vsfold4
RNASLOpt
Multilign(20)
RNAwolf
Murlet(seed)
MCFold
Vsfold5
RDfolder
RSpredict(seed)
PPfold(seed)
NanoFolder
Mastr(seed)
CMfinder(seed)
Multilign(seed)
PETfold_pre2.0(seed)
TurboFold(20)
CMfinder(20)
PETfold_pre2.0(20)
TurboFold(seed)
PPfold(20)
Carnac(seed)

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets).