CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

All RNAs from the PDB dataset
standard base pair definition


  All RNAs from the PDB dataset were used to benchmark RNA secondary structure prediction methods.

Standard base pair definition is used - it corresponds to the Leontis and Westhof classification i.e. the canonical A-U, G-C and Wobble G-U pairs that belong to the cis Watson-Crick/Watson-Crick geometry are considered as the secondary structure.

 
   Updated: June 16, 2013
   Base pair definition: standard
   Type of RNA structures: all (including pseudoknotted)
   RNA sequence length range: 20, 30000
   Number of RNA sequences: 153
   Robustness test: no

Important! This ranking will be updated in future, when predictions for new RNAs are generated by the methods being tested. This ranking shows the performance of methods as of June 16, 2013.

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 23) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

58 methods predicting RNA secondary structure were compared with each other (thus each method has 57 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 34 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Tested since Wins Losses = ? Predictions attempted Predictions generated
1 CentroidAlifold(20) yes 2009-07-01 49 1 0 7 40 39
2 PETfold_pre2.0(seed) no 2011-11-03 46 0 0 11 27 27
3 ContextFold yes 2011-11-15 42 4 1 10 62 61
4 MXScarna(seed) no 2009-06-15 41 8 0 8 63 61
5 RNAalifold(20) no 2009-02-18 40 5 1 11 42 41
6 PETfold_pre2.0(20) no 2011-11-03 39 2 0 16 19 19
7 PPfold(20) yes 2011-04-18 38 3 0 16 23 21
8 IPknot yes 2011-07-01 37 11 1 8 70 69
9 MXScarna(20) no 2009-02-18 36 9 1 11 42 41
10 RNASampler(20) no 2009-02-18 36 10 0 11 42 41
11 TurboFold(20) no 2011-04-20 35 5 1 16 23 21
12 CentroidAlifold(seed) yes 2009-07-01 35 13 3 6 62 61
13 CentroidFold yes 2009-08-01 35 14 3 5 138 137
14 Sfold no 2009-02-18 34 16 0 7 153 152
15 CentroidHomfold‑LAST yes 2011-05-11 32 15 2 8 77 67
16 Contrafold yes 2009-02-18 31 17 2 7 153 152
17 RNAalifold(seed) no 2009-02-18 29 10 4 14 64 38
18 HotKnots no 2009-02-18 28 20 2 7 153 147
18 MaxExpect yes 2009-10-15 28 20 2 7 132 130
20 UNAFold no 2009-02-18 26 19 5 7 153 152
21 ProbKnot yes 2010-08-10 26 23 1 7 106 105
22 Murlet(20) yes 2009-02-18 25 16 5 11 42 41
23 PknotsRG no 2009-02-18 25 20 5 7 153 151
24 RNAfold no 2009-02-18 24 25 1 7 153 152
25 RNAshapes no 2009-02-18 22 23 5 7 153 147
26 Fold no 2010-03-15 22 25 3 7 114 113
27 TurboFold(seed) no 2011-04-20 21 7 2 27 31 15
28 Afold no 2009-02-18 20 22 1 14 153 109
29 Carnac(20) no 2009-02-18 20 21 5 11 42 41
30 RSpredict(20) no 2009-06-03 20 26 4 7 41 40
31 RNAsubopt no 2009-02-18 19 29 2 7 153 147
32 Cylofold no 2010-06-08 16 29 2 10 108 77
33 McQFold yes 2009-02-18 16 31 3 7 153 151
34 RNASLOpt no 2011-11-01 14 29 3 11 62 59
35 Pknots no 2009-02-18 14 36 0 7 153 142
36 CRWrnafold yes 2011-08-23 13 11 3 30 68 32
37 Multilign(20) no 2011-03-01 13 27 1 16 24 21
38 RNASampler(seed) no 2009-02-18 12 15 2 28 64 22
39 Alterna no 2009-02-18 12 28 2 15 153 89
40 Mastr(20) yes 2009-02-18 12 34 0 11 42 41
41 Vsfold4 no 2009-02-18 10 39 0 8 153 140
42 Murlet(seed) yes 2009-02-18 8 32 2 15 64 34
43 MCFold yes 2009-02-18 7 43 1 6 153 131
44 Vsfold5 no 2009-02-18 6 43 1 7 153 138
45 RNAwolf yes 2011-07-01 6 44 1 6 70 69
46 NanoFolder yes 2011-11-17 4 31 0 22 52 36
47 RDfolder no 2009-02-18 4 38 2 13 153 99
48 RSpredict(seed) no 2009-06-03 3 48 0 6 63 62
49 PPfold(seed) yes 2011-04-18 2 44 0 11 31 27
50 Carnac(seed) no 2009-02-18 1 46 0 10 64 39
51 Mastr(seed) yes 2009-02-18 1 48 0 8 64 62
N/A CMfinder(20) yes 2012-09-01 4 4 1 48 14 11
N/A Multilign(seed) no 2011-03-01 2 2 0 53 32 11
N/A CMfinder(seed) yes 2012-09-01 0 0 0 57 19 0
N/A DAFS(seed) yes 2012-11-20 0 0 0 57 17 0
N/A DAFS(20) yes 2012-11-20 0 0 0 57 13 0
N/A PETfold_2.0(seed) no 2013-05-20 0 0 0 57 2 2
N/A PETfold_2.0(20) no 2013-05-20 0 0 0 57 1 1


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
CentroidAlifold(20)
PETfold_pre2.0(seed)
ContextFold
MXScarna(seed)
RNAalifold(20)
PETfold_pre2.0(20)
PPfold(20)
IPknot
MXScarna(20)
RNASampler(20)
TurboFold(20)
CentroidAlifold(seed)
CentroidFold
Sfold
CentroidHomfold‑LAST
Contrafold
RNAalifold(seed)
MaxExpect
HotKnots
UNAFold
ProbKnot
Murlet(20)
PknotsRG
RNAfold
RNAshapes
Fold
TurboFold(seed)
Afold
Carnac(20)
RSpredict(20)
RNAsubopt
Cylofold
McQFold
RNASLOpt
Pknots
CRWrnafold
Multilign(20)
RNASampler(seed)
Alterna
Mastr(20)
Vsfold4
Murlet(seed)
MCFold
Vsfold5
RNAwolf
CMfinder(20)
NanoFolder
RDfolder
RSpredict(seed)
Multilign(seed)
PPfold(seed)
Carnac(seed)
Mastr(seed)
CMfinder(seed)
PETfold_2.0(seed)
DAFS(20)
PETfold_2.0(20)
DAFS(seed)

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets).