CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

All RNAs from the PDB dataset
extended base pair definition


  All RNAs from the PDB dataset were used to benchmark RNA secondary structure prediction methods.

The "extended" secondary structure definition includes all interacting bases using Watson-Crick, Hoogsteen or sugar edges, in both cis and trans orientations, including also base pairs conforming to the "standard" definition (the canonical A-U, G-C and Wobble G-U pairs that belong to the cis Watson-Crick/Watson-Crick geometry).

Input for comparative methods consisted of a sequence collection / an alignment containing the query sequence and all sequences from a given seed alignment for an Rfam family identified for the query.

 
   Updated: June 16, 2013
   Base pair definition: extended
   Type of RNA structures: all (including pseudoknotted)
   RNA sequence length range: 20, 30000
   Number of RNA sequences: 153
   Robustness test: no

Important! This ranking will be updated in future, when predictions for new RNAs are generated by the methods being tested. This ranking shows the performance of methods as of June 16, 2013.

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 23) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

58 methods predicting RNA secondary structure were compared with each other (thus each method has 57 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 34 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Tested since Wins Losses = ? Predictions attempted Predictions generated
1 CentroidAlifold(20) yes 2009-07-01 49 1 0 7 40 39
2 PETfold_pre2.0(seed) no 2011-11-03 46 0 0 11 27 27
3 ContextFold yes 2011-11-15 42 4 1 10 62 61
4 MXScarna(seed) no 2009-06-15 42 6 1 8 63 61
5 RNAalifold(20) no 2009-02-18 39 6 1 11 42 41
6 PPfold(20) yes 2011-04-18 38 2 1 16 23 21
6 PETfold_pre2.0(20) no 2011-11-03 38 2 1 16 19 19
8 IPknot yes 2011-07-01 37 11 1 8 70 69
9 CentroidAlifold(seed) yes 2009-07-01 37 12 2 6 62 61
10 RNASampler(20) no 2009-02-18 36 9 1 11 42 41
11 TurboFold(20) no 2011-04-20 35 5 1 16 23 21
12 MXScarna(20) no 2009-02-18 35 9 2 11 42 41
13 Sfold no 2009-02-18 34 16 0 7 153 152
14 CentroidHomfold‑LAST yes 2011-05-11 31 14 4 8 77 67
15 Contrafold yes 2009-02-18 31 16 3 7 153 152
16 RNAalifold(seed) no 2009-02-18 30 10 3 14 64 38
17 HotKnots no 2009-02-18 29 19 2 7 153 147
18 MaxExpect yes 2009-10-15 29 20 1 7 132 130
19 CentroidFold yes 2009-08-01 29 22 1 5 138 137
20 UNAFold no 2009-02-18 26 20 4 7 153 152
20 PknotsRG no 2009-02-18 26 20 4 7 153 151
22 ProbKnot yes 2010-08-10 26 23 1 7 106 105
23 Murlet(20) yes 2009-02-18 24 18 4 11 42 41
24 RNAfold no 2009-02-18 24 23 3 7 153 152
25 Carnac(20) no 2009-02-18 23 20 3 11 42 41
26 RNAshapes no 2009-02-18 23 26 1 7 153 147
27 Fold no 2010-03-15 22 25 3 7 114 113
28 TurboFold(seed) no 2011-04-20 20 6 4 27 31 15
29 RSpredict(20) no 2009-06-03 20 26 4 7 41 40
30 Afold no 2009-02-18 19 21 3 14 153 109
31 RNAsubopt no 2009-02-18 19 29 2 7 153 147
32 RNASLOpt no 2011-11-01 16 29 1 11 62 59
33 Cylofold no 2010-06-08 16 28 3 10 108 77
34 McQFold yes 2009-02-18 16 32 2 7 153 151
35 Multilign(20) no 2011-03-01 14 27 0 16 24 21
36 CRWrnafold yes 2011-08-23 13 11 3 30 68 32
37 Pknots no 2009-02-18 13 36 1 7 153 142
38 RNASampler(seed) no 2009-02-18 12 14 3 28 64 22
39 Alterna no 2009-02-18 12 29 1 15 153 89
40 Mastr(20) yes 2009-02-18 12 34 0 11 42 41
41 Vsfold4 no 2009-02-18 10 39 0 8 153 140
42 Murlet(seed) yes 2009-02-18 9 32 1 15 64 34
43 Vsfold5 no 2009-02-18 6 41 3 7 153 138
44 MCFold yes 2009-02-18 6 44 1 6 153 131
45 RDfolder no 2009-02-18 5 39 0 13 153 99
46 RNAwolf yes 2011-07-01 5 44 2 6 70 69
47 NanoFolder yes 2011-11-17 4 31 0 22 52 36
48 PPfold(seed) yes 2011-04-18 2 43 1 11 31 27
49 RSpredict(seed) no 2009-06-03 2 48 1 6 63 62
50 Carnac(seed) no 2009-02-18 1 46 0 10 64 39
51 Mastr(seed) yes 2009-02-18 1 48 0 8 64 62
N/A CMfinder(20) yes 2012-09-01 4 4 1 48 14 11
N/A Multilign(seed) no 2011-03-01 3 1 0 53 32 11
N/A CMfinder(seed) yes 2012-09-01 0 0 0 57 19 0
N/A DAFS(seed) yes 2012-11-20 0 0 0 57 17 0
N/A DAFS(20) yes 2012-11-20 0 0 0 57 13 0
N/A PETfold_2.0(seed) no 2013-05-20 0 0 0 57 2 2
N/A PETfold_2.0(20) no 2013-05-20 0 0 0 57 1 1


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
CentroidAlifold(20)
PETfold_pre2.0(seed)
ContextFold
MXScarna(seed)
RNAalifold(20)
PETfold_pre2.0(20)
PPfold(20)
IPknot
CentroidAlifold(seed)
RNASampler(20)
TurboFold(20)
MXScarna(20)
Sfold
CentroidHomfold‑LAST
Contrafold
RNAalifold(seed)
HotKnots
MaxExpect
CentroidFold
UNAFold
PknotsRG
ProbKnot
Murlet(20)
RNAfold
Carnac(20)
RNAshapes
Fold
TurboFold(seed)
RSpredict(20)
Afold
RNAsubopt
RNASLOpt
Cylofold
McQFold
Multilign(20)
CRWrnafold
Pknots
RNASampler(seed)
Alterna
Mastr(20)
Vsfold4
Murlet(seed)
Vsfold5
MCFold
RDfolder
RNAwolf
CMfinder(20)
NanoFolder
Multilign(seed)
PPfold(seed)
RSpredict(seed)
Carnac(seed)
Mastr(seed)
CMfinder(seed)
PETfold_2.0(seed)
DAFS(20)
PETfold_2.0(20)
DAFS(seed)

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets).