CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Pseudoknotted RNAs
extended base pair definition


  Only pseudoknotted RNAs from the PDB dataset were used to benchmark RNA secondary structure prediction methods.

The "extended" secondary structure definition includes all interacting bases using Watson-Crick, Hoogsteen or sugar edges, in both cis and trans orientations, including also base pairs conforming to the "standard" definition (the canonical A-U, G-C and Wobble G-U pairs that belong to the cis Watson-Crick/Watson-Crick geometry).

Input for comparative methods consisted of a sequence collection / an alignment containing the query sequence and all sequences from a given seed alignment for an Rfam family identified for the query.

 
   Updated: June 16, 2013
   Base pair definition: extended
   Type of RNA structures: pseudoknotted
   RNA sequence length range: 20, 30000
   Number of RNA sequences: 88
   Robustness test: no

Important! This ranking will be updated in future, when predictions for new RNAs are generated by the methods being tested. This ranking shows the performance of methods as of June 16, 2013.

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 23) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

58 methods predicting RNA secondary structure were compared with each other (thus each method has 57 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 34 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Tested since Wins Losses = ? Predictions attempted Predictions generated
1 CentroidAlifold(20) yes 2009-07-01 48 1 0 8 37 36
2 PETfold_pre2.0(seed) no 2011-11-03 46 0 0 11 27 27
3 RNAalifold(20) no 2009-02-18 45 5 0 7 39 38
4 MXScarna(seed) no 2009-06-15 44 6 1 6 53 51
5 PPfold(20) yes 2011-04-18 42 3 1 11 22 20
6 ContextFold yes 2011-11-15 40 7 2 8 44 43
7 RNASampler(20) no 2009-02-18 39 10 1 7 39 38
8 PETfold_pre2.0(20) no 2011-11-03 38 2 1 16 19 19
9 TurboFold(20) no 2011-04-20 38 7 1 11 22 20
10 MXScarna(20) no 2009-02-18 38 11 1 7 39 38
11 RNAalifold(seed) no 2009-02-18 37 6 3 11 54 28
12 Murlet(20) yes 2009-02-18 36 12 2 7 39 38
13 CentroidAlifold(seed) yes 2009-07-01 35 13 3 6 52 51
14 Carnac(20) no 2009-02-18 34 12 4 7 39 38
15 IPknot yes 2011-07-01 34 15 1 7 47 46
16 CentroidHomfold‑LAST yes 2011-05-11 31 17 2 7 48 39
17 Contrafold yes 2009-02-18 31 20 1 5 88 87
18 Sfold no 2009-02-18 30 19 3 5 88 87
19 HotKnots no 2009-02-18 30 21 1 5 88 82
20 CentroidFold yes 2009-08-01 28 21 3 5 82 81
21 TurboFold(seed) no 2011-04-20 27 11 3 16 30 14
22 MaxExpect yes 2009-10-15 27 21 4 5 79 77
23 RSpredict(20) no 2009-06-03 25 24 1 7 38 37
24 PknotsRG no 2009-02-18 25 23 4 5 88 86
25 CMfinder(20) yes 2012-09-01 24 14 2 17 14 11
26 UNAFold no 2009-02-18 23 27 2 5 88 87
27 ProbKnot yes 2010-08-10 22 27 3 5 62 61
28 Cylofold no 2010-06-08 21 22 4 10 63 41
29 RNAfold no 2009-02-18 21 28 3 5 88 87
30 RNASampler(seed) no 2009-02-18 20 6 6 25 54 14
31 RNAsubopt no 2009-02-18 20 30 2 5 88 82
32 Multilign(20) no 2011-03-01 19 22 2 14 22 19
33 Afold no 2009-02-18 18 20 4 15 88 54
34 RNAshapes no 2009-02-18 17 33 2 5 88 82
35 Fold no 2010-03-15 16 33 3 5 64 63
36 McQFold yes 2009-02-18 15 35 2 5 88 87
37 RNASLOpt no 2011-11-01 14 32 3 8 44 41
38 Mastr(20) yes 2009-02-18 13 34 3 7 39 38
39 Multilign(seed) no 2011-03-01 12 3 9 33 30 10
40 Pknots no 2009-02-18 12 37 3 5 88 77
41 Vsfold4 no 2009-02-18 11 40 0 6 88 75
42 CRWrnafold yes 2011-08-23 10 13 1 33 46 15
43 Murlet(seed) yes 2009-02-18 8 36 2 11 54 24
44 Vsfold5 no 2009-02-18 7 42 2 6 88 75
45 MCFold yes 2009-02-18 6 45 0 6 88 66
46 Alterna no 2009-02-18 5 27 1 24 88 35
47 RDfolder no 2009-02-18 5 36 1 15 88 42
48 RNAwolf yes 2011-07-01 5 44 1 7 47 46
49 NanoFolder yes 2011-11-17 3 39 0 15 40 26
50 Carnac(seed) no 2009-02-18 2 44 1 10 54 29
51 RSpredict(seed) no 2009-06-03 2 48 1 6 53 52
52 PPfold(seed) yes 2011-04-18 1 45 2 9 30 27
53 Mastr(seed) yes 2009-02-18 0 51 0 6 54 52
N/A CMfinder(seed) yes 2012-09-01 0 0 0 57 19 0
N/A DAFS(seed) yes 2012-11-20 0 0 0 57 17 0
N/A DAFS(20) yes 2012-11-20 0 0 0 57 13 0
N/A PETfold_2.0(seed) no 2013-05-20 0 0 0 57 2 2
N/A PETfold_2.0(20) no 2013-05-20 0 0 0 57 1 1


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
CentroidAlifold(20)
PETfold_pre2.0(seed)
RNAalifold(20)
MXScarna(seed)
PPfold(20)
ContextFold
RNASampler(20)
PETfold_pre2.0(20)
TurboFold(20)
MXScarna(20)
RNAalifold(seed)
Murlet(20)
CentroidAlifold(seed)
Carnac(20)
IPknot
CentroidHomfold‑LAST
Contrafold
Sfold
HotKnots
CentroidFold
TurboFold(seed)
MaxExpect
RSpredict(20)
PknotsRG
CMfinder(20)
UNAFold
ProbKnot
Cylofold
RNAfold
RNASampler(seed)
RNAsubopt
Multilign(20)
Afold
RNAshapes
Fold
McQFold
RNASLOpt
Mastr(20)
Multilign(seed)
Pknots
Vsfold4
CRWrnafold
Murlet(seed)
Vsfold5
MCFold
Alterna
RDfolder
RNAwolf
NanoFolder
Carnac(seed)
RSpredict(seed)
PPfold(seed)
CMfinder(seed)
PETfold_2.0(seed)
DAFS(20)
PETfold_2.0(20)
DAFS(seed)
Mastr(seed)

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets).