CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Pseudoknotted RNAs from RNAStrand dataset


  Only pseudoknotted RNAs from RNAstrand dataset were used to benchmark RNA secondary structure prediction methods.

Input for comparative methods consisted of a sequence collection / an alignment containing the query sequence and all sequences from a given seed alignment for an Rfam family identified for the query.
 
   Updated: Oct. 3, 2012
   Base pair definition: extended
   Type of RNA structures: pseudoknotted
   RNA sequence length range: 20, 30000
   Number of RNA sequences: 919
   Robustness test: no

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 21) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

54 methods predicting RNA secondary structure were compared with each other (thus each method has 53 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 32 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Wins Losses = ? Predictions attempted Predictions generated
1 PETfold_pre2.0(seed) no 48 3 0 2 610 610
2 ContextFold yes 45 4 2 2 919 919
3 CentroidAlifold(seed) yes 44 6 1 2 610 358
4 CentroidHomfold‑LAST yes 41 10 0 2 919 627
5 TurboFold(20) no 38 2 0 13 286 209
6 PETfold_pre2.0(20) no 37 3 0 13 286 286
7 PPfold(20) yes 36 4 0 13 286 208
8 Cylofold no 35 0 0 18 919 47
9 CentroidAlifold(20) yes 34 6 0 13 286 286
10 IPknot yes 34 16 1 2 919 909
11 Pknots no 33 2 1 17 919 46
12 MXScarna(seed) no 33 18 0 2 610 363
13 RNAalifold(20) no 32 8 0 13 286 286
14 CentroidFold yes 31 20 0 2 919 862
15 RNASampler(seed) no 30 6 0 17 610 13
16 Multilign(20) no 30 9 1 13 286 196
17 RNAalifold(seed) no 30 20 1 2 610 363
18 MXScarna(20) no 27 13 0 13 286 286
18 RNASampler(20) no 27 13 0 13 286 50
20 Contrafold yes 27 22 2 2 919 910
21 Murlet(20) yes 26 14 0 13 286 159
22 Sfold no 25 25 1 2 919 910
23 ProbKnot yes 25 26 0 2 919 910
24 MaxExpect yes 24 26 1 2 919 910
25 McQFold yes 23 28 0 2 919 669
26 Murlet(seed) yes 22 14 0 17 610 14
27 Fold no 22 27 2 2 919 910
28 PknotsRG no 22 29 0 2 919 818
29 UNAFold no 21 30 0 2 919 919
30 TurboFold(seed) no 20 6 1 26 610 10
31 RNAsubopt no 20 31 0 2 919 627
32 Afold no 18 32 0 3 919 595
33 RNASLOpt no 18 32 1 2 919 534
34 RNAfold no 18 33 0 2 919 910
35 HotKnots no 17 14 5 17 919 53
36 RDfolder no 17 15 4 17 919 46
37 CMfinder(20) yes 14 26 0 13 286 209
38 RNAshapes no 14 37 0 2 919 579
39 CRWrnafold yes 13 37 1 2 919 790
40 Carnac(20) no 12 27 1 13 286 286
41 Vsfold5 no 12 39 0 2 919 591
42 Vsfold4 no 11 40 0 2 919 600
43 RNAwolf yes 10 40 1 2 919 740
44 CMfinder(seed) yes 6 27 1 19 610 26
45 Carnac(seed) no 5 31 1 16 610 62
46 RSpredict(20) no 5 34 1 13 286 286
47 Alterna no 3 32 1 17 919 46
48 NanoFolder yes 3 37 0 13 919 112
49 RSpredict(seed) no 3 47 1 2 610 354
50 Mastr(20) yes 2 38 0 13 286 236
51 Mastr(seed) yes 1 50 0 2 610 278
52 MCFold yes 0 35 0 18 919 48
N/A Multilign(seed) no 0 0 0 53 610 8
N/A PPfold(seed) yes 0 0 0 53 610 8


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
PETfold_pre2.0(seed)
ContextFold
CentroidAlifold(seed)
CentroidHomfold‑LAST
TurboFold(20)
PETfold_pre2.0(20)
PPfold(20)
Cylofold
CentroidAlifold(20)
IPknot
Pknots
MXScarna(seed)
RNAalifold(20)
CentroidFold
RNASampler(seed)
Multilign(20)
RNAalifold(seed)
RNASampler(20)
MXScarna(20)
Contrafold
Murlet(20)
Sfold
ProbKnot
MaxExpect
McQFold
Murlet(seed)
Fold
PknotsRG
UNAFold
TurboFold(seed)
RNAsubopt
Afold
RNASLOpt
RNAfold
HotKnots
RDfolder
CMfinder(20)
RNAshapes
CRWrnafold
Carnac(20)
Vsfold5
Vsfold4
RNAwolf
CMfinder(seed)
Carnac(seed)
RSpredict(20)
Alterna
NanoFolder
RSpredict(seed)
Mastr(20)
Mastr(seed)
Multilign(seed)
PPfold(seed)
MCFold

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets).