CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

All short RNAs from the RNAstrand dataset


  All RNAs from RNAstrand dataset were used to benchmark RNA secondary structure prediction methods.

Input for comparative methods consisted of a sequence collection / an alignment containing the query sequence and all sequences from a given seed alignment for an Rfam family identified for the query.
 
   Updated: Oct. 3, 2012
   Base pair definition: extended
   Type of RNA structures: all (including pseudoknotted)
   RNA sequence length range: 20, 200
   Number of RNA sequences: 882
   Robustness test: no

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 21) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

54 methods predicting RNA secondary structure were compared with each other (thus each method has 53 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 32 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Wins Losses = ? Predictions attempted Predictions generated
1 ContextFold yes 53 0 0 0 882 882
2 TurboFold(20) no 51 1 0 1 320 320
3 CentroidAlifold(seed) yes 49 3 1 0 501 216
4 CentroidHomfold‑LAST yes 49 4 0 0 882 882
4 PETfold_pre2.0(seed) no 49 4 0 0 501 498
6 TurboFold(seed) no 48 3 2 0 501 33
7 MXScarna(seed) no 47 6 0 0 501 233
8 CentroidAlifold(20) yes 45 7 0 1 320 320
9 MXScarna(20) no 43 9 0 1 320 320
9 PETfold_pre2.0(20) no 43 9 0 1 320 320
11 PPfold(20) yes 41 11 0 1 320 314
12 RSpredict(20) no 39 12 1 1 320 320
12 RNAalifold(20) no 39 12 1 1 320 320
14 IPknot yes 39 14 0 0 882 882
15 Multilign(20) no 37 15 0 1 320 311
16 CentroidFold yes 35 17 1 0 882 882
17 Contrafold yes 34 18 1 0 882 882
18 Murlet(20) yes 33 19 0 1 320 319
18 RNASampler(20) no 33 19 0 1 320 315
20 CMfinder(seed) yes 32 17 4 0 501 78
21 MaxExpect yes 32 20 1 0 882 882
22 CMfinder(20) yes 31 19 2 1 320 316
23 CRWrnafold yes 31 21 1 0 882 881
24 ProbKnot yes 29 21 3 0 882 882
25 Mastr(20) yes 27 21 4 1 320 320
26 Fold no 27 24 2 0 882 882
27 UNAFold no 26 26 1 0 882 882
28 PknotsRG no 26 27 0 0 882 882
29 Sfold no 24 29 0 0 882 882
30 RNAalifold(seed) no 23 30 0 0 501 205
31 RNAsubopt no 21 31 1 0 882 882
32 HotKnots no 20 31 2 0 882 875
33 RNAfold no 19 33 1 0 882 882
34 RNAshapes no 18 32 3 0 882 882
34 RNASampler(seed) no 18 32 3 0 501 86
36 Afold no 18 33 2 0 882 530
37 Pknots no 17 33 3 0 882 697
38 McQFold yes 16 36 1 0 882 882
39 Carnac(20) no 15 37 0 1 320 320
40 RNASLOpt no 15 38 0 0 882 882
41 Cylofold no 14 39 0 0 882 796
42 Multilign(seed) no 13 23 3 14 501 13
43 RNAwolf yes 9 43 1 0 882 882
44 PPfold(seed) yes 8 42 2 1 501 40
45 Murlet(seed) yes 8 40 5 0 501 91
46 Alterna no 7 44 2 0 882 693
47 Carnac(seed) no 7 45 1 0 501 123
48 Vsfold4 no 6 47 0 0 882 882
49 RSpredict(seed) no 5 45 3 0 501 216
50 Vsfold5 no 5 48 0 0 882 882
51 RDfolder no 4 47 2 0 882 693
52 MCFold yes 3 47 3 0 882 761
53 NanoFolder yes 2 49 0 2 882 117
54 Mastr(seed) yes 1 51 1 0 501 234


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
ContextFold
TurboFold(20)
CentroidAlifold(seed)
CentroidHomfold‑LAST
PETfold_pre2.0(seed)
TurboFold(seed)
MXScarna(seed)
CentroidAlifold(20)
PETfold_pre2.0(20)
MXScarna(20)
PPfold(20)
RNAalifold(20)
RSpredict(20)
IPknot
Multilign(20)
CentroidFold
Contrafold
RNASampler(20)
Murlet(20)
CMfinder(seed)
MaxExpect
CMfinder(20)
CRWrnafold
ProbKnot
Mastr(20)
Fold
UNAFold
PknotsRG
Sfold
RNAalifold(seed)
RNAsubopt
HotKnots
RNAfold
RNAshapes
RNASampler(seed)
Afold
Pknots
McQFold
Carnac(20)
RNASLOpt
Cylofold
Multilign(seed)
RNAwolf
PPfold(seed)
Murlet(seed)
Alterna
Carnac(seed)
Vsfold4
RSpredict(seed)
Vsfold5
RDfolder
MCFold
NanoFolder
Mastr(seed)

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets).