CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

All RNAs from the RNAstrand dataset
robustness test


  All RNAs from RNAstrand dataset were used to benchmark RNA secondary structure prediction methods.

Input for comparative methods consisted of a sequence collection / an alignment containing the query sequence and all sequences from a given seed alignment for an Rfam family identified for the query.

This ranking tests the robustness of methods predicting RNA secondary structure - if a given method did not return any prediction (i.e. because it crashed or couldn't be run on a sequence longer than a given threshold), it was assumed that according to that method a given RNA was single stranded. In this benchmark all 1987 sequences were included.
 
   Updated: Oct. 3, 2012
   Base pair definition: extended
   Type of RNA structures: all (including pseudoknotted)
   RNA sequence length range: 20, 30000
   Number of RNA sequences: 1987
   Robustness test: yes, on all RNAs

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 21) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

54 methods predicting RNA secondary structure were compared with each other (thus each method has 53 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 32 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Wins Losses = ? Predictions attempted Predictions generated MCC
1 ContextFold yes 53 0 0 0 1987 1987 0.680
2 IPknot yes 52 1 0 0 1987 1972 0.548
3 PETfold_pre2.0(seed) no 51 2 0 0 1987 1224 0.529
4 Contrafold yes 50 3 0 0 1987 1973 0.508
5 CentroidFold yes 49 4 0 0 1987 1915 0.470
6 Sfold no 48 5 0 0 1987 1973 0.469
7 MaxExpect yes 47 6 0 0 1987 1973 0.465
8 ProbKnot yes 46 7 0 0 1987 1973 0.460
9 UNAFold no 45 8 0 0 1987 1987 0.445
10 Fold no 43 9 1 0 1987 1973 0.437
10 CentroidAlifold(seed) yes 43 9 1 0 1987 706 0.438
12 RNAfold no 42 11 0 0 1987 1973 0.429
13 CentroidHomfold‑LAST yes 41 12 0 0 1987 1641 0.428
14 PETfold_pre2.0(20) no 40 13 0 0 1987 639 0.413
15 CentroidAlifold(20) yes 39 14 0 0 1987 642 0.411
16 PknotsRG no 37 15 1 0 1987 1868 0.386
16 MXScarna(seed) no 37 15 1 0 1987 728 0.385
18 RNAalifold(20) no 36 17 0 0 1987 642 0.381
19 RNAalifold(seed) no 35 18 0 0 1987 700 0.375
20 Afold no 33 19 1 0 1987 1275 0.358
20 MXScarna(20) no 33 19 1 0 1987 642 0.359
22 CRWrnafold yes 32 21 0 0 1987 1837 0.346
23 RNAsubopt no 31 22 0 0 1987 1641 0.336
24 McQFold yes 30 23 0 0 1987 1707 0.314
25 TurboFold(20) no 29 24 0 0 1987 565 0.301
26 PPfold(20) yes 28 25 0 0 1987 558 0.298
27 RNAshapes no 27 26 0 0 1987 1588 0.295
28 Carnac(20) no 26 27 0 0 1987 642 0.288
29 RNASLOpt no 25 28 0 0 1987 1543 0.281
30 RSpredict(20) no 24 29 0 0 1987 642 0.270
31 RNAwolf yes 22 30 1 0 1987 1788 0.251
31 Multilign(20) no 22 30 1 0 1987 534 0.251
33 Murlet(20) yes 21 32 0 0 1987 504 0.230
34 Vsfold4 no 20 33 0 0 1987 1612 0.217
35 Vsfold5 no 18 34 1 0 1987 1602 0.211
36 RSpredict(seed) no 17 34 2 0 1987 702 0.212
37 CMfinder(20) yes 17 35 1 0 1987 561 0.210
38 HotKnots no 16 37 0 0 1987 876 0.179
39 RNASampler(20) no 15 38 0 0 1987 393 0.176
40 Cylofold no 14 39 0 0 1987 798 0.155
41 Pknots no 13 40 0 0 1987 697 0.143
42 Mastr(20) yes 12 41 0 0 1987 592 0.134
43 Alterna no 11 42 0 0 1987 693 0.129
44 MCFold yes 10 43 0 0 1987 761 0.125
45 RDfolder no 9 44 0 0 1987 693 0.122
46 CMfinder(seed) yes 7 45 1 0 1987 129 0.080
46 Murlet(seed) yes 7 45 1 0 1987 123 0.079
48 TurboFold(seed) no 6 47 0 0 1987 51 0.076
49 PPfold(seed) yes 5 48 0 0 1987 63 0.069
50 Carnac(seed) no 4 49 0 0 1987 269 0.066
51 RNASampler(seed) no 3 50 0 0 1987 98 0.063
52 Mastr(seed) yes 1 51 1 0 1987 644 0.051
52 NanoFolder yes 1 51 1 0 1987 251 0.052
54 Multilign(seed) no 0 53 0 0 1987 23 0.043


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
ContextFold
IPknot
PETfold_pre2.0(seed)
Contrafold
CentroidFold
Sfold
MaxExpect
ProbKnot
UNAFold
CentroidAlifold(seed)
Fold
RNAfold
CentroidHomfold‑LAST
PETfold_pre2.0(20)
CentroidAlifold(20)
PknotsRG
MXScarna(seed)
RNAalifold(20)
RNAalifold(seed)
Afold
MXScarna(20)
CRWrnafold
RNAsubopt
McQFold
TurboFold(20)
PPfold(20)
RNAshapes
Carnac(20)
RNASLOpt
RSpredict(20)
Multilign(20)
RNAwolf
Murlet(20)
Vsfold4
Vsfold5
RSpredict(seed)
CMfinder(20)
HotKnots
RNASampler(20)
Cylofold
Pknots
Mastr(20)
Alterna
MCFold
RDfolder
Murlet(seed)
CMfinder(seed)
TurboFold(seed)
PPfold(seed)
Carnac(seed)
RNASampler(seed)
NanoFolder
Mastr(seed)
Multilign(seed)

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets). The number on the right to each bar, corresponds to the percent of RNAs for which a given method managed to generate a prediction in 24 hours when run on a single processor (2.4 GHZ).