CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

All RNAs from the RNAstrand dataset


  All RNAs from RNAstrand dataset were used to benchmark RNA secondary structure prediction methods.

Input for comparative methods consisted of a sequence collection / an alignment containing the query sequence and all sequences from a given seed alignment for an Rfam family identified for the query.
 
   Updated: Oct. 3, 2012
   Base pair definition: extended
   Type of RNA structures: all (including pseudoknotted)
   RNA sequence length range: 20, 30000
   Number of RNA sequences: 1987
   Robustness test: no

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 21) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

54 methods predicting RNA secondary structure were compared with each other (thus each method has 53 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 32 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Wins Losses = ? Predictions attempted Predictions generated
1 TurboFold(seed) no 52 1 0 0 1242 51
2 ContextFold yes 51 2 0 0 1987 1987
2 PETfold_pre2.0(seed) no 51 2 0 0 1242 1224
4 TurboFold(20) no 50 2 0 1 642 565
5 CentroidAlifold(seed) yes 49 4 0 0 1242 706
6 PETfold_pre2.0(20) no 46 6 0 1 642 639
7 CentroidAlifold(20) yes 45 6 1 1 642 642
8 CentroidHomfold‑LAST yes 45 8 0 0 1987 1641
9 PPfold(20) yes 44 7 1 1 642 558
10 RNAalifold(20) no 42 10 0 1 642 642
11 MXScarna(seed) no 42 11 0 0 1242 728
12 MXScarna(20) no 41 11 0 1 642 642
13 IPknot yes 39 14 0 0 1987 1972
14 Multilign(20) no 38 14 0 1 642 534
15 Murlet(20) yes 36 15 1 1 642 504
16 RNAalifold(seed) no 36 17 0 0 1242 700
17 CentroidFold yes 35 17 1 0 1987 1915
18 RNASampler(20) no 34 18 0 1 642 393
19 Contrafold yes 34 19 0 0 1987 1973
20 Sfold no 31 22 0 0 1987 1973
21 Multilign(seed) no 30 9 0 14 1242 23
22 MaxExpect yes 30 23 0 0 1987 1973
23 ProbKnot yes 29 24 0 0 1987 1973
24 Fold no 27 25 1 0 1987 1973
25 RNASampler(seed) no 26 24 3 0 1242 98
26 UNAFold no 26 27 0 0 1987 1987
27 RNAsubopt no 25 27 1 0 1987 1641
28 RSpredict(20) no 24 26 2 1 642 642
29 CMfinder(20) yes 24 28 0 1 642 561
30 PknotsRG no 23 28 2 0 1987 1868
31 Carnac(20) no 22 30 0 1 642 642
32 RNAfold no 22 29 2 0 1987 1973
33 CMfinder(seed) yes 20 30 3 0 1242 129
34 Afold no 20 32 1 0 1987 1275
35 HotKnots no 19 31 3 0 1987 876
36 CRWrnafold yes 18 34 1 0 1987 1837
37 Pknots no 17 34 2 0 1987 697
38 McQFold yes 17 35 1 0 1987 1707
38 RNAshapes no 17 35 1 0 1987 1588
40 PPfold(seed) yes 15 37 1 0 1242 63
41 Mastr(20) yes 14 38 0 1 642 592
42 Murlet(seed) yes 14 36 3 0 1242 123
43 RNASLOpt no 14 38 1 0 1987 1543
44 Cylofold no 13 39 1 0 1987 798
45 RNAwolf yes 9 43 1 0 1987 1788
46 Vsfold4 no 9 44 0 0 1987 1612
47 Vsfold5 no 7 45 1 0 1987 1602
48 Alterna no 6 43 4 0 1987 693
49 RDfolder no 4 48 1 0 1987 693
50 Carnac(seed) no 3 46 4 0 1242 269
51 MCFold yes 3 47 3 0 1987 761
52 RSpredict(seed) no 3 48 2 0 1242 702
53 NanoFolder yes 1 50 1 1 1987 251
54 Mastr(seed) yes 0 53 0 0 1242 644


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
TurboFold(seed)
PETfold_pre2.0(seed)
ContextFold
TurboFold(20)
CentroidAlifold(seed)
PETfold_pre2.0(20)
CentroidAlifold(20)
CentroidHomfold‑LAST
PPfold(20)
RNAalifold(20)
MXScarna(seed)
MXScarna(20)
IPknot
Multilign(20)
Murlet(20)
RNAalifold(seed)
CentroidFold
RNASampler(20)
Contrafold
Sfold
Multilign(seed)
MaxExpect
ProbKnot
Fold
RNASampler(seed)
UNAFold
RNAsubopt
RSpredict(20)
CMfinder(20)
PknotsRG
Carnac(20)
RNAfold
CMfinder(seed)
Afold
HotKnots
CRWrnafold
Pknots
McQFold
RNAshapes
PPfold(seed)
Mastr(20)
Murlet(seed)
RNASLOpt
Cylofold
RNAwolf
Vsfold4
Vsfold5
Alterna
RDfolder
Carnac(seed)
MCFold
RSpredict(seed)
NanoFolder
Mastr(seed)

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets).