CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

All medium-sized RNAs from RNAstrand dataset


  All RNAs from RNAstrand dataset were used to benchmark RNA secondary structure prediction methods.

Input for comparative methods consisted of a sequence collection / an alignment containing the query sequence and all sequences from a given seed alignment for an Rfam family identified for the query.
 
   Updated: Oct. 3, 2012
   Base pair definition: extended
   Type of RNA structures: all (including pseudoknotted)
   RNA sequence length range: 201, 800
   Number of RNA sequences: 818
   Robustness test: no

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 21) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

54 methods predicting RNA secondary structure were compared with each other (thus each method has 53 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 32 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Wins Losses = ? Predictions attempted Predictions generated
1 PETfold_pre2.0(seed) no 44 2 0 7 658 643
2 ContextFold yes 44 3 0 6 818 818
3 TurboFold(20) no 39 2 0 12 245 245
4 CentroidAlifold(seed) yes 39 6 1 7 658 426
5 PETfold_pre2.0(20) no 38 3 0 12 245 242
6 PPfold(20) yes 37 4 0 12 245 238
7 CentroidHomfold‑LAST yes 37 10 0 6 818 759
8 CentroidAlifold(20) yes 35 5 1 12 245 245
9 MXScarna(seed) no 35 11 0 7 658 431
10 IPknot yes 33 13 1 6 818 818
11 RNAalifold(20) no 32 9 0 12 245 245
12 TurboFold(seed) no 30 0 0 23 658 18
13 Multilign(20) no 29 11 0 13 245 223
14 MXScarna(20) no 29 11 1 12 245 245
15 CentroidFold yes 29 17 1 6 818 818
16 Contrafold yes 28 19 0 6 818 818
17 PPfold(seed) yes 26 4 0 23 658 23
18 Sfold no 26 20 1 6 818 818
19 MaxExpect yes 26 21 0 6 818 818
20 ProbKnot yes 25 22 0 6 818 818
21 RNASampler(seed) no 24 6 0 23 658 12
22 Fold no 23 24 0 6 818 818
23 RNASampler(20) no 22 17 1 13 245 78
24 Murlet(20) yes 22 17 2 12 245 176
25 UNAFold no 22 25 0 6 818 818
26 PknotsRG no 20 26 1 6 818 818
26 RNAfold no 20 26 1 6 818 818
28 Afold no 19 26 1 7 818 525
29 RNAsubopt no 19 28 0 6 818 759
30 RNAalifold(seed) no 18 28 0 7 658 427
31 CMfinder(20) yes 15 26 0 12 245 245
32 McQFold yes 14 33 0 6 818 818
33 CRWrnafold yes 12 34 1 6 818 818
33 RNASLOpt no 12 34 1 6 818 660
35 RNAshapes no 11 34 1 7 818 706
36 Murlet(seed) yes 10 21 1 21 658 31
37 RSpredict(20) no 10 31 0 12 245 245
38 Carnac(20) no 8 33 0 12 245 245
39 RNAwolf yes 7 40 0 6 818 818
40 CMfinder(seed) yes 6 25 0 22 658 51
41 RSpredict(seed) no 6 40 0 7 658 429
42 Vsfold4 no 6 41 0 6 818 730
43 Vsfold5 no 5 42 0 6 818 720
44 NanoFolder yes 3 39 0 11 818 134
45 Carnac(seed) no 1 42 1 9 658 145
46 Mastr(seed) yes 1 45 0 7 658 407
47 Mastr(20) yes 0 40 1 12 245 245
N/A Multilign(seed) no 19 0 0 34 658 10
N/A Cylofold no 0 0 0 53 818 2
N/A HotKnots no 0 0 0 53 818 1
N/A MCFold yes 0 0 0 53 818 0
N/A RDfolder no 0 0 0 53 818 0
N/A Pknots no 0 0 0 53 818 0
N/A Alterna no 0 0 0 53 818 0


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
PETfold_pre2.0(seed)
ContextFold
TurboFold(20)
CentroidAlifold(seed)
PETfold_pre2.0(20)
PPfold(20)
CentroidHomfold‑LAST
CentroidAlifold(20)
MXScarna(seed)
IPknot
RNAalifold(20)
TurboFold(seed)
Multilign(20)
MXScarna(20)
CentroidFold
Contrafold
PPfold(seed)
Sfold
MaxExpect
ProbKnot
RNASampler(seed)
Fold
RNASampler(20)
Murlet(20)
UNAFold
PknotsRG
RNAfold
Multilign(seed)
Afold
RNAsubopt
RNAalifold(seed)
CMfinder(20)
McQFold
CRWrnafold
RNASLOpt
RNAshapes
Murlet(seed)
RSpredict(20)
Carnac(20)
RNAwolf
CMfinder(seed)
RSpredict(seed)
Vsfold4
Vsfold5
NanoFolder
Carnac(seed)
Mastr(seed)
HotKnots
Pknots
Cylofold
MCFold
RDfolder
Alterna
Mastr(20)

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets).