CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

All long RNAs from the RNAstrand dataset


  All RNAs from RNAstrand dataset were used to benchmark RNA secondary structure prediction methods.

Input for comparative methods consisted of a sequence collection / an alignment containing the query sequence and all sequences from a given seed alignment for an Rfam family identified for the query.
 
   Updated: Oct. 3, 2012
   Base pair definition: extended
   Type of RNA structures: all (including pseudoknotted)
   RNA sequence length range: 801, 30000
   Number of RNA sequences: 287
   Robustness test: no

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 21) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

54 methods predicting RNA secondary structure were compared with each other (thus each method has 53 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 32 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Wins Losses = ? Predictions attempted Predictions generated
1 PETfold_pre2.0(seed) no 24 0 0 29 83 83
2 ContextFold yes 22 1 2 28 287 287
3 CentroidAlifold(seed) yes 21 2 0 30 83 64
4 PETfold_pre2.0(20) no 21 2 1 29 77 77
5 CentroidAlifold(20) yes 20 3 1 29 77 77
6 RNAalifold(seed) no 19 5 0 29 83 68
7 RNAalifold(20) no 18 6 0 29 77 77
8 MXScarna(seed) no 16 7 0 30 83 64
9 MXScarna(20) no 16 8 0 29 77 77
10 Carnac(20) no 15 9 0 29 77 77
11 IPknot yes 15 10 0 28 287 272
12 CentroidFold yes 13 12 0 28 287 215
13 Contrafold yes 12 13 0 28 287 273
14 RSpredict(seed) no 11 12 0 30 83 57
15 RSpredict(20) no 11 13 0 29 77 77
16 Sfold no 10 15 0 28 287 273
17 MaxExpect yes 9 16 0 28 287 273
18 ProbKnot yes 8 17 0 28 287 273
19 UNAFold no 7 18 0 28 287 287
20 CRWrnafold yes 5 19 0 29 287 138
21 Afold no 5 20 0 28 287 220
22 Fold no 4 21 0 28 287 273
23 RNAfold no 2 22 1 28 287 273
24 PknotsRG no 1 22 1 29 287 168
N/A McQFold yes 0 0 0 53 287 7
N/A RNASLOpt no 0 0 0 53 287 1
N/A MCFold yes 0 0 0 53 287 0
N/A Vsfold5 no 0 0 0 53 287 0
N/A Vsfold4 no 0 0 0 53 287 0
N/A CentroidHomfold‑LAST yes 0 0 0 53 287 0
N/A RDfolder no 0 0 0 53 287 0
N/A RNAshapes no 0 0 0 53 287 0
N/A NanoFolder yes 0 0 0 53 287 0
N/A RNAsubopt no 0 0 0 53 287 0
N/A HotKnots no 0 0 0 53 287 0
N/A Pknots no 0 0 0 53 287 0
N/A Alterna no 0 0 0 53 287 0
N/A Cylofold no 0 0 0 53 287 0
N/A Mastr(seed) yes 0 0 0 53 83 3
N/A Murlet(seed) yes 0 0 0 53 83 1
N/A Carnac(seed) no 0 0 0 53 83 1
N/A TurboFold(seed) no 0 0 0 53 83 0
N/A CMfinder(seed) yes 0 0 0 53 83 0
N/A PPfold(seed) yes 0 0 0 53 83 0
N/A RNASampler(seed) no 0 0 0 53 83 0
N/A Multilign(seed) no 0 0 0 53 83 0
N/A Murlet(20) yes 0 0 0 53 77 9
N/A PPfold(20) yes 0 0 0 53 77 6
N/A CMfinder(20) yes 0 0 0 53 77 0
N/A RNASampler(20) no 0 0 0 53 77 0
N/A Multilign(20) no 0 0 0 53 77 0
N/A TurboFold(20) no 0 0 0 53 77 0
N/A RNAwolf yes 0 13 0 40 287 88
N/A Mastr(20) yes 0 19 0 34 77 27


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
PETfold_pre2.0(seed)
ContextFold
CentroidAlifold(seed)
PETfold_pre2.0(20)
CentroidAlifold(20)
RNAalifold(seed)
RNAalifold(20)
MXScarna(seed)
MXScarna(20)
Carnac(20)
IPknot
CentroidFold
Contrafold
RSpredict(seed)
RSpredict(20)
Sfold
MaxExpect
ProbKnot
UNAFold
CRWrnafold
Afold
Fold
RNAfold
PknotsRG
Murlet(seed)
McQFold
NanoFolder
CentroidHomfold‑LAST
CMfinder(seed)
RNAshapes
Multilign(seed)
Multilign(20)
TurboFold(20)
Mastr(seed)
RNAsubopt
HotKnots
Pknots
Cylofold
MCFold
Vsfold5
Vsfold4
Carnac(seed)
RNASampler(20)
CMfinder(20)
PPfold(seed)
RDfolder
TurboFold(seed)
RNASampler(seed)
RNASLOpt
Murlet(20)
Alterna
PPfold(20)
RNAwolf
Mastr(20)

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets).