CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Pseudoknotted medium-sized RNAs from RNAstrand dataset


  Only pseudoknotted RNAs from RNAstrand dataset were used to benchmark RNA secondary structure prediction methods.

Input for comparative methods consisted of a sequence collection / an alignment containing the query sequence and all sequences from a given seed alignment for an Rfam family identified for the query.
 
   Updated: Oct. 3, 2012
   Base pair definition: extended
   Type of RNA structures: pseudoknotted
   RNA sequence length range: 201, 800
   Number of RNA sequences: 610
   Robustness test: no

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 21) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

54 methods predicting RNA secondary structure were compared with each other (thus each method has 53 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 32 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Wins Losses = ? Predictions attempted Predictions generated
1 ContextFold yes 42 0 0 11 610 610
2 PETfold_pre2.0(seed) no 41 1 0 11 508 508
3 TurboFold(20) no 38 2 0 13 204 204
4 PETfold_pre2.0(20) no 36 3 1 13 204 204
4 PPfold(20) yes 36 3 1 13 204 197
6 CentroidAlifold(20) yes 35 5 0 13 204 204
7 CentroidAlifold(seed) yes 35 6 0 12 508 277
8 CentroidHomfold‑LAST yes 35 7 0 11 610 574
9 RNAalifold(20) no 32 8 0 13 204 204
10 IPknot yes 32 9 1 11 610 610
11 Multilign(20) no 30 9 1 13 204 191
12 CentroidFold yes 29 13 0 11 610 610
13 Contrafold yes 28 14 0 11 610 610
14 MXScarna(20) no 27 13 0 13 204 204
15 MXScarna(seed) no 27 14 0 12 508 282
16 MaxExpect yes 26 16 0 11 610 610
17 ProbKnot yes 25 17 0 11 610 610
18 RNASampler(20) no 23 11 0 19 204 45
19 Sfold no 23 18 1 11 610 610
20 Fold no 22 20 0 11 610 610
21 Murlet(20) yes 21 18 1 13 204 145
22 PknotsRG no 20 21 1 11 610 610
22 UNAFold no 20 21 1 11 610 610
24 RNAfold no 19 23 0 11 610 610
25 RNAsubopt no 18 24 0 11 610 574
25 Afold no 18 24 0 11 610 361
27 McQFold yes 15 27 0 11 610 610
28 CMfinder(20) yes 14 26 0 13 204 204
29 RNASLOpt no 14 28 0 11 610 480
30 RNAalifold(seed) no 12 29 0 12 508 278
31 CRWrnafold yes 12 30 0 11 610 610
31 RNAshapes no 12 30 0 11 610 526
33 RNAwolf yes 9 33 0 11 610 610
33 Vsfold4 no 9 33 0 11 610 547
35 Vsfold5 no 7 34 1 11 610 538
36 CMfinder(seed) yes 5 22 1 25 508 15
37 Carnac(20) no 5 35 0 13 204 204
38 RSpredict(20) no 4 36 0 13 204 204
39 NanoFolder yes 3 36 0 14 610 105
40 RSpredict(seed) no 3 38 0 12 508 280
41 Carnac(seed) no 0 27 1 25 508 48
42 Mastr(20) yes 0 39 1 13 204 204
43 Mastr(seed) yes 0 39 2 12 508 258
N/A Cylofold no 0 0 0 53 610 2
N/A MCFold yes 0 0 0 53 610 0
N/A RDfolder no 0 0 0 53 610 0
N/A HotKnots no 0 0 0 53 610 0
N/A Pknots no 0 0 0 53 610 0
N/A Alterna no 0 0 0 53 610 0
N/A TurboFold(seed) no 0 0 0 53 508 1
N/A Murlet(seed) yes 0 0 0 53 508 0
N/A Multilign(seed) no 0 0 0 53 508 0
N/A RNASampler(seed) no 0 0 0 53 508 0
N/A PPfold(seed) yes 0 0 0 53 508 0


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
ContextFold
PETfold_pre2.0(seed)
TurboFold(20)
PETfold_pre2.0(20)
PPfold(20)
CentroidAlifold(20)
CentroidAlifold(seed)
CentroidHomfold‑LAST
RNAalifold(20)
IPknot
Multilign(20)
CentroidFold
Contrafold
MXScarna(20)
MXScarna(seed)
MaxExpect
ProbKnot
RNASampler(20)
Sfold
Fold
Murlet(20)
UNAFold
PknotsRG
RNAfold
Afold
RNAsubopt
McQFold
CMfinder(20)
RNASLOpt
RNAalifold(seed)
RNAshapes
CRWrnafold
Vsfold4
RNAwolf
Vsfold5
CMfinder(seed)
Carnac(20)
RSpredict(20)
NanoFolder
RSpredict(seed)
Murlet(seed)
Multilign(seed)
HotKnots
Pknots
Cylofold
MCFold
PPfold(seed)
RDfolder
TurboFold(seed)
RNASampler(seed)
Alterna
Carnac(seed)
Mastr(20)
Mastr(seed)

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets).