CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Pseudoknotted short RNAs from RNAStrand dataset


  Only pseudoknotted RNAs from RNAstrand dataset were used to benchmark RNA secondary structure prediction methods.

Input for comparative methods consisted of a sequence collection / an alignment containing the query sequence and all sequences from a given seed alignment for an Rfam family identified for the query.
 
   Updated: Oct. 3, 2012
   Base pair definition: extended
   Type of RNA structures: pseudoknotted
   RNA sequence length range: 20, 200
   Number of RNA sequences: 53
   Robustness test: no

In the summary below ranks have been assigned only to methods, for which at least 40% of comparisons with other methods is valid (i.e. at least 40% (in case of this ranking: 21) of the comparisons were based on at least 10 predictions for common targets).

The ratio 40% / 60% was chosen in order not to bias methods for which not enough predictions have been collected.


Summary of pairwise comparisons

54 methods predicting RNA secondary structure were compared with each other (thus each method has 53 comparisons with other methods (sum of values from columns "Wins", "Losses", "=" and "?" in each row)).

Legend:

= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)
N/A a method for which more than 32 (60%) of comparisons with other methods is invalid (see column "?").

Rank Method Name Trained Wins Losses = ? Predictions attempted Predictions generated
1 Cylofold no 35 0 0 18 53 45
2 McQFold yes 35 1 0 17 53 53
3 Pknots no 33 2 1 17 53 46
4 PknotsRG no 32 3 1 17 53 53
4 CentroidAlifold(seed) yes 32 3 1 17 19 17
6 PETfold_pre2.0(seed) no 31 5 0 17 19 19
7 RNASampler(seed) no 30 6 0 17 19 13
8 IPknot yes 27 8 1 17 53 53
9 RNAalifold(seed) no 27 9 0 17 19 17
10 CentroidFold yes 26 10 0 17 53 53
11 Vsfold5 no 25 9 2 17 53 53
11 CentroidHomfold‑LAST yes 25 9 2 17 53 53
13 Carnac(seed) no 22 11 3 17 19 13
14 ProbKnot yes 21 13 2 17 53 53
15 RNASLOpt no 19 12 5 17 53 53
16 Contrafold yes 18 12 6 17 53 53
17 HotKnots no 17 14 5 17 53 53
18 RDfolder no 17 15 4 17 53 46
19 Fold no 16 15 5 17 53 53
20 ContextFold yes 16 16 4 17 53 53
21 MaxExpect yes 16 18 2 17 53 53
22 Murlet(seed) yes 15 19 2 17 19 13
23 Sfold no 13 21 2 17 53 53
24 UNAFold no 11 25 0 17 53 53
25 Vsfold4 no 10 24 2 17 53 53
26 RNAshapes no 10 26 0 17 53 53
27 RNAfold no 9 26 1 17 53 53
28 MXScarna(seed) no 8 25 3 17 19 17
29 RNAsubopt no 7 27 2 17 53 53
30 RSpredict(seed) no 6 28 2 17 19 17
31 CRWrnafold yes 6 29 1 17 53 53
32 CMfinder(seed) yes 4 23 6 20 19 11
33 RNAwolf yes 4 32 0 17 53 53
34 Afold no 3 31 1 18 53 33
35 Alterna no 3 33 0 17 53 46
36 Mastr(seed) yes 1 35 0 17 19 17
37 MCFold yes 0 35 0 18 53 48
N/A NanoFolder yes 0 0 0 53 53 7
N/A TurboFold(seed) no 0 0 0 53 19 9
N/A PPfold(seed) yes 0 0 0 53 19 8
N/A Multilign(seed) no 0 0 0 53 19 8
N/A Mastr(20) yes 0 0 0 53 5 5
N/A RNASampler(20) no 0 0 0 53 5 5
N/A Carnac(20) no 0 0 0 53 5 5
N/A Murlet(20) yes 0 0 0 53 5 5
N/A Multilign(20) no 0 0 0 53 5 5
N/A RSpredict(20) no 0 0 0 53 5 5
N/A TurboFold(20) no 0 0 0 53 5 5
N/A PPfold(20) yes 0 0 0 53 5 5
N/A CMfinder(20) yes 0 0 0 53 5 5
N/A CentroidAlifold(20) yes 0 0 0 53 5 5
N/A MXScarna(20) no 0 0 0 53 5 5
N/A RNAalifold(20) no 0 0 0 53 5 5
N/A PETfold_pre2.0(20) no 0 0 0 53 5 5


Detailed results of pairwise comparisons between methods

Legend:

+ method on the left scored higher in this pairwise comparison
- method on the left scored lower in this pairwise comparison
= draw - it is assigned when both methods have generated >= 10 predictions for common targets if:
  1) the accuracies of their results are statistically not distinguishable (P-value greater than 0.001),
  or
  2) the numbers of base pairs classified to categories True Positivies (TP), True Negatives (TN), False Positivies (FP, including 3 subcategories) and False Negatives (FN) for both methods are the same.
? two methods cannot be compared ("no winner") - not enough predictions for a given pair of methods (minimum is 10)

P-values were obtained using Wilcoxon signed-rank test taking 2 sets of 40 MCC values obtained for 40 random subsets of 90% of the dataset for each pair of methods. If the P-value is lower than 0.001 and there are at least 10 sequences on which benchmark was performed, the difference between the performance of two methods is assumed to be statistically sound.



 
Cylofold
McQFold
Pknots
CentroidAlifold(seed)
PknotsRG
PETfold_pre2.0(seed)
RNASampler(seed)
IPknot
RNAalifold(seed)
CentroidFold
CentroidHomfold‑LAST
Vsfold5
Carnac(seed)
ProbKnot
RNASLOpt
Contrafold
HotKnots
RDfolder
Fold
ContextFold
MaxExpect
Murlet(seed)
Sfold
UNAFold
Vsfold4
RNAshapes
RNAfold
MXScarna(seed)
RNAsubopt
RSpredict(seed)
CRWrnafold
CMfinder(seed)
RNAwolf
Afold
Alterna
Mastr(seed)
Mastr(20)
NanoFolder
Carnac(20)
Multilign(seed)
Multilign(20)
TurboFold(20)
RNAalifold(20)
RNASampler(20)
CMfinder(20)
PETfold_pre2.0(20)
PPfold(seed)
TurboFold(seed)
MXScarna(20)
RSpredict(20)
CentroidAlifold(20)
Murlet(20)
PPfold(20)
MCFold

Matthews Correlation Coefficients (MCC) plotted for all methods in a ranking. MCCs were calculated by taking into account all reference and predicted RNA structures by a given method in the entire ranking. The plot includes only methods for which at least 40% of comparisons with other methods is valid (i.e. at least 40% of the comparisons were based on more than 10 predictions for common targets).