CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of RNASLOpt - scored higher in this pairwise comparison

  4. Performance of Carnac(seed) - scored lower in this pairwise comparison

  5. Compile and download dataset for RNASLOpt & Carnac(seed) [.zip] - may take several seconds...


Overview

Metric RNASLOpt Carnac(seed)
MCC 0.423 > 0.167
Average MCC ± 95% Confidence Intervals 0.418 ± 0.038 > 0.064 ± 0.037
Sensitivity 0.401 > 0.032
Positive Predictive Value 0.448 < 0.885
Total TP 4086 > 323
Total TN 5176122 < 5184870
Total FP 5360 > 74
Total FP CONTRA 751 > 3
Total FP INCONS 4276 > 39
Total FP COMP 333 > 32
Total FN 6091 < 9854
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of RNASLOpt and Carnac(seed). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for RNASLOpt and Carnac(seed)).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for RNASLOpt and Carnac(seed)).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for RNASLOpt and Carnac(seed). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for RNASLOpt and Carnac(seed)).

^top





Performance of RNASLOpt - scored higher in this pairwise comparison

1. Total counts & total scores for RNASLOpt

Total Base Pair Counts
Total TP 4086
Total TN 5176122
Total FP 5360
Total FP CONTRA 751
Total FP INCONS 4276
Total FP COMP 333
Total FN 6091
Total Scores
MCC 0.423
Average MCC ± 95% Confidence Intervals 0.418 ± 0.038
Sensitivity 0.401
Positive Predictive Value 0.448
Nr of predictions 116

^top



2. Individual counts for RNASLOpt [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00001 0.55 0.49 0.61 37 34130 30 3 21 6 38
ASE_00002 0.64 0.58 0.71 42 35452 20 2 15 3 31
ASE_00009 0.44 0.40 0.47 27 26049 30 3 27 0 40
ASE_00014 0.59 0.54 0.64 57 54196 33 3 29 1 49
ASE_00116 0.24 0.21 0.29 14 31577 35 3 32 0 52
ASE_00120 0.68 0.63 0.75 59 46892 20 4 16 0 35
ASE_00121 0.46 0.42 0.50 39 45072 39 6 33 0 53
ASE_00145 0.45 0.40 0.51 44 70039 45 1 41 3 67
ASE_00154 0.41 0.36 0.45 39 65255 48 3 44 1 68
ASE_00187 0.46 0.43 0.50 49 68537 52 7 42 3 66
ASE_00191 0.58 0.56 0.61 34 29590 26 1 21 4 27
ASE_00196 0.41 0.37 0.46 28 31565 36 2 31 3 47
ASE_00205 0.43 0.41 0.46 37 45975 47 6 38 3 53
ASE_00209 0.50 0.42 0.59 37 43008 26 2 24 0 51
ASE_00210 0.36 0.34 0.38 22 27203 38 4 32 2 43
ASE_00213 0.21 0.18 0.24 12 27210 41 2 37 2 54
ASE_00346 0.53 0.47 0.61 46 48129 32 3 27 2 51
ASE_00359 0.51 0.46 0.57 37 42713 28 2 26 0 43
ASE_00360 0.43 0.38 0.49 25 29352 26 7 19 0 40
ASE_00417 0.43 0.41 0.45 41 54523 53 4 47 2 59
ASE_00418 0.07 0.07 0.07 5 38710 68 4 62 2 70
CRW_00016 0.48 0.45 0.52 54 77317 58 5 45 8 66
CRW_00610 0.06 0.06 0.07 5 36246 64 9 55 0 76
CRW_00613 0.59 0.50 0.70 39 34924 24 1 16 7 39
CRW_00618 0.21 0.21 0.21 13 56218 62 6 43 13 48
CRW_00633 0.50 0.49 0.51 52 63444 52 5 45 2 54
CRW_00634 0.50 0.50 0.51 47 64527 48 11 35 2 47
CRW_00670 0.45 0.43 0.47 52 70015 58 8 50 0 68
CRW_00671 0.60 0.53 0.68 62 62744 32 2 27 3 55
CRW_00672 0.43 0.41 0.45 45 72291 57 6 48 3 66
CRW_00692 0.33 0.34 0.33 31 68540 77 20 44 13 59
PDB_00827 0.74 0.60 0.91 49 26974 7 0 5 2 32
RFA_00606 0.18 0.18 0.18 7 21281 39 4 29 6 32
RFA_00620 0.66 0.64 0.68 25 21908 22 2 10 10 14
RFA_00621 0.23 0.22 0.24 17 46288 66 6 49 11 59
RFA_00623 0.50 0.51 0.48 40 43873 51 6 37 8 38
RFA_00624 0.52 0.51 0.53 31 30077 30 4 23 3 30
RFA_00626 0.68 0.64 0.72 56 56538 30 6 16 8 31
RFA_00627 0.61 0.60 0.62 52 56869 41 11 21 9 35
RFA_00628 0.89 0.84 0.95 72 57215 15 0 4 11 14
RFA_00630 0.69 0.67 0.72 58 56872 32 5 18 9 29
RFA_00639 0.29 0.29 0.29 25 54528 67 12 50 5 62
RFA_00814 0.51 0.51 0.51 21 25159 29 9 11 9 20
RFA_00815 0.41 0.41 0.41 17 24490 31 1 23 7 24
RFA_00816 0.58 0.63 0.53 26 23171 26 13 10 3 15
RFA_00817 0.24 0.27 0.22 11 21896 42 12 26 4 30
RFA_00818 0.21 0.22 0.21 9 20258 39 14 20 5 32
RFA_00819 0.60 0.68 0.53 28 27913 34 16 9 9 13
SRP_00009 0.50 0.47 0.54 48 46576 41 6 35 0 54
SRP_00011 0.17 0.16 0.19 17 46271 72 8 64 0 87
SRP_00012 0.32 0.29 0.36 29 46279 52 1 51 0 72
SRP_00015 0.28 0.25 0.31 25 46279 57 7 49 1 74
SRP_00016 0.74 0.71 0.77 77 47795 24 5 18 1 32
SRP_00018 0.61 0.57 0.66 46 38433 28 2 22 4 35
SRP_00019 0.19 0.16 0.22 15 35979 54 2 50 2 76
SRP_00069 0.44 0.41 0.48 35 34380 38 6 32 0 50
SRP_00073 0.43 0.42 0.43 30 38991 47 12 27 8 41
SRP_00079 0.62 0.57 0.68 50 41832 24 3 20 1 38
SRP_00085 0.55 0.50 0.60 51 45366 36 5 29 2 50
SRP_00087 0.20 0.22 0.19 13 37332 62 23 33 6 45
SRP_00091 0.71 0.68 0.75 65 52888 24 4 18 2 30
SRP_00099 0.43 0.41 0.44 39 44762 49 7 42 0 56
SRP_00104 0.30 0.32 0.29 22 34376 58 6 49 3 47
SRP_00124 0.39 0.36 0.43 31 37056 41 8 33 0 54
SRP_00127 0.39 0.38 0.41 45 72281 65 13 51 1 74
SRP_00130 0.51 0.48 0.55 47 49684 44 5 34 5 50
SRP_00136 0.29 0.31 0.27 22 44172 63 15 44 4 49
SRP_00142 0.18 0.17 0.18 16 45968 72 8 64 0 77
SRP_00143 0.65 0.62 0.68 63 49677 35 0 30 5 38
SRP_00144 0.04 0.04 0.05 4 46578 83 17 66 0 90
SRP_00148 0.24 0.23 0.25 20 50007 65 11 48 6 67
SRP_00152 0.23 0.21 0.26 22 45667 64 10 54 0 81
SRP_00153 0.33 0.29 0.38 31 46889 52 8 43 1 76
SRP_00157 0.39 0.38 0.40 27 44186 48 15 25 8 45
SRP_00175 0.11 0.11 0.11 9 44472 72 10 60 2 75
SRP_00178 0.11 0.10 0.12 10 45669 74 1 73 0 92
SRP_00179 0.65 0.59 0.72 62 45970 27 3 21 3 43
SRP_00181 0.08 0.07 0.09 7 45975 74 2 72 0 95
SRP_00182 0.54 0.51 0.57 52 45965 39 11 28 0 49
SRP_00184 0.04 0.04 0.05 4 45981 73 3 68 2 96
SRP_00188 0.18 0.18 0.20 14 31054 57 9 48 0 65
SRP_00197 0.23 0.21 0.25 21 49370 64 9 55 0 80
SRP_00198 0.41 0.38 0.45 39 50954 47 5 42 0 64
SRP_00199 0.24 0.23 0.26 24 50949 67 3 64 0 81
SRP_00201 0.73 0.72 0.74 80 47787 30 3 25 2 31
SRP_00202 0.52 0.49 0.56 57 54184 44 6 38 0 60
SRP_00203 0.65 0.64 0.66 67 48727 39 4 30 5 38
SRP_00204 0.56 0.53 0.59 54 49049 39 8 30 1 48
SRP_00206 0.84 0.80 0.90 86 47799 11 5 5 1 22
SRP_00230 0.69 0.63 0.75 57 39545 23 3 16 4 33
SRP_00239 0.00 0.00 0.00 0 50313 90 17 73 0 93
SRP_00240 0.38 0.36 0.41 37 48115 55 6 47 2 65
SRP_00241 0.26 0.27 0.26 22 45970 66 14 50 2 60
SRP_00252 0.44 0.42 0.46 37 49061 48 6 37 5 52
SRP_00253 0.23 0.22 0.23 20 49369 69 9 57 3 70
SRP_00254 0.11 0.11 0.11 7 32323 57 13 42 2 54
SRP_00255 0.29 0.30 0.29 28 47489 69 11 58 0 66
SRP_00257 0.56 0.52 0.61 56 50948 37 1 35 1 52
SRP_00260 0.69 0.66 0.72 66 47803 28 8 18 2 34
SRP_00265 0.19 0.21 0.18 15 41245 72 19 49 4 58
SRP_00276 0.66 0.63 0.70 69 50622 31 7 23 1 41
SRP_00278 0.68 0.65 0.72 68 45961 29 4 23 2 37
SRP_00301 0.55 0.50 0.60 43 32568 31 2 27 2 43
SRP_00302 0.18 0.17 0.19 14 41543 61 7 52 2 67
SRP_00308 0.14 0.14 0.15 12 36233 70 11 59 0 76
SRP_00322 0.57 0.54 0.61 59 48419 38 1 37 0 50
SRP_00323 0.75 0.69 0.82 74 47188 19 0 16 3 33
SRP_00328 0.60 0.60 0.61 51 39537 36 7 26 3 34
SRP_00329 0.34 0.32 0.36 27 39829 50 4 43 3 58
SRP_00330 0.19 0.18 0.20 17 50002 67 11 56 0 78
SRP_00331 0.15 0.13 0.17 11 37612 52 1 51 0 76
SRP_00332 0.10 0.09 0.10 10 50305 88 5 83 0 96
SRP_00339 0.44 0.43 0.45 40 44462 51 6 43 2 54
SRP_00340 0.30 0.32 0.29 26 41238 64 12 52 0 56
SRP_00361 0.15 0.14 0.16 13 41824 69 8 60 1 77
SRP_00371 0.70 0.68 0.71 56 36236 29 3 20 6 26

^top



Performance of Carnac(seed) - scored lower in this pairwise comparison

1. Total counts & total scores for Carnac(seed)

Total Base Pair Counts
Total TP 323
Total TN 5184870
Total FP 74
Total FP CONTRA 3
Total FP INCONS 39
Total FP COMP 32
Total FN 9854
Total Scores
MCC 0.167
Average MCC ± 95% Confidence Intervals 0.064 ± 0.037
Sensitivity 0.032
Positive Predictive Value 0.885
Nr of predictions 116

^top



2. Individual counts for Carnac(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00001 0.00 0.00 0.00 0 34191 0 0 0 0 75
ASE_00002 0.00 0.00 0.00 0 35511 0 0 0 0 73
ASE_00009 0.00 0.00 0.00 0 26106 0 0 0 0 67
ASE_00014 0.00 0.00 0.00 0 54285 0 0 0 0 106
ASE_00116 0.00 0.00 0.00 0 31626 0 0 0 0 66
ASE_00120 0.00 0.00 0.00 0 46971 0 0 0 0 94
ASE_00121 0.00 0.00 0.00 0 45150 0 0 0 0 92
ASE_00145 0.00 0.00 0.00 0 70125 0 0 0 0 111
ASE_00154 0.00 0.00 0.00 0 65341 0 0 0 0 107
ASE_00187 0.00 0.00 0.00 0 68635 0 0 0 0 115
ASE_00191 0.00 0.00 0.00 0 29646 0 0 0 0 61
ASE_00196 0.00 0.00 0.00 0 31626 0 0 0 0 75
ASE_00205 0.00 0.00 0.00 0 46056 0 0 0 0 90
ASE_00209 0.00 0.00 0.00 0 43071 0 0 0 0 88
ASE_00210 0.00 0.00 0.00 0 27261 0 0 0 0 65
ASE_00213 0.00 0.00 0.00 0 27261 0 0 0 0 66
ASE_00346 0.00 0.00 0.00 0 48205 0 0 0 0 97
ASE_00359 0.00 0.00 0.00 0 42778 0 0 0 0 80
ASE_00360 0.00 0.00 0.00 0 29403 0 0 0 0 65
ASE_00417 0.00 0.00 0.00 0 54615 0 0 0 0 100
ASE_00418 0.00 0.00 0.00 0 38781 0 0 0 0 75
CRW_00016 0.00 0.00 0.00 0 77421 0 0 0 0 120
CRW_00610 0.00 0.00 0.00 0 36315 0 0 0 0 81
CRW_00613 0.00 0.00 0.00 0 34980 0 0 0 0 78
CRW_00618 0.00 0.00 0.00 0 56280 0 0 0 0 61
CRW_00633 0.00 0.00 0.00 0 63546 0 0 0 0 106
CRW_00634 0.00 0.00 0.00 0 64620 0 0 0 0 94
CRW_00670 0.00 0.00 0.00 0 70125 0 0 0 0 120
CRW_00671 0.00 0.00 0.00 0 62835 0 0 0 0 117
CRW_00672 0.00 0.00 0.00 0 72390 0 0 0 0 111
CRW_00692 0.00 0.00 0.00 0 68635 0 0 0 0 90
PDB_00827 0.00 0.00 0.00 0 27028 0 0 0 0 81
RFA_00606 0.17 0.08 0.38 3 21313 6 1 4 1 36
RFA_00620 0.13 0.08 0.23 3 21932 11 2 8 1 36
RFA_00621 0.00 0.00 0.00 0 46360 0 0 0 0 76
RFA_00623 0.00 0.00 0.00 0 43956 0 0 0 0 78
RFA_00624 0.00 0.00 0.00 0 30135 0 0 0 0 61
RFA_00626 0.67 0.52 0.87 45 56564 11 0 7 4 42
RFA_00627 0.69 0.55 0.86 48 56897 10 0 8 2 39
RFA_00628 0.66 0.43 1.00 37 57254 6 0 0 6 49
RFA_00630 0.79 0.67 0.94 58 56891 12 0 4 8 29
RFA_00639 0.00 0.00 0.00 0 54608 7 0 7 0 87
RFA_00814 0.66 0.44 1.00 18 25182 2 0 0 2 23
RFA_00815 0.75 0.56 1.00 23 24508 1 0 0 1 18
RFA_00816 0.66 0.44 1.00 18 23202 2 0 0 2 23
RFA_00817 0.80 0.66 0.96 27 21917 1 0 1 0 14
RFA_00818 0.70 0.49 1.00 20 20281 3 0 0 3 21
RFA_00819 0.75 0.56 1.00 23 27943 2 0 0 2 18
SRP_00009 0.00 0.00 0.00 0 46665 0 0 0 0 102
SRP_00011 0.00 0.00 0.00 0 46360 0 0 0 0 104
SRP_00012 0.00 0.00 0.00 0 46360 0 0 0 0 101
SRP_00015 0.00 0.00 0.00 0 46360 0 0 0 0 99
SRP_00016 0.00 0.00 0.00 0 47895 0 0 0 0 109
SRP_00018 0.00 0.00 0.00 0 38503 0 0 0 0 81
SRP_00019 0.00 0.00 0.00 0 36046 0 0 0 0 91
SRP_00069 0.00 0.00 0.00 0 34453 0 0 0 0 85
SRP_00073 0.00 0.00 0.00 0 39060 0 0 0 0 71
SRP_00079 0.00 0.00 0.00 0 41905 0 0 0 0 88
SRP_00085 0.00 0.00 0.00 0 45451 0 0 0 0 101
SRP_00087 0.00 0.00 0.00 0 37401 0 0 0 0 58
SRP_00091 0.00 0.00 0.00 0 52975 0 0 0 0 95
SRP_00099 0.00 0.00 0.00 0 44850 0 0 0 0 95
SRP_00104 0.00 0.00 0.00 0 34453 0 0 0 0 69
SRP_00124 0.00 0.00 0.00 0 37128 0 0 0 0 85
SRP_00127 0.00 0.00 0.00 0 72390 0 0 0 0 119
SRP_00130 0.00 0.00 0.00 0 49770 0 0 0 0 97
SRP_00136 0.00 0.00 0.00 0 44253 0 0 0 0 71
SRP_00142 0.00 0.00 0.00 0 46056 0 0 0 0 93
SRP_00143 0.00 0.00 0.00 0 49770 0 0 0 0 101
SRP_00144 0.00 0.00 0.00 0 46665 0 0 0 0 94
SRP_00148 0.00 0.00 0.00 0 50086 0 0 0 0 87
SRP_00152 0.00 0.00 0.00 0 45753 0 0 0 0 103
SRP_00153 0.00 0.00 0.00 0 46971 0 0 0 0 107
SRP_00157 0.00 0.00 0.00 0 44253 0 0 0 0 72
SRP_00175 0.00 0.00 0.00 0 44551 0 0 0 0 84
SRP_00178 0.00 0.00 0.00 0 45753 0 0 0 0 102
SRP_00179 0.00 0.00 0.00 0 46056 0 0 0 0 105
SRP_00181 0.00 0.00 0.00 0 46056 0 0 0 0 102
SRP_00182 0.00 0.00 0.00 0 46056 0 0 0 0 101
SRP_00184 0.00 0.00 0.00 0 46056 0 0 0 0 100
SRP_00188 0.00 0.00 0.00 0 31125 0 0 0 0 79
SRP_00197 0.00 0.00 0.00 0 49455 0 0 0 0 101
SRP_00198 0.00 0.00 0.00 0 51040 0 0 0 0 103
SRP_00199 0.00 0.00 0.00 0 51040 0 0 0 0 105
SRP_00201 0.00 0.00 0.00 0 47895 0 0 0 0 111
SRP_00202 0.00 0.00 0.00 0 54285 0 0 0 0 117
SRP_00203 0.00 0.00 0.00 0 48828 0 0 0 0 105
SRP_00204 0.00 0.00 0.00 0 49141 0 0 0 0 102
SRP_00206 0.00 0.00 0.00 0 47895 0 0 0 0 108
SRP_00230 0.00 0.00 0.00 0 39621 0 0 0 0 90
SRP_00239 0.00 0.00 0.00 0 50403 0 0 0 0 93
SRP_00240 0.00 0.00 0.00 0 48205 0 0 0 0 102
SRP_00241 0.00 0.00 0.00 0 46056 0 0 0 0 82
SRP_00252 0.00 0.00 0.00 0 49141 0 0 0 0 89
SRP_00253 0.00 0.00 0.00 0 49455 0 0 0 0 90
SRP_00254 0.00 0.00 0.00 0 32385 0 0 0 0 61
SRP_00255 0.00 0.00 0.00 0 47586 0 0 0 0 94
SRP_00257 0.00 0.00 0.00 0 51040 0 0 0 0 108
SRP_00260 0.00 0.00 0.00 0 47895 0 0 0 0 100
SRP_00265 0.00 0.00 0.00 0 41328 0 0 0 0 73
SRP_00276 0.00 0.00 0.00 0 50721 0 0 0 0 110
SRP_00278 0.00 0.00 0.00 0 46056 0 0 0 0 105
SRP_00301 0.00 0.00 0.00 0 32640 0 0 0 0 86
SRP_00302 0.00 0.00 0.00 0 41616 0 0 0 0 81
SRP_00308 0.00 0.00 0.00 0 36315 0 0 0 0 88
SRP_00322 0.00 0.00 0.00 0 48516 0 0 0 0 109
SRP_00323 0.00 0.00 0.00 0 47278 0 0 0 0 107
SRP_00328 0.00 0.00 0.00 0 39621 0 0 0 0 85
SRP_00329 0.00 0.00 0.00 0 39903 0 0 0 0 85
SRP_00330 0.00 0.00 0.00 0 50086 0 0 0 0 95
SRP_00331 0.00 0.00 0.00 0 37675 0 0 0 0 87
SRP_00332 0.00 0.00 0.00 0 50403 0 0 0 0 106
SRP_00339 0.00 0.00 0.00 0 44551 0 0 0 0 94
SRP_00340 0.00 0.00 0.00 0 41328 0 0 0 0 82
SRP_00361 0.00 0.00 0.00 0 41905 0 0 0 0 90
SRP_00371 0.00 0.00 0.00 0 36315 0 0 0 0 82

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.