CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of Vsfold4 - scored higher in this pairwise comparison

  4. Performance of Carnac(seed) - scored lower in this pairwise comparison

  5. Compile and download dataset for Vsfold4 & Carnac(seed) [.zip] - may take several seconds...


Overview

Metric Vsfold4 Carnac(seed)
MCC 0.222 > 0.160
Average MCC ± 95% Confidence Intervals 0.219 ± 0.024 > 0.060 ± 0.035
Sensitivity 0.205 > 0.029
Positive Predictive Value 0.243 < 0.885
Total TP 2283 > 323
Total TN 5915040 < 5924062
Total FP 7392 > 74
Total FP CONTRA 903 > 3
Total FP INCONS 6201 > 39
Total FP COMP 288 > 32
Total FN 8864 < 10824
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of Vsfold4 and Carnac(seed). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for Vsfold4 and Carnac(seed)).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for Vsfold4 and Carnac(seed)).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for Vsfold4 and Carnac(seed). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for Vsfold4 and Carnac(seed)).

^top





Performance of Vsfold4 - scored higher in this pairwise comparison

1. Total counts & total scores for Vsfold4

Total Base Pair Counts
Total TP 2283
Total TN 5915040
Total FP 7392
Total FP CONTRA 903
Total FP INCONS 6201
Total FP COMP 288
Total FN 8864
Total Scores
MCC 0.222
Average MCC ± 95% Confidence Intervals 0.219 ± 0.024
Sensitivity 0.205
Positive Predictive Value 0.243
Nr of predictions 124

^top



2. Individual counts for Vsfold4 [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00001 0.52 0.45 0.60 34 34134 27 5 18 4 41
ASE_00002 0.33 0.30 0.37 22 35451 38 2 36 0 51
ASE_00009 0.20 0.18 0.22 12 26051 43 3 40 0 55
ASE_00014 0.38 0.36 0.41 38 54192 56 7 48 1 68
ASE_00116 0.17 0.15 0.19 10 31573 43 2 41 0 56
ASE_00120 0.35 0.31 0.39 29 46897 45 3 42 0 65
ASE_00121 0.20 0.17 0.22 16 45078 58 4 52 2 76
ASE_00145 0.25 0.23 0.27 25 70033 69 3 64 2 86
ASE_00154 0.34 0.30 0.40 32 65260 50 5 44 1 75
ASE_00155 0.31 0.27 0.36 39 93419 73 2 68 3 103
ASE_00187 0.25 0.23 0.28 26 68543 67 7 59 1 89
ASE_00191 0.56 0.52 0.59 32 29592 26 0 22 4 29
ASE_00196 0.17 0.15 0.19 11 31569 47 4 42 1 64
ASE_00205 0.19 0.18 0.20 16 45976 65 6 58 1 74
ASE_00209 0.29 0.25 0.33 22 43004 45 2 43 0 66
ASE_00210 0.10 0.09 0.11 6 27207 50 7 41 2 59
ASE_00213 0.32 0.30 0.34 20 27203 40 10 28 2 46
ASE_00346 0.23 0.21 0.26 20 48128 58 3 54 1 77
ASE_00359 0.36 0.34 0.39 27 42708 43 4 39 0 53
ASE_00360 0.20 0.20 0.21 13 29342 50 10 38 2 52
ASE_00417 0.46 0.44 0.47 44 54522 50 3 46 1 56
ASE_00418 0.29 0.27 0.32 20 38719 44 0 42 2 55
CRW_00016 0.53 0.46 0.61 55 77331 44 4 31 9 65
CRW_00610 0.40 0.36 0.46 29 36252 37 5 29 3 52
CRW_00613 0.48 0.42 0.54 33 34919 34 2 26 6 45
CRW_00618 0.33 0.31 0.35 19 56225 57 6 30 21 42
CRW_00633 0.38 0.35 0.42 37 63457 52 6 46 0 69
CRW_00634 0.29 0.28 0.30 26 64533 68 8 53 7 68
CRW_00670 0.37 0.34 0.41 41 70025 60 3 56 1 79
CRW_00671 0.55 0.50 0.60 59 62736 40 2 38 0 58
CRW_00672 0.30 0.29 0.32 32 72291 71 14 53 4 79
CRW_00674 0.29 0.26 0.33 32 83339 67 8 57 2 92
CRW_00676 0.34 0.34 0.34 39 93413 80 20 56 4 76
CRW_00692 0.26 0.24 0.28 22 68555 62 20 38 4 68
PDB_00827 0.65 0.54 0.79 44 26972 13 0 12 1 37
RFA_00597 0.15 0.15 0.15 16 97796 96 11 80 5 93
RFA_00598 0.14 0.13 0.14 13 84576 88 14 63 11 88
RFA_00601 0.10 0.11 0.11 12 99122 105 21 80 4 102
RFA_00606 0.34 0.36 0.33 14 21278 40 5 24 11 25
RFA_00620 0.22 0.23 0.21 9 21903 40 11 22 7 30
RFA_00621 0.11 0.11 0.11 8 46290 71 4 58 9 68
RFA_00623 0.25 0.23 0.27 18 43889 55 5 44 6 60
RFA_00624 0.00 0.00 0.00 0 30084 53 7 44 2 61
RFA_00626 0.20 0.18 0.22 16 56543 61 11 46 4 71
RFA_00627 0.23 0.22 0.24 19 56874 63 13 47 3 68
RFA_00628 0.24 0.22 0.26 19 57218 64 8 46 10 67
RFA_00630 0.30 0.29 0.32 25 56875 58 3 50 5 62
RFA_00639 0.23 0.22 0.24 19 54537 63 12 47 4 68
RFA_00814 0.41 0.39 0.43 16 25163 33 3 18 12 25
RFA_00815 0.11 0.12 0.10 5 24483 47 6 37 4 36
RFA_00816 0.12 0.12 0.13 5 23181 42 9 25 8 36
RFA_00817 0.11 0.12 0.11 5 21898 42 16 26 0 36
RFA_00818 0.00 0.00 0.00 0 20265 46 10 26 10 41
RFA_00819 0.20 0.20 0.21 8 27928 45 6 24 15 33
SRP_00009 0.18 0.16 0.20 16 46585 64 7 57 0 86
SRP_00011 0.15 0.13 0.17 14 46277 69 9 60 0 90
SRP_00012 0.10 0.09 0.12 9 46283 68 10 58 0 92
SRP_00015 0.19 0.17 0.22 17 46281 62 6 56 0 82
SRP_00016 0.15 0.13 0.18 14 47816 66 1 64 1 95
SRP_00018 0.28 0.26 0.30 21 38433 52 6 43 3 60
SRP_00019 0.15 0.13 0.17 12 35975 60 6 53 1 79
SRP_00069 0.12 0.11 0.14 9 34389 56 3 52 1 76
SRP_00073 0.20 0.20 0.22 14 38995 52 18 33 1 57
SRP_00079 0.09 0.09 0.10 8 41827 71 4 66 1 80
SRP_00085 0.13 0.12 0.15 12 45373 66 8 58 0 89
SRP_00087 0.18 0.19 0.17 11 37337 56 19 34 3 47
SRP_00091 0.25 0.23 0.28 22 52895 59 5 53 1 73
SRP_00099 0.27 0.24 0.31 23 44776 51 7 44 0 72
SRP_00104 0.18 0.17 0.19 12 34389 55 9 43 3 57
SRP_00124 0.16 0.14 0.19 12 37066 50 5 45 0 73
SRP_00127 0.08 0.08 0.10 9 72297 84 8 76 0 110
SRP_00130 0.10 0.09 0.12 9 49695 67 7 59 1 88
SRP_00136 0.13 0.14 0.13 10 44177 72 12 54 6 61
SRP_00142 0.25 0.24 0.26 22 45971 65 6 57 2 71
SRP_00143 0.21 0.20 0.24 20 49685 65 3 62 0 81
SRP_00144 0.18 0.17 0.20 16 46584 67 5 60 2 78
SRP_00148 0.00 0.00 0.00 0 50007 84 12 67 5 87
SRP_00152 0.14 0.13 0.17 13 45676 64 11 53 0 90
SRP_00153 0.06 0.06 0.07 6 46890 75 7 68 0 101
SRP_00157 0.17 0.17 0.17 12 44184 59 14 43 2 60
SRP_00160 0.25 0.22 0.29 28 86222 70 12 58 0 102
SRP_00175 0.11 0.11 0.12 9 44476 68 10 56 2 75
SRP_00178 0.08 0.07 0.09 7 45673 73 8 65 0 95
SRP_00179 0.12 0.10 0.14 11 45977 68 9 59 0 94
SRP_00181 0.18 0.17 0.20 17 45972 67 9 58 0 85
SRP_00182 0.34 0.32 0.36 32 45968 56 10 46 0 69
SRP_00184 0.13 0.11 0.15 11 45985 60 7 53 0 89
SRP_00188 0.13 0.13 0.14 10 31055 60 6 54 0 69
SRP_00197 0.08 0.07 0.09 7 49377 71 8 63 0 94
SRP_00198 0.00 0.00 0.00 0 50960 80 8 72 0 103
SRP_00199 0.16 0.15 0.18 16 50949 75 5 70 0 89
SRP_00201 0.29 0.26 0.32 29 47805 62 5 56 1 82
SRP_00202 0.58 0.56 0.61 65 54179 41 3 38 0 52
SRP_00203 0.28 0.28 0.30 29 48730 70 6 63 1 76
SRP_00204 0.00 0.00 0.00 0 49070 72 8 63 1 102
SRP_00206 0.36 0.31 0.41 34 47812 49 3 46 0 74
SRP_00230 0.04 0.03 0.04 3 39549 69 7 62 0 87
SRP_00239 0.04 0.04 0.05 4 50317 82 12 70 0 89
SRP_00240 0.17 0.15 0.20 15 48131 59 9 50 0 87
SRP_00241 0.07 0.07 0.08 6 45978 74 13 59 2 76
SRP_00252 0.13 0.12 0.13 11 49058 75 11 61 3 78
SRP_00253 0.15 0.14 0.16 13 49376 68 7 59 2 77
SRP_00254 0.09 0.10 0.09 6 32320 59 12 47 0 55
SRP_00255 0.21 0.21 0.22 20 47495 71 10 61 0 74
SRP_00257 0.10 0.08 0.13 9 50971 60 1 59 0 99
SRP_00260 0.40 0.36 0.45 36 47815 44 4 40 0 64
SRP_00265 0.22 0.22 0.22 16 41254 59 10 48 1 57
SRP_00276 0.14 0.13 0.16 14 50632 75 9 66 0 96
SRP_00278 0.13 0.12 0.14 13 45965 78 5 73 0 92
SRP_00290 0.15 0.14 0.17 19 100466 100 5 86 9 116
SRP_00301 0.18 0.16 0.21 14 32574 54 4 48 2 72
SRP_00302 0.24 0.22 0.25 18 41545 55 10 43 2 63
SRP_00308 0.13 0.11 0.14 10 36245 60 12 48 0 78
SRP_00322 0.18 0.17 0.19 18 48423 75 5 70 0 91
SRP_00323 0.64 0.59 0.71 63 47189 28 1 25 2 44
SRP_00328 0.18 0.16 0.20 14 39550 57 11 46 0 71
SRP_00329 0.17 0.15 0.20 13 39839 51 5 46 0 72
SRP_00330 0.09 0.08 0.10 8 50005 73 8 65 0 87
SRP_00331 0.14 0.13 0.16 11 37608 56 5 51 0 76
SRP_00332 0.23 0.21 0.25 22 50316 66 2 63 1 84
SRP_00339 0.18 0.17 0.19 16 44467 68 9 59 0 78
SRP_00340 0.18 0.17 0.19 14 41256 58 11 47 0 68
SRP_00361 0.08 0.08 0.10 7 41832 69 3 63 3 83
SRP_00371 0.16 0.16 0.17 13 36239 64 12 51 1 69

^top



Performance of Carnac(seed) - scored lower in this pairwise comparison

1. Total counts & total scores for Carnac(seed)

Total Base Pair Counts
Total TP 323
Total TN 5924062
Total FP 74
Total FP CONTRA 3
Total FP INCONS 39
Total FP COMP 32
Total FN 10824
Total Scores
MCC 0.160
Average MCC ± 95% Confidence Intervals 0.060 ± 0.035
Sensitivity 0.029
Positive Predictive Value 0.885
Nr of predictions 124

^top



2. Individual counts for Carnac(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00001 0.00 0.00 0.00 0 34191 0 0 0 0 75
ASE_00002 0.00 0.00 0.00 0 35511 0 0 0 0 73
ASE_00009 0.00 0.00 0.00 0 26106 0 0 0 0 67
ASE_00014 0.00 0.00 0.00 0 54285 0 0 0 0 106
ASE_00116 0.00 0.00 0.00 0 31626 0 0 0 0 66
ASE_00120 0.00 0.00 0.00 0 46971 0 0 0 0 94
ASE_00121 0.00 0.00 0.00 0 45150 0 0 0 0 92
ASE_00145 0.00 0.00 0.00 0 70125 0 0 0 0 111
ASE_00154 0.00 0.00 0.00 0 65341 0 0 0 0 107
ASE_00155 0.00 0.00 0.00 0 93528 0 0 0 0 142
ASE_00187 0.00 0.00 0.00 0 68635 0 0 0 0 115
ASE_00191 0.00 0.00 0.00 0 29646 0 0 0 0 61
ASE_00196 0.00 0.00 0.00 0 31626 0 0 0 0 75
ASE_00205 0.00 0.00 0.00 0 46056 0 0 0 0 90
ASE_00209 0.00 0.00 0.00 0 43071 0 0 0 0 88
ASE_00210 0.00 0.00 0.00 0 27261 0 0 0 0 65
ASE_00213 0.00 0.00 0.00 0 27261 0 0 0 0 66
ASE_00346 0.00 0.00 0.00 0 48205 0 0 0 0 97
ASE_00359 0.00 0.00 0.00 0 42778 0 0 0 0 80
ASE_00360 0.00 0.00 0.00 0 29403 0 0 0 0 65
ASE_00417 0.00 0.00 0.00 0 54615 0 0 0 0 100
ASE_00418 0.00 0.00 0.00 0 38781 0 0 0 0 75
CRW_00016 0.00 0.00 0.00 0 77421 0 0 0 0 120
CRW_00610 0.00 0.00 0.00 0 36315 0 0 0 0 81
CRW_00613 0.00 0.00 0.00 0 34980 0 0 0 0 78
CRW_00618 0.00 0.00 0.00 0 56280 0 0 0 0 61
CRW_00633 0.00 0.00 0.00 0 63546 0 0 0 0 106
CRW_00634 0.00 0.00 0.00 0 64620 0 0 0 0 94
CRW_00670 0.00 0.00 0.00 0 70125 0 0 0 0 120
CRW_00671 0.00 0.00 0.00 0 62835 0 0 0 0 117
CRW_00672 0.00 0.00 0.00 0 72390 0 0 0 0 111
CRW_00674 0.00 0.00 0.00 0 83436 0 0 0 0 124
CRW_00676 0.00 0.00 0.00 0 93528 0 0 0 0 115
CRW_00692 0.00 0.00 0.00 0 68635 0 0 0 0 90
PDB_00827 0.00 0.00 0.00 0 27028 0 0 0 0 81
RFA_00597 0.00 0.00 0.00 0 97903 0 0 0 0 109
RFA_00598 0.00 0.00 0.00 0 84666 0 0 0 0 101
RFA_00601 0.00 0.00 0.00 0 99235 0 0 0 0 114
RFA_00606 0.17 0.08 0.38 3 21313 6 1 4 1 36
RFA_00620 0.13 0.08 0.23 3 21932 11 2 8 1 36
RFA_00621 0.00 0.00 0.00 0 46360 0 0 0 0 76
RFA_00623 0.00 0.00 0.00 0 43956 0 0 0 0 78
RFA_00624 0.00 0.00 0.00 0 30135 0 0 0 0 61
RFA_00626 0.67 0.52 0.87 45 56564 11 0 7 4 42
RFA_00627 0.69 0.55 0.86 48 56897 10 0 8 2 39
RFA_00628 0.66 0.43 1.00 37 57254 6 0 0 6 49
RFA_00630 0.79 0.67 0.94 58 56891 12 0 4 8 29
RFA_00639 0.00 0.00 0.00 0 54608 7 0 7 0 87
RFA_00814 0.66 0.44 1.00 18 25182 2 0 0 2 23
RFA_00815 0.75 0.56 1.00 23 24508 1 0 0 1 18
RFA_00816 0.66 0.44 1.00 18 23202 2 0 0 2 23
RFA_00817 0.80 0.66 0.96 27 21917 1 0 1 0 14
RFA_00818 0.70 0.49 1.00 20 20281 3 0 0 3 21
RFA_00819 0.75 0.56 1.00 23 27943 2 0 0 2 18
SRP_00009 0.00 0.00 0.00 0 46665 0 0 0 0 102
SRP_00011 0.00 0.00 0.00 0 46360 0 0 0 0 104
SRP_00012 0.00 0.00 0.00 0 46360 0 0 0 0 101
SRP_00015 0.00 0.00 0.00 0 46360 0 0 0 0 99
SRP_00016 0.00 0.00 0.00 0 47895 0 0 0 0 109
SRP_00018 0.00 0.00 0.00 0 38503 0 0 0 0 81
SRP_00019 0.00 0.00 0.00 0 36046 0 0 0 0 91
SRP_00069 0.00 0.00 0.00 0 34453 0 0 0 0 85
SRP_00073 0.00 0.00 0.00 0 39060 0 0 0 0 71
SRP_00079 0.00 0.00 0.00 0 41905 0 0 0 0 88
SRP_00085 0.00 0.00 0.00 0 45451 0 0 0 0 101
SRP_00087 0.00 0.00 0.00 0 37401 0 0 0 0 58
SRP_00091 0.00 0.00 0.00 0 52975 0 0 0 0 95
SRP_00099 0.00 0.00 0.00 0 44850 0 0 0 0 95
SRP_00104 0.00 0.00 0.00 0 34453 0 0 0 0 69
SRP_00124 0.00 0.00 0.00 0 37128 0 0 0 0 85
SRP_00127 0.00 0.00 0.00 0 72390 0 0 0 0 119
SRP_00130 0.00 0.00 0.00 0 49770 0 0 0 0 97
SRP_00136 0.00 0.00 0.00 0 44253 0 0 0 0 71
SRP_00142 0.00 0.00 0.00 0 46056 0 0 0 0 93
SRP_00143 0.00 0.00 0.00 0 49770 0 0 0 0 101
SRP_00144 0.00 0.00 0.00 0 46665 0 0 0 0 94
SRP_00148 0.00 0.00 0.00 0 50086 0 0 0 0 87
SRP_00152 0.00 0.00 0.00 0 45753 0 0 0 0 103
SRP_00153 0.00 0.00 0.00 0 46971 0 0 0 0 107
SRP_00157 0.00 0.00 0.00 0 44253 0 0 0 0 72
SRP_00160 0.00 0.00 0.00 0 86320 0 0 0 0 130
SRP_00175 0.00 0.00 0.00 0 44551 0 0 0 0 84
SRP_00178 0.00 0.00 0.00 0 45753 0 0 0 0 102
SRP_00179 0.00 0.00 0.00 0 46056 0 0 0 0 105
SRP_00181 0.00 0.00 0.00 0 46056 0 0 0 0 102
SRP_00182 0.00 0.00 0.00 0 46056 0 0 0 0 101
SRP_00184 0.00 0.00 0.00 0 46056 0 0 0 0 100
SRP_00188 0.00 0.00 0.00 0 31125 0 0 0 0 79
SRP_00197 0.00 0.00 0.00 0 49455 0 0 0 0 101
SRP_00198 0.00 0.00 0.00 0 51040 0 0 0 0 103
SRP_00199 0.00 0.00 0.00 0 51040 0 0 0 0 105
SRP_00201 0.00 0.00 0.00 0 47895 0 0 0 0 111
SRP_00202 0.00 0.00 0.00 0 54285 0 0 0 0 117
SRP_00203 0.00 0.00 0.00 0 48828 0 0 0 0 105
SRP_00204 0.00 0.00 0.00 0 49141 0 0 0 0 102
SRP_00206 0.00 0.00 0.00 0 47895 0 0 0 0 108
SRP_00230 0.00 0.00 0.00 0 39621 0 0 0 0 90
SRP_00239 0.00 0.00 0.00 0 50403 0 0 0 0 93
SRP_00240 0.00 0.00 0.00 0 48205 0 0 0 0 102
SRP_00241 0.00 0.00 0.00 0 46056 0 0 0 0 82
SRP_00252 0.00 0.00 0.00 0 49141 0 0 0 0 89
SRP_00253 0.00 0.00 0.00 0 49455 0 0 0 0 90
SRP_00254 0.00 0.00 0.00 0 32385 0 0 0 0 61
SRP_00255 0.00 0.00 0.00 0 47586 0 0 0 0 94
SRP_00257 0.00 0.00 0.00 0 51040 0 0 0 0 108
SRP_00260 0.00 0.00 0.00 0 47895 0 0 0 0 100
SRP_00265 0.00 0.00 0.00 0 41328 0 0 0 0 73
SRP_00276 0.00 0.00 0.00 0 50721 0 0 0 0 110
SRP_00278 0.00 0.00 0.00 0 46056 0 0 0 0 105
SRP_00290 0.00 0.00 0.00 0 100576 0 0 0 0 135
SRP_00301 0.00 0.00 0.00 0 32640 0 0 0 0 86
SRP_00302 0.00 0.00 0.00 0 41616 0 0 0 0 81
SRP_00308 0.00 0.00 0.00 0 36315 0 0 0 0 88
SRP_00322 0.00 0.00 0.00 0 48516 0 0 0 0 109
SRP_00323 0.00 0.00 0.00 0 47278 0 0 0 0 107
SRP_00328 0.00 0.00 0.00 0 39621 0 0 0 0 85
SRP_00329 0.00 0.00 0.00 0 39903 0 0 0 0 85
SRP_00330 0.00 0.00 0.00 0 50086 0 0 0 0 95
SRP_00331 0.00 0.00 0.00 0 37675 0 0 0 0 87
SRP_00332 0.00 0.00 0.00 0 50403 0 0 0 0 106
SRP_00339 0.00 0.00 0.00 0 44551 0 0 0 0 94
SRP_00340 0.00 0.00 0.00 0 41328 0 0 0 0 82
SRP_00361 0.00 0.00 0.00 0 41905 0 0 0 0 90
SRP_00371 0.00 0.00 0.00 0 36315 0 0 0 0 82

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.