CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of Vsfold5 - scored higher in this pairwise comparison

  4. Performance of Carnac(seed) - scored lower in this pairwise comparison

  5. Compile and download dataset for Vsfold5 & Carnac(seed) [.zip] - may take several seconds...


Overview

Metric Vsfold5 Carnac(seed)
MCC 0.210 > 0.163
Average MCC ± 95% Confidence Intervals 0.207 ± 0.021 > 0.061 ± 0.036
Sensitivity 0.201 > 0.030
Positive Predictive Value 0.221 < 0.885
Total TP 2169 > 323
Total TN 5616916 < 5626348
Total FP 7920 > 74
Total FP CONTRA 1039 > 3
Total FP INCONS 6589 > 39
Total FP COMP 292 > 32
Total FN 8620 < 10466
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of Vsfold5 and Carnac(seed). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for Vsfold5 and Carnac(seed)).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for Vsfold5 and Carnac(seed)).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for Vsfold5 and Carnac(seed). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for Vsfold5 and Carnac(seed)).

^top





Performance of Vsfold5 - scored higher in this pairwise comparison

1. Total counts & total scores for Vsfold5

Total Base Pair Counts
Total TP 2169
Total TN 5616916
Total FP 7920
Total FP CONTRA 1039
Total FP INCONS 6589
Total FP COMP 292
Total FN 8620
Total Scores
MCC 0.210
Average MCC ± 95% Confidence Intervals 0.207 ± 0.021
Sensitivity 0.201
Positive Predictive Value 0.221
Nr of predictions 121

^top



2. Individual counts for Vsfold5 [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00001 0.40 0.37 0.44 28 34127 39 6 30 3 47
ASE_00002 0.48 0.42 0.53 31 35453 31 4 23 4 42
ASE_00009 0.20 0.19 0.20 13 26042 52 0 51 1 54
ASE_00014 0.44 0.44 0.45 47 54180 59 8 50 1 59
ASE_00116 0.16 0.15 0.17 10 31566 51 5 45 1 56
ASE_00120 0.20 0.19 0.21 18 46885 68 5 63 0 76
ASE_00121 0.13 0.13 0.14 12 45065 75 9 64 2 80
ASE_00145 0.31 0.29 0.33 32 70027 67 6 60 1 79
ASE_00154 0.19 0.18 0.22 19 65253 70 4 65 1 88
ASE_00155 0.31 0.30 0.34 42 93403 86 7 76 3 100
ASE_00187 0.30 0.30 0.30 34 68523 78 11 67 0 81
ASE_00191 0.18 0.18 0.17 11 29583 53 8 44 1 50
ASE_00196 0.30 0.28 0.32 21 31561 48 4 40 4 54
ASE_00205 0.24 0.24 0.25 22 45967 68 10 57 1 68
ASE_00209 0.16 0.15 0.17 13 42993 65 5 60 0 75
ASE_00210 0.20 0.20 0.20 13 27197 53 8 43 2 52
ASE_00213 0.22 0.21 0.23 14 27200 49 10 37 2 52
ASE_00346 0.22 0.21 0.24 20 48120 66 3 62 1 77
ASE_00359 0.18 0.19 0.18 15 42695 69 10 58 1 65
ASE_00360 0.22 0.22 0.23 14 29343 46 10 36 0 51
ASE_00417 0.37 0.37 0.37 37 54514 65 13 51 1 63
ASE_00418 0.22 0.21 0.23 16 38712 58 1 52 5 59
CRW_00016 0.40 0.38 0.42 45 77315 67 4 57 6 75
CRW_00610 0.29 0.28 0.30 23 36238 57 6 48 3 58
CRW_00613 0.48 0.46 0.51 36 34909 41 2 33 6 42
CRW_00618 0.24 0.26 0.23 16 56210 76 9 45 22 45
CRW_00633 0.00 0.00 0.00 0 63546 0 0 0 0 106
CRW_00634 0.32 0.33 0.32 31 64523 75 10 56 9 63
CRW_00670 0.40 0.38 0.43 45 70020 61 3 57 1 75
CRW_00671 0.29 0.26 0.33 31 62740 64 2 62 0 86
CRW_00672 0.39 0.40 0.39 44 72278 70 14 54 2 67
CRW_00674 0.33 0.31 0.36 38 83330 74 5 63 6 86
CRW_00676 0.34 0.36 0.33 41 93405 88 19 63 6 74
CRW_00692 0.36 0.38 0.34 34 68534 67 21 46 0 56
PDB_00827 0.51 0.42 0.62 34 26973 22 0 21 1 47
RFA_00598 0.07 0.07 0.06 7 84557 111 20 82 9 94
RFA_00606 0.41 0.49 0.35 19 21267 43 15 20 8 20
RFA_00620 0.23 0.23 0.24 9 21907 41 6 23 12 30
RFA_00621 0.13 0.13 0.14 10 46288 73 5 57 11 66
RFA_00623 0.38 0.37 0.39 29 43881 53 2 44 7 49
RFA_00624 0.14 0.13 0.15 8 30082 48 5 40 3 53
RFA_00626 0.31 0.31 0.32 27 56531 62 15 43 4 60
RFA_00627 0.26 0.26 0.26 23 56863 69 16 51 2 64
RFA_00628 0.34 0.35 0.34 30 57202 66 13 46 7 56
RFA_00630 0.27 0.29 0.26 25 56857 78 13 58 7 62
RFA_00639 0.06 0.06 0.06 5 54526 88 15 69 4 82
RFA_00814 0.31 0.34 0.29 14 25151 46 10 25 11 27
RFA_00815 0.22 0.24 0.20 10 24480 46 18 23 5 31
RFA_00816 0.31 0.32 0.30 13 23177 38 12 18 8 28
RFA_00817 0.32 0.34 0.30 14 21898 35 10 23 2 27
RFA_00818 0.19 0.20 0.19 8 20258 41 8 27 6 33
RFA_00819 0.27 0.27 0.28 11 27926 53 6 23 24 30
SRP_00009 0.16 0.16 0.18 16 46574 75 9 66 0 86
SRP_00011 0.05 0.05 0.06 5 46277 78 8 70 0 99
SRP_00012 0.08 0.08 0.09 8 46275 77 7 70 0 93
SRP_00015 0.15 0.14 0.17 14 46276 72 10 60 2 85
SRP_00016 0.14 0.13 0.15 14 47800 82 3 78 1 95
SRP_00018 0.23 0.22 0.25 18 38430 57 5 50 2 63
SRP_00019 0.26 0.24 0.28 22 35968 57 6 50 1 69
SRP_00069 0.00 0.00 0.00 0 34377 76 2 74 0 85
SRP_00073 0.26 0.27 0.25 19 38984 59 18 39 2 52
SRP_00079 0.14 0.14 0.15 12 41823 72 4 66 2 76
SRP_00085 0.18 0.17 0.20 17 45366 70 5 63 2 84
SRP_00087 0.18 0.21 0.16 12 37327 65 19 43 3 46
SRP_00091 0.29 0.28 0.30 27 52885 65 6 57 2 68
SRP_00099 0.07 0.06 0.08 6 44770 74 8 66 0 89
SRP_00104 0.14 0.14 0.14 10 34379 66 12 52 2 59
SRP_00124 0.00 0.00 0.00 0 37128 0 0 0 0 85
SRP_00127 0.13 0.13 0.14 15 72281 95 7 87 1 104
SRP_00130 0.11 0.10 0.12 10 49686 75 10 64 1 87
SRP_00136 0.16 0.18 0.15 13 44164 77 23 53 1 58
SRP_00142 0.25 0.26 0.25 24 45959 75 10 63 2 69
SRP_00143 0.20 0.20 0.20 20 49668 82 4 78 0 81
SRP_00144 0.15 0.14 0.16 13 46582 71 11 59 1 81
SRP_00148 0.06 0.06 0.06 5 50006 81 8 67 6 82
SRP_00152 0.00 0.00 0.00 0 45753 0 0 0 0 103
SRP_00153 0.11 0.10 0.12 11 46876 84 13 71 0 96
SRP_00157 0.11 0.13 0.11 9 44169 76 19 56 1 63
SRP_00160 0.17 0.16 0.18 21 86201 99 15 83 1 109
SRP_00175 0.13 0.13 0.14 11 44470 70 14 56 0 73
SRP_00178 0.07 0.07 0.08 7 45668 78 9 69 0 95
SRP_00179 0.00 0.00 0.00 0 46056 0 0 0 0 105
SRP_00181 0.07 0.07 0.08 7 45963 86 6 80 0 95
SRP_00182 0.16 0.16 0.17 16 45964 76 7 69 0 85
SRP_00184 0.07 0.07 0.08 7 45963 87 11 75 1 93
SRP_00188 0.08 0.09 0.08 7 31041 77 13 64 0 72
SRP_00197 0.23 0.23 0.24 23 49359 73 9 64 0 78
SRP_00198 0.11 0.11 0.13 11 50953 76 11 65 0 92
SRP_00199 0.24 0.24 0.25 25 50938 77 10 67 0 80
SRP_00201 0.27 0.26 0.28 29 47792 74 12 62 0 82
SRP_00202 0.51 0.50 0.53 59 54173 53 8 45 0 58
SRP_00203 0.19 0.19 0.19 20 48725 84 4 79 1 85
SRP_00204 0.06 0.06 0.07 6 49055 80 11 69 0 96
SRP_00206 0.22 0.20 0.24 22 47803 74 5 65 4 86
SRP_00230 0.12 0.11 0.13 10 39543 70 3 65 2 80
SRP_00239 0.04 0.04 0.04 4 50301 98 18 80 0 89
SRP_00240 0.21 0.20 0.22 20 48114 71 10 61 0 82
SRP_00241 0.07 0.07 0.07 6 45966 85 13 71 1 76
SRP_00252 0.18 0.18 0.17 16 49049 79 7 69 3 73
SRP_00253 0.00 0.00 0.00 0 49361 96 14 80 2 90
SRP_00254 0.24 0.25 0.24 15 32322 49 12 36 1 46
SRP_00255 0.22 0.22 0.22 21 47490 75 8 67 0 73
SRP_00257 0.30 0.27 0.34 29 50955 58 2 54 2 79
SRP_00260 0.12 0.11 0.13 11 47812 72 7 65 0 89
SRP_00265 0.24 0.25 0.23 18 41251 63 9 50 4 55
SRP_00276 0.25 0.25 0.27 27 50620 75 8 66 1 83
SRP_00278 0.17 0.16 0.17 17 45958 81 7 74 0 88
SRP_00301 0.11 0.10 0.12 9 32566 65 5 60 0 77
SRP_00302 0.25 0.23 0.26 19 41543 57 6 48 3 62
SRP_00308 0.29 0.27 0.32 24 36239 52 11 41 0 64
SRP_00322 0.16 0.16 0.17 17 48417 82 4 78 0 92
SRP_00323 0.23 0.21 0.25 23 47187 69 2 66 1 84
SRP_00328 0.19 0.18 0.21 15 39548 60 10 48 2 70
SRP_00329 0.23 0.22 0.25 19 39826 58 15 43 0 66
SRP_00330 0.12 0.13 0.12 12 49989 85 13 72 0 83
SRP_00331 0.19 0.18 0.21 16 37597 62 7 55 0 71
SRP_00332 0.29 0.29 0.29 31 50297 75 8 67 0 75
SRP_00339 0.19 0.18 0.19 17 44463 71 13 58 0 77
SRP_00340 0.11 0.11 0.11 9 41246 75 9 64 2 73
SRP_00361 0.00 0.00 0.00 0 41825 80 8 72 0 90
SRP_00371 0.14 0.13 0.15 11 36243 63 7 54 2 71

^top



Performance of Carnac(seed) - scored lower in this pairwise comparison

1. Total counts & total scores for Carnac(seed)

Total Base Pair Counts
Total TP 323
Total TN 5626348
Total FP 74
Total FP CONTRA 3
Total FP INCONS 39
Total FP COMP 32
Total FN 10466
Total Scores
MCC 0.163
Average MCC ± 95% Confidence Intervals 0.061 ± 0.036
Sensitivity 0.030
Positive Predictive Value 0.885
Nr of predictions 121

^top



2. Individual counts for Carnac(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00001 0.00 0.00 0.00 0 34191 0 0 0 0 75
ASE_00002 0.00 0.00 0.00 0 35511 0 0 0 0 73
ASE_00009 0.00 0.00 0.00 0 26106 0 0 0 0 67
ASE_00014 0.00 0.00 0.00 0 54285 0 0 0 0 106
ASE_00116 0.00 0.00 0.00 0 31626 0 0 0 0 66
ASE_00120 0.00 0.00 0.00 0 46971 0 0 0 0 94
ASE_00121 0.00 0.00 0.00 0 45150 0 0 0 0 92
ASE_00145 0.00 0.00 0.00 0 70125 0 0 0 0 111
ASE_00154 0.00 0.00 0.00 0 65341 0 0 0 0 107
ASE_00155 0.00 0.00 0.00 0 93528 0 0 0 0 142
ASE_00187 0.00 0.00 0.00 0 68635 0 0 0 0 115
ASE_00191 0.00 0.00 0.00 0 29646 0 0 0 0 61
ASE_00196 0.00 0.00 0.00 0 31626 0 0 0 0 75
ASE_00205 0.00 0.00 0.00 0 46056 0 0 0 0 90
ASE_00209 0.00 0.00 0.00 0 43071 0 0 0 0 88
ASE_00210 0.00 0.00 0.00 0 27261 0 0 0 0 65
ASE_00213 0.00 0.00 0.00 0 27261 0 0 0 0 66
ASE_00346 0.00 0.00 0.00 0 48205 0 0 0 0 97
ASE_00359 0.00 0.00 0.00 0 42778 0 0 0 0 80
ASE_00360 0.00 0.00 0.00 0 29403 0 0 0 0 65
ASE_00417 0.00 0.00 0.00 0 54615 0 0 0 0 100
ASE_00418 0.00 0.00 0.00 0 38781 0 0 0 0 75
CRW_00016 0.00 0.00 0.00 0 77421 0 0 0 0 120
CRW_00610 0.00 0.00 0.00 0 36315 0 0 0 0 81
CRW_00613 0.00 0.00 0.00 0 34980 0 0 0 0 78
CRW_00618 0.00 0.00 0.00 0 56280 0 0 0 0 61
CRW_00633 0.00 0.00 0.00 0 63546 0 0 0 0 106
CRW_00634 0.00 0.00 0.00 0 64620 0 0 0 0 94
CRW_00670 0.00 0.00 0.00 0 70125 0 0 0 0 120
CRW_00671 0.00 0.00 0.00 0 62835 0 0 0 0 117
CRW_00672 0.00 0.00 0.00 0 72390 0 0 0 0 111
CRW_00674 0.00 0.00 0.00 0 83436 0 0 0 0 124
CRW_00676 0.00 0.00 0.00 0 93528 0 0 0 0 115
CRW_00692 0.00 0.00 0.00 0 68635 0 0 0 0 90
PDB_00827 0.00 0.00 0.00 0 27028 0 0 0 0 81
RFA_00598 0.00 0.00 0.00 0 84666 0 0 0 0 101
RFA_00606 0.17 0.08 0.38 3 21313 6 1 4 1 36
RFA_00620 0.13 0.08 0.23 3 21932 11 2 8 1 36
RFA_00621 0.00 0.00 0.00 0 46360 0 0 0 0 76
RFA_00623 0.00 0.00 0.00 0 43956 0 0 0 0 78
RFA_00624 0.00 0.00 0.00 0 30135 0 0 0 0 61
RFA_00626 0.67 0.52 0.87 45 56564 11 0 7 4 42
RFA_00627 0.69 0.55 0.86 48 56897 10 0 8 2 39
RFA_00628 0.66 0.43 1.00 37 57254 6 0 0 6 49
RFA_00630 0.79 0.67 0.94 58 56891 12 0 4 8 29
RFA_00639 0.00 0.00 0.00 0 54608 7 0 7 0 87
RFA_00814 0.66 0.44 1.00 18 25182 2 0 0 2 23
RFA_00815 0.75 0.56 1.00 23 24508 1 0 0 1 18
RFA_00816 0.66 0.44 1.00 18 23202 2 0 0 2 23
RFA_00817 0.80 0.66 0.96 27 21917 1 0 1 0 14
RFA_00818 0.70 0.49 1.00 20 20281 3 0 0 3 21
RFA_00819 0.75 0.56 1.00 23 27943 2 0 0 2 18
SRP_00009 0.00 0.00 0.00 0 46665 0 0 0 0 102
SRP_00011 0.00 0.00 0.00 0 46360 0 0 0 0 104
SRP_00012 0.00 0.00 0.00 0 46360 0 0 0 0 101
SRP_00015 0.00 0.00 0.00 0 46360 0 0 0 0 99
SRP_00016 0.00 0.00 0.00 0 47895 0 0 0 0 109
SRP_00018 0.00 0.00 0.00 0 38503 0 0 0 0 81
SRP_00019 0.00 0.00 0.00 0 36046 0 0 0 0 91
SRP_00069 0.00 0.00 0.00 0 34453 0 0 0 0 85
SRP_00073 0.00 0.00 0.00 0 39060 0 0 0 0 71
SRP_00079 0.00 0.00 0.00 0 41905 0 0 0 0 88
SRP_00085 0.00 0.00 0.00 0 45451 0 0 0 0 101
SRP_00087 0.00 0.00 0.00 0 37401 0 0 0 0 58
SRP_00091 0.00 0.00 0.00 0 52975 0 0 0 0 95
SRP_00099 0.00 0.00 0.00 0 44850 0 0 0 0 95
SRP_00104 0.00 0.00 0.00 0 34453 0 0 0 0 69
SRP_00124 0.00 0.00 0.00 0 37128 0 0 0 0 85
SRP_00127 0.00 0.00 0.00 0 72390 0 0 0 0 119
SRP_00130 0.00 0.00 0.00 0 49770 0 0 0 0 97
SRP_00136 0.00 0.00 0.00 0 44253 0 0 0 0 71
SRP_00142 0.00 0.00 0.00 0 46056 0 0 0 0 93
SRP_00143 0.00 0.00 0.00 0 49770 0 0 0 0 101
SRP_00144 0.00 0.00 0.00 0 46665 0 0 0 0 94
SRP_00148 0.00 0.00 0.00 0 50086 0 0 0 0 87
SRP_00152 0.00 0.00 0.00 0 45753 0 0 0 0 103
SRP_00153 0.00 0.00 0.00 0 46971 0 0 0 0 107
SRP_00157 0.00 0.00 0.00 0 44253 0 0 0 0 72
SRP_00160 0.00 0.00 0.00 0 86320 0 0 0 0 130
SRP_00175 0.00 0.00 0.00 0 44551 0 0 0 0 84
SRP_00178 0.00 0.00 0.00 0 45753 0 0 0 0 102
SRP_00179 0.00 0.00 0.00 0 46056 0 0 0 0 105
SRP_00181 0.00 0.00 0.00 0 46056 0 0 0 0 102
SRP_00182 0.00 0.00 0.00 0 46056 0 0 0 0 101
SRP_00184 0.00 0.00 0.00 0 46056 0 0 0 0 100
SRP_00188 0.00 0.00 0.00 0 31125 0 0 0 0 79
SRP_00197 0.00 0.00 0.00 0 49455 0 0 0 0 101
SRP_00198 0.00 0.00 0.00 0 51040 0 0 0 0 103
SRP_00199 0.00 0.00 0.00 0 51040 0 0 0 0 105
SRP_00201 0.00 0.00 0.00 0 47895 0 0 0 0 111
SRP_00202 0.00 0.00 0.00 0 54285 0 0 0 0 117
SRP_00203 0.00 0.00 0.00 0 48828 0 0 0 0 105
SRP_00204 0.00 0.00 0.00 0 49141 0 0 0 0 102
SRP_00206 0.00 0.00 0.00 0 47895 0 0 0 0 108
SRP_00230 0.00 0.00 0.00 0 39621 0 0 0 0 90
SRP_00239 0.00 0.00 0.00 0 50403 0 0 0 0 93
SRP_00240 0.00 0.00 0.00 0 48205 0 0 0 0 102
SRP_00241 0.00 0.00 0.00 0 46056 0 0 0 0 82
SRP_00252 0.00 0.00 0.00 0 49141 0 0 0 0 89
SRP_00253 0.00 0.00 0.00 0 49455 0 0 0 0 90
SRP_00254 0.00 0.00 0.00 0 32385 0 0 0 0 61
SRP_00255 0.00 0.00 0.00 0 47586 0 0 0 0 94
SRP_00257 0.00 0.00 0.00 0 51040 0 0 0 0 108
SRP_00260 0.00 0.00 0.00 0 47895 0 0 0 0 100
SRP_00265 0.00 0.00 0.00 0 41328 0 0 0 0 73
SRP_00276 0.00 0.00 0.00 0 50721 0 0 0 0 110
SRP_00278 0.00 0.00 0.00 0 46056 0 0 0 0 105
SRP_00301 0.00 0.00 0.00 0 32640 0 0 0 0 86
SRP_00302 0.00 0.00 0.00 0 41616 0 0 0 0 81
SRP_00308 0.00 0.00 0.00 0 36315 0 0 0 0 88
SRP_00322 0.00 0.00 0.00 0 48516 0 0 0 0 109
SRP_00323 0.00 0.00 0.00 0 47278 0 0 0 0 107
SRP_00328 0.00 0.00 0.00 0 39621 0 0 0 0 85
SRP_00329 0.00 0.00 0.00 0 39903 0 0 0 0 85
SRP_00330 0.00 0.00 0.00 0 50086 0 0 0 0 95
SRP_00331 0.00 0.00 0.00 0 37675 0 0 0 0 87
SRP_00332 0.00 0.00 0.00 0 50403 0 0 0 0 106
SRP_00339 0.00 0.00 0.00 0 44551 0 0 0 0 94
SRP_00340 0.00 0.00 0.00 0 41328 0 0 0 0 82
SRP_00361 0.00 0.00 0.00 0 41905 0 0 0 0 90
SRP_00371 0.00 0.00 0.00 0 36315 0 0 0 0 82

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.