CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of Afold - scored higher in this pairwise comparison

  4. Performance of Carnac(seed) - scored lower in this pairwise comparison

  5. Compile and download dataset for Afold & Carnac(seed) [.zip] - may take several seconds...


Overview

Metric Afold Carnac(seed)
MCC 0.556 > 0.158
Average MCC ± 95% Confidence Intervals 0.551 ± 0.036 > 0.059 ± 0.033
Sensitivity 0.561 > 0.030
Positive Predictive Value 0.553 < 0.840
Total TP 6648 > 352
Total TN 7524614 < 7536224
Total FP 6590 > 110
Total FP CONTRA 920 > 4
Total FP INCONS 4461 > 63
Total FP COMP 1209 > 43
Total FN 5200 < 11496
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of Afold and Carnac(seed). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for Afold and Carnac(seed)).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for Afold and Carnac(seed)).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for Afold and Carnac(seed). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for Afold and Carnac(seed)).

^top





Performance of Afold - scored higher in this pairwise comparison

1. Total counts & total scores for Afold

Total Base Pair Counts
Total TP 6648
Total TN 7524614
Total FP 6590
Total FP CONTRA 920
Total FP INCONS 4461
Total FP COMP 1209
Total FN 5200
Total Scores
MCC 0.556
Average MCC ± 95% Confidence Intervals 0.551 ± 0.036
Sensitivity 0.561
Positive Predictive Value 0.553
Nr of predictions 124

^top



2. Individual counts for Afold [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00001 0.75 0.72 0.78 54 34122 23 3 12 8 21
ASE_00002 0.62 0.62 0.63 45 35439 33 4 23 6 28
ASE_00009 0.79 0.76 0.82 51 26044 13 0 11 2 16
ASE_00014 0.24 0.25 0.24 26 54178 82 8 73 1 80
ASE_00116 0.24 0.24 0.23 16 31557 54 8 45 1 50
ASE_00120 0.37 0.35 0.38 33 46885 53 9 44 0 61
ASE_00121 0.29 0.29 0.29 27 45058 65 8 57 0 65
ASE_00145 0.40 0.40 0.40 44 70016 69 9 56 4 67
ASE_00148 0.63 0.60 0.65 93 112433 53 4 45 4 61
ASE_00154 0.57 0.55 0.58 59 65240 45 4 38 3 48
ASE_00155 0.50 0.48 0.52 68 93396 69 10 54 5 74
ASE_00187 0.50 0.48 0.51 55 68528 58 7 45 6 60
ASE_00191 0.56 0.52 0.59 32 29592 32 1 21 10 29
ASE_00196 0.76 0.75 0.77 56 31553 21 2 15 4 19
ASE_00209 0.29 0.28 0.30 25 42989 57 4 53 0 63
ASE_00210 0.61 0.62 0.60 40 27194 28 6 21 1 25
ASE_00213 0.72 0.71 0.73 47 27197 18 3 14 1 19
ASE_00222 0.56 0.55 0.56 87 111947 68 8 59 1 70
ASE_00346 0.31 0.30 0.32 29 48114 65 7 55 3 68
ASE_00359 0.49 0.49 0.49 39 42698 49 3 38 8 41
ASE_00417 0.36 0.37 0.35 37 54508 73 10 60 3 63
ASE_00418 0.20 0.21 0.20 16 38701 67 12 52 3 59
CRW_00013 0.22 0.22 0.22 25 103625 113 13 77 23 90
CRW_00016 0.65 0.62 0.69 74 77313 47 5 29 13 46
CRW_00610 0.42 0.42 0.43 34 36235 46 6 40 0 47
CRW_00613 0.46 0.42 0.51 33 34915 41 2 30 9 45
CRW_00614 0.07 0.09 0.06 5 121689 141 24 53 64 52
CRW_00619 0.38 0.42 0.34 31 164933 138 25 36 77 42
CRW_00620 0.16 0.21 0.12 13 150869 161 32 61 68 50
CRW_00621 0.10 0.10 0.10 7 153108 157 13 53 91 64
CRW_00623 0.30 0.29 0.32 24 129719 123 11 41 71 59
CRW_00633 0.45 0.44 0.47 47 63445 58 10 44 4 59
CRW_00670 0.77 0.74 0.81 89 70015 26 0 21 5 31
CRW_00671 0.66 0.64 0.68 75 62725 41 8 27 6 42
CRW_00672 0.74 0.73 0.74 81 72281 36 10 18 8 30
CRW_00674 0.55 0.55 0.55 68 83312 61 8 48 5 56
CRW_00676 0.54 0.56 0.52 64 93406 75 17 41 17 51
CRW_00692 0.57 0.57 0.57 51 68545 54 15 24 15 39
PDB_00827 0.67 0.59 0.76 48 26965 16 0 15 1 33
RFA_00597 0.51 0.51 0.51 56 97794 82 5 48 29 53
RFA_00599 0.67 0.70 0.64 78 101353 66 16 28 22 33
RFA_00601 0.09 0.11 0.08 12 99092 139 35 96 8 102
RFA_00602 0.50 0.53 0.47 62 101344 91 18 51 22 54
RFA_00606 0.67 0.67 0.68 26 21283 26 7 5 14 13
RFA_00620 0.64 0.62 0.67 24 21909 26 3 9 14 15
RFA_00621 0.39 0.39 0.38 30 46281 63 7 42 14 46
RFA_00623 0.46 0.45 0.47 35 43881 54 5 35 14 43
RFA_00624 0.43 0.46 0.41 28 30067 45 5 35 5 33
RFA_00626 0.89 0.86 0.93 75 56535 31 2 4 25 12
RFA_00627 0.48 0.47 0.48 41 56868 50 10 34 6 46
RFA_00628 0.83 0.80 0.85 69 57210 38 1 11 26 17
RFA_00630 0.60 0.60 0.60 52 56866 46 8 27 11 35
RFA_00639 0.49 0.49 0.49 43 54528 61 8 36 17 44
RFA_00814 0.35 0.37 0.33 15 25155 42 7 23 12 26
RFA_00815 0.56 0.61 0.51 25 24482 35 8 16 11 16
RFA_00816 0.59 0.66 0.53 27 23169 33 13 11 9 14
RFA_00818 0.19 0.22 0.17 9 20247 48 19 26 3 32
RFA_00819 0.96 0.95 0.98 39 27926 32 0 1 31 2
RFA_00820 0.90 0.91 0.88 145 151361 34 1 18 15 15
RFA_00830 0.50 0.47 0.53 73 152490 110 5 60 45 81
SRP_00011 0.43 0.42 0.45 44 46262 54 4 50 0 60
SRP_00015 0.54 0.53 0.55 52 46265 48 8 35 5 47
SRP_00016 0.92 0.89 0.94 97 47792 8 0 6 2 12
SRP_00018 0.71 0.68 0.73 55 38428 33 1 19 13 26
SRP_00019 0.91 0.89 0.93 81 35959 12 4 2 6 10
SRP_00069 0.57 0.55 0.59 47 34374 34 1 31 2 38
SRP_00070 0.28 0.32 0.25 39 126599 128 30 88 10 82
SRP_00073 0.65 0.70 0.60 50 38977 46 8 25 13 21
SRP_00079 0.52 0.55 0.50 48 41809 53 8 40 5 40
SRP_00085 0.64 0.63 0.65 64 45352 37 5 30 2 37
SRP_00087 0.24 0.28 0.21 16 37323 71 20 42 9 42
SRP_00091 0.80 0.82 0.79 78 52876 26 4 17 5 17
SRP_00099 0.57 0.57 0.57 54 44755 43 5 36 2 41
SRP_00104 0.41 0.46 0.36 32 34365 59 12 44 3 37
SRP_00124 0.53 0.52 0.55 44 37048 38 5 31 2 41
SRP_00127 0.61 0.61 0.60 73 72268 55 8 41 6 46
SRP_00130 0.24 0.24 0.24 23 49676 72 10 61 1 74
SRP_00136 0.35 0.39 0.31 28 44163 69 13 49 7 43
SRP_00142 0.57 0.58 0.57 54 45961 43 5 36 2 39
SRP_00143 0.36 0.38 0.35 38 49662 72 8 62 2 63
SRP_00144 0.74 0.74 0.74 70 46571 31 1 23 7 24
SRP_00148 0.31 0.33 0.29 29 49987 81 4 66 11 58
SRP_00152 0.56 0.56 0.57 58 45651 45 1 43 1 45
SRP_00157 0.37 0.40 0.34 29 44167 65 15 42 8 43
SRP_00158 0.30 0.37 0.25 42 124084 130 48 77 5 73
SRP_00160 0.41 0.41 0.42 53 86194 74 4 69 1 77
SRP_00175 0.51 0.52 0.49 44 44461 52 7 39 6 40
SRP_00178 0.83 0.78 0.87 80 45661 14 1 11 2 22
SRP_00181 0.33 0.31 0.35 32 45964 60 1 59 0 70
SRP_00182 0.76 0.73 0.79 74 45962 21 3 17 1 27
SRP_00184 0.62 0.60 0.64 60 45962 39 2 32 5 40
SRP_00197 0.24 0.24 0.25 24 49359 72 7 65 0 77
SRP_00199 0.67 0.66 0.68 69 50939 34 2 30 2 36
SRP_00201 0.84 0.84 0.84 93 47784 20 5 13 2 18
SRP_00202 0.95 0.93 0.96 109 54172 6 0 4 2 8
SRP_00203 0.73 0.75 0.72 79 48718 37 3 28 6 26
SRP_00204 0.68 0.68 0.69 69 49041 33 4 27 2 33
SRP_00206 0.84 0.83 0.84 90 47788 18 5 12 1 18
SRP_00230 0.80 0.81 0.79 73 39529 23 3 16 4 17
SRP_00241 0.54 0.59 0.51 48 45961 50 11 36 3 34
SRP_00252 0.62 0.64 0.59 57 49045 42 12 27 3 32
SRP_00253 0.61 0.63 0.59 57 49359 44 7 32 5 33
SRP_00254 0.43 0.48 0.39 29 32311 50 13 32 5 32
SRP_00257 0.85 0.87 0.84 94 50928 20 4 14 2 14
SRP_00265 0.56 0.62 0.52 45 41241 55 9 33 13 28
SRP_00276 0.86 0.85 0.86 94 50612 16 5 10 1 16
SRP_00278 0.72 0.72 0.71 76 45949 33 4 27 2 29
SRP_00286 0.57 0.56 0.59 97 136861 77 4 64 9 77
SRP_00287 0.64 0.65 0.64 105 141613 74 8 52 14 57
SRP_00290 0.66 0.66 0.65 89 100440 57 2 45 10 46
SRP_00292 0.58 0.55 0.61 96 137392 69 2 60 7 77
SRP_00294 0.61 0.59 0.62 99 129126 67 4 57 6 68
SRP_00301 0.85 0.85 0.86 73 32555 18 1 11 6 13
SRP_00302 0.39 0.40 0.39 32 41533 61 7 44 10 49
SRP_00308 0.55 0.55 0.56 48 36229 42 9 29 4 40
SRP_00322 0.60 0.60 0.61 65 48409 43 2 40 1 44
SRP_00323 0.90 0.89 0.90 95 47173 14 0 10 4 12
SRP_00330 0.44 0.44 0.45 42 49992 54 2 50 2 53
SRP_00332 0.90 0.90 0.91 95 50299 15 1 8 6 11
SRP_00339 0.56 0.56 0.56 53 44456 45 5 37 3 41
SRP_00340 0.55 0.59 0.52 48 41235 47 9 36 2 34
SRP_00361 0.71 0.71 0.70 64 41814 37 3 24 10 26
SRP_00371 0.53 0.54 0.52 44 36231 43 3 37 3 38
SRP_00382 0.58 0.57 0.58 97 141612 80 6 63 11 73

^top



Performance of Carnac(seed) - scored lower in this pairwise comparison

1. Total counts & total scores for Carnac(seed)

Total Base Pair Counts
Total TP 352
Total TN 7536224
Total FP 110
Total FP CONTRA 4
Total FP INCONS 63
Total FP COMP 43
Total FN 11496
Total Scores
MCC 0.158
Average MCC ± 95% Confidence Intervals 0.059 ± 0.033
Sensitivity 0.030
Positive Predictive Value 0.840
Nr of predictions 124

^top



2. Individual counts for Carnac(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00001 0.00 0.00 0.00 0 34191 0 0 0 0 75
ASE_00002 0.00 0.00 0.00 0 35511 0 0 0 0 73
ASE_00009 0.00 0.00 0.00 0 26106 0 0 0 0 67
ASE_00014 0.00 0.00 0.00 0 54285 0 0 0 0 106
ASE_00116 0.00 0.00 0.00 0 31626 0 0 0 0 66
ASE_00120 0.00 0.00 0.00 0 46971 0 0 0 0 94
ASE_00121 0.00 0.00 0.00 0 45150 0 0 0 0 92
ASE_00145 0.00 0.00 0.00 0 70125 0 0 0 0 111
ASE_00148 0.00 0.00 0.00 0 112575 0 0 0 0 154
ASE_00154 0.00 0.00 0.00 0 65341 0 0 0 0 107
ASE_00155 0.00 0.00 0.00 0 93528 0 0 0 0 142
ASE_00187 0.00 0.00 0.00 0 68635 0 0 0 0 115
ASE_00191 0.00 0.00 0.00 0 29646 0 0 0 0 61
ASE_00196 0.00 0.00 0.00 0 31626 0 0 0 0 75
ASE_00209 0.00 0.00 0.00 0 43071 0 0 0 0 88
ASE_00210 0.00 0.00 0.00 0 27261 0 0 0 0 65
ASE_00213 0.00 0.00 0.00 0 27261 0 0 0 0 66
ASE_00222 0.00 0.00 0.00 0 112101 0 0 0 0 157
ASE_00346 0.00 0.00 0.00 0 48205 0 0 0 0 97
ASE_00359 0.00 0.00 0.00 0 42778 0 0 0 0 80
ASE_00417 0.00 0.00 0.00 0 54615 0 0 0 0 100
ASE_00418 0.00 0.00 0.00 0 38781 0 0 0 0 75
CRW_00013 0.00 0.00 0.00 0 103740 0 0 0 0 115
CRW_00016 0.00 0.00 0.00 0 77421 0 0 0 0 120
CRW_00610 0.00 0.00 0.00 0 36315 0 0 0 0 81
CRW_00613 0.00 0.00 0.00 0 34980 0 0 0 0 78
CRW_00614 0.00 0.00 0.00 0 121771 0 0 0 0 57
CRW_00619 0.00 0.00 0.00 0 165025 0 0 0 0 73
CRW_00620 0.00 0.00 0.00 0 150975 0 0 0 0 63
CRW_00621 0.00 0.00 0.00 0 153181 0 0 0 0 71
CRW_00623 0.00 0.00 0.00 0 129795 0 0 0 0 83
CRW_00633 0.00 0.00 0.00 0 63546 0 0 0 0 106
CRW_00670 0.00 0.00 0.00 0 70125 0 0 0 0 120
CRW_00671 0.00 0.00 0.00 0 62835 0 0 0 0 117
CRW_00672 0.00 0.00 0.00 0 72390 0 0 0 0 111
CRW_00674 0.00 0.00 0.00 0 83436 0 0 0 0 124
CRW_00676 0.00 0.00 0.00 0 93528 0 0 0 0 115
CRW_00692 0.00 0.00 0.00 0 68635 0 0 0 0 90
PDB_00827 0.00 0.00 0.00 0 27028 0 0 0 0 81
RFA_00597 0.00 0.00 0.00 0 97903 0 0 0 0 109
RFA_00599 0.00 0.00 0.00 0 101475 0 0 0 0 111
RFA_00601 0.00 0.00 0.00 0 99235 0 0 0 0 114
RFA_00602 0.00 0.00 0.00 0 101475 0 0 0 0 116
RFA_00606 0.17 0.08 0.38 3 21313 6 1 4 1 36
RFA_00620 0.13 0.08 0.23 3 21932 11 2 8 1 36
RFA_00621 0.00 0.00 0.00 0 46360 0 0 0 0 76
RFA_00623 0.00 0.00 0.00 0 43956 0 0 0 0 78
RFA_00624 0.00 0.00 0.00 0 30135 0 0 0 0 61
RFA_00626 0.67 0.52 0.87 45 56564 11 0 7 4 42
RFA_00627 0.69 0.55 0.86 48 56897 10 0 8 2 39
RFA_00628 0.66 0.43 1.00 37 57254 6 0 0 6 49
RFA_00630 0.79 0.67 0.94 58 56891 12 0 4 8 29
RFA_00639 0.00 0.00 0.00 0 54608 7 0 7 0 87
RFA_00814 0.66 0.44 1.00 18 25182 2 0 0 2 23
RFA_00815 0.75 0.56 1.00 23 24508 1 0 0 1 18
RFA_00816 0.66 0.44 1.00 18 23202 2 0 0 2 23
RFA_00818 0.70 0.49 1.00 20 20281 3 0 0 3 21
RFA_00819 0.75 0.56 1.00 23 27943 2 0 0 2 18
RFA_00820 0.39 0.19 0.78 31 151485 12 0 9 3 129
RFA_00830 0.31 0.16 0.60 25 152586 25 1 16 8 129
SRP_00011 0.00 0.00 0.00 0 46360 0 0 0 0 104
SRP_00015 0.00 0.00 0.00 0 46360 0 0 0 0 99
SRP_00016 0.00 0.00 0.00 0 47895 0 0 0 0 109
SRP_00018 0.00 0.00 0.00 0 38503 0 0 0 0 81
SRP_00019 0.00 0.00 0.00 0 36046 0 0 0 0 91
SRP_00069 0.00 0.00 0.00 0 34453 0 0 0 0 85
SRP_00070 0.00 0.00 0.00 0 126756 0 0 0 0 121
SRP_00073 0.00 0.00 0.00 0 39060 0 0 0 0 71
SRP_00079 0.00 0.00 0.00 0 41905 0 0 0 0 88
SRP_00085 0.00 0.00 0.00 0 45451 0 0 0 0 101
SRP_00087 0.00 0.00 0.00 0 37401 0 0 0 0 58
SRP_00091 0.00 0.00 0.00 0 52975 0 0 0 0 95
SRP_00099 0.00 0.00 0.00 0 44850 0 0 0 0 95
SRP_00104 0.00 0.00 0.00 0 34453 0 0 0 0 69
SRP_00124 0.00 0.00 0.00 0 37128 0 0 0 0 85
SRP_00127 0.00 0.00 0.00 0 72390 0 0 0 0 119
SRP_00130 0.00 0.00 0.00 0 49770 0 0 0 0 97
SRP_00136 0.00 0.00 0.00 0 44253 0 0 0 0 71
SRP_00142 0.00 0.00 0.00 0 46056 0 0 0 0 93
SRP_00143 0.00 0.00 0.00 0 49770 0 0 0 0 101
SRP_00144 0.00 0.00 0.00 0 46665 0 0 0 0 94
SRP_00148 0.00 0.00 0.00 0 50086 0 0 0 0 87
SRP_00152 0.00 0.00 0.00 0 45753 0 0 0 0 103
SRP_00157 0.00 0.00 0.00 0 44253 0 0 0 0 72
SRP_00158 0.00 0.00 0.00 0 124251 0 0 0 0 115
SRP_00160 0.00 0.00 0.00 0 86320 0 0 0 0 130
SRP_00175 0.00 0.00 0.00 0 44551 0 0 0 0 84
SRP_00178 0.00 0.00 0.00 0 45753 0 0 0 0 102
SRP_00181 0.00 0.00 0.00 0 46056 0 0 0 0 102
SRP_00182 0.00 0.00 0.00 0 46056 0 0 0 0 101
SRP_00184 0.00 0.00 0.00 0 46056 0 0 0 0 100
SRP_00197 0.00 0.00 0.00 0 49455 0 0 0 0 101
SRP_00199 0.00 0.00 0.00 0 51040 0 0 0 0 105
SRP_00201 0.00 0.00 0.00 0 47895 0 0 0 0 111
SRP_00202 0.00 0.00 0.00 0 54285 0 0 0 0 117
SRP_00203 0.00 0.00 0.00 0 48828 0 0 0 0 105
SRP_00204 0.00 0.00 0.00 0 49141 0 0 0 0 102
SRP_00206 0.00 0.00 0.00 0 47895 0 0 0 0 108
SRP_00230 0.00 0.00 0.00 0 39621 0 0 0 0 90
SRP_00241 0.00 0.00 0.00 0 46056 0 0 0 0 82
SRP_00252 0.00 0.00 0.00 0 49141 0 0 0 0 89
SRP_00253 0.00 0.00 0.00 0 49455 0 0 0 0 90
SRP_00254 0.00 0.00 0.00 0 32385 0 0 0 0 61
SRP_00257 0.00 0.00 0.00 0 51040 0 0 0 0 108
SRP_00265 0.00 0.00 0.00 0 41328 0 0 0 0 73
SRP_00276 0.00 0.00 0.00 0 50721 0 0 0 0 110
SRP_00278 0.00 0.00 0.00 0 46056 0 0 0 0 105
SRP_00286 0.00 0.00 0.00 0 137026 0 0 0 0 174
SRP_00287 0.00 0.00 0.00 0 141778 0 0 0 0 162
SRP_00290 0.00 0.00 0.00 0 100576 0 0 0 0 135
SRP_00292 0.00 0.00 0.00 0 137550 0 0 0 0 173
SRP_00294 0.00 0.00 0.00 0 129286 0 0 0 0 167
SRP_00301 0.00 0.00 0.00 0 32640 0 0 0 0 86
SRP_00302 0.00 0.00 0.00 0 41616 0 0 0 0 81
SRP_00308 0.00 0.00 0.00 0 36315 0 0 0 0 88
SRP_00322 0.00 0.00 0.00 0 48516 0 0 0 0 109
SRP_00323 0.00 0.00 0.00 0 47278 0 0 0 0 107
SRP_00330 0.00 0.00 0.00 0 50086 0 0 0 0 95
SRP_00332 0.00 0.00 0.00 0 50403 0 0 0 0 106
SRP_00339 0.00 0.00 0.00 0 44551 0 0 0 0 94
SRP_00340 0.00 0.00 0.00 0 41328 0 0 0 0 82
SRP_00361 0.00 0.00 0.00 0 41905 0 0 0 0 90
SRP_00371 0.00 0.00 0.00 0 36315 0 0 0 0 82
SRP_00382 0.00 0.00 0.00 0 141778 0 0 0 0 170

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.