CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of RNAshapes - scored higher in this pairwise comparison

  4. Performance of Carnac(seed) - scored lower in this pairwise comparison

  5. Compile and download dataset for RNAshapes & Carnac(seed) [.zip] - may take several seconds...


Overview

Metric RNAshapes Carnac(seed)
MCC 0.522 > 0.149
Average MCC ± 95% Confidence Intervals 0.516 ± 0.035 > 0.056 ± 0.034
Sensitivity 0.519 > 0.025
Positive Predictive Value 0.528 < 0.875
Total TP 5434 > 265
Total TN 5415528 < 5425526
Total FP 5489 > 62
Total FP CONTRA 698 > 3
Total FP INCONS 4169 > 35
Total FP COMP 622 > 24
Total FN 5033 < 10202
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of RNAshapes and Carnac(seed). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for RNAshapes and Carnac(seed)).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for RNAshapes and Carnac(seed)).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for RNAshapes and Carnac(seed). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for RNAshapes and Carnac(seed)).

^top





Performance of RNAshapes - scored higher in this pairwise comparison

1. Total counts & total scores for RNAshapes

Total Base Pair Counts
Total TP 5434
Total TN 5415528
Total FP 5489
Total FP CONTRA 698
Total FP INCONS 4169
Total FP COMP 622
Total FN 5033
Total Scores
MCC 0.522
Average MCC ± 95% Confidence Intervals 0.516 ± 0.035
Sensitivity 0.519
Positive Predictive Value 0.528
Nr of predictions 118

^top



2. Individual counts for RNAshapes [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00001 0.65 0.61 0.69 46 34124 28 3 18 7 29
ASE_00002 0.47 0.45 0.49 33 35444 39 4 30 5 40
ASE_00009 0.40 0.37 0.43 25 26048 35 2 31 2 42
ASE_00014 0.12 0.12 0.13 13 54184 89 8 80 1 93
ASE_00116 0.14 0.14 0.15 9 31567 53 5 45 3 57
ASE_00120 0.30 0.29 0.31 27 46884 60 9 51 0 67
ASE_00121 0.32 0.32 0.33 29 45063 60 8 50 2 63
ASE_00145 0.36 0.35 0.38 39 70021 70 7 58 5 72
ASE_00154 0.54 0.52 0.55 56 65239 49 6 40 3 51
ASE_00187 0.41 0.40 0.43 46 68528 66 5 56 5 69
ASE_00191 0.54 0.54 0.53 33 29584 38 4 25 9 28
ASE_00196 0.49 0.49 0.49 37 31551 41 2 36 3 38
ASE_00205 0.46 0.47 0.45 42 45962 55 9 43 3 48
ASE_00209 0.49 0.45 0.53 40 42995 36 3 33 0 48
ASE_00210 0.55 0.55 0.55 36 27196 29 5 24 0 29
ASE_00213 0.30 0.29 0.32 19 27201 43 2 39 2 47
ASE_00346 0.35 0.34 0.37 33 48116 59 5 51 3 64
ASE_00359 0.63 0.60 0.66 48 42705 30 1 24 5 32
ASE_00360 0.29 0.29 0.29 19 29337 51 10 37 4 46
ASE_00417 0.35 0.36 0.34 36 54510 73 4 65 4 64
ASE_00418 0.33 0.33 0.32 25 38704 54 4 48 2 50
CRW_00016 0.49 0.48 0.51 57 77309 66 7 48 11 63
CRW_00610 0.55 0.53 0.57 43 36240 37 3 29 5 38
CRW_00613 0.64 0.58 0.70 45 34916 28 1 18 9 33
CRW_00618 0.17 0.18 0.16 11 56213 77 9 47 21 50
CRW_00633 0.55 0.54 0.56 57 63445 45 8 36 1 49
CRW_00634 0.33 0.34 0.33 32 64522 75 8 58 9 62
CRW_00670 0.74 0.72 0.76 86 70012 32 0 27 5 34
CRW_00671 0.65 0.62 0.67 73 62726 43 1 35 7 44
CRW_00672 0.66 0.66 0.66 73 72279 45 8 30 7 38
CRW_00674 0.68 0.67 0.69 83 83316 41 6 31 4 41
CRW_00676 0.58 0.62 0.55 71 93398 67 17 42 8 44
CRW_00692 0.43 0.46 0.40 41 68533 70 18 43 9 49
PDB_00827 0.64 0.56 0.74 45 26967 17 0 16 1 36
RFA_00598 0.53 0.53 0.53 54 84565 71 7 40 24 47
RFA_00606 0.67 0.67 0.67 26 21282 32 1 12 19 13
RFA_00620 0.44 0.46 0.42 18 21902 36 7 18 11 21
RFA_00621 0.45 0.45 0.45 34 46285 59 1 40 18 42
RFA_00623 0.54 0.53 0.55 41 43882 47 3 30 14 37
RFA_00624 0.50 0.52 0.48 32 30068 42 12 23 7 29
RFA_00626 0.45 0.45 0.46 39 56531 61 9 37 15 48
RFA_00627 0.63 0.61 0.66 53 56873 42 4 23 15 34
RFA_00628 0.82 0.80 0.84 69 57209 33 1 12 20 17
RFA_00639 0.22 0.24 0.21 21 54515 82 20 59 3 66
RFA_00814 0.62 0.63 0.60 26 25157 26 6 11 9 15
RFA_00815 0.67 0.71 0.63 29 24485 31 1 16 14 12
RFA_00816 0.21 0.24 0.18 10 23164 47 18 28 1 31
RFA_00817 0.10 0.12 0.08 5 21881 60 21 38 1 36
RFA_00818 0.52 0.56 0.48 23 20253 31 9 16 6 18
RFA_00819 0.84 0.80 0.87 33 27928 36 0 5 31 8
SRP_00009 0.30 0.28 0.32 29 46573 66 11 52 3 73
SRP_00011 0.47 0.44 0.50 46 46268 48 3 43 2 58
SRP_00012 0.69 0.65 0.73 66 46270 27 2 22 3 35
SRP_00015 0.78 0.74 0.82 73 46271 22 3 13 6 26
SRP_00016 0.76 0.74 0.78 81 47791 24 1 22 1 28
SRP_00018 0.57 0.56 0.58 45 38425 42 5 28 9 36
SRP_00019 0.46 0.44 0.48 40 35962 48 5 39 4 51
SRP_00069 0.74 0.71 0.78 60 34376 19 1 16 2 25
SRP_00073 0.68 0.70 0.66 50 38984 39 8 18 13 21
SRP_00079 0.53 0.52 0.53 46 41819 52 1 39 12 42
SRP_00085 0.48 0.47 0.51 47 45358 46 3 43 0 54
SRP_00087 0.19 0.22 0.16 13 37318 77 22 48 7 45
SRP_00091 0.77 0.81 0.74 77 52871 31 8 19 4 18
SRP_00099 0.54 0.55 0.54 52 44754 45 5 39 1 43
SRP_00104 0.42 0.46 0.39 32 34371 52 12 38 2 37
SRP_00124 0.42 0.41 0.42 35 37045 51 5 43 3 50
SRP_00127 0.69 0.68 0.70 81 72275 39 11 23 5 38
SRP_00130 0.18 0.18 0.19 17 49679 74 10 64 0 80
SRP_00136 0.44 0.49 0.40 35 44166 60 15 37 8 36
SRP_00142 0.16 0.16 0.15 15 45959 84 8 74 2 78
SRP_00143 0.37 0.38 0.36 38 49664 70 6 62 2 63
SRP_00144 0.39 0.38 0.39 36 46573 60 8 48 4 58
SRP_00148 0.19 0.20 0.18 17 49991 87 11 67 9 70
SRP_00152 0.36 0.35 0.38 36 45658 60 5 54 1 67
SRP_00153 0.68 0.64 0.72 69 46875 29 6 21 2 38
SRP_00157 0.35 0.38 0.33 27 44172 60 15 39 6 45
SRP_00160 0.26 0.25 0.27 33 86199 88 6 82 0 97
SRP_00175 0.51 0.52 0.51 44 44464 49 7 36 6 40
SRP_00178 0.74 0.71 0.78 72 45661 23 2 18 3 30
SRP_00179 0.79 0.75 0.84 79 45962 19 1 14 4 26
SRP_00181 0.27 0.25 0.30 26 45968 62 2 60 0 76
SRP_00182 0.59 0.56 0.61 57 45963 37 5 31 1 44
SRP_00184 0.00 0.00 0.00 0 45968 88 12 76 0 100
SRP_00188 0.20 0.20 0.21 16 31048 61 8 53 0 63
SRP_00197 0.31 0.31 0.32 31 49358 67 4 62 1 70
SRP_00198 0.46 0.45 0.47 46 50943 56 3 48 5 57
SRP_00199 0.31 0.30 0.33 31 50945 67 3 61 3 74
SRP_00201 0.80 0.76 0.84 84 47795 19 1 15 3 27
SRP_00202 0.78 0.79 0.78 92 54167 30 3 23 4 25
SRP_00203 0.64 0.64 0.64 67 48724 44 2 35 7 38
SRP_00204 0.77 0.75 0.79 77 49043 22 5 16 1 25
SRP_00206 0.76 0.74 0.78 80 47793 24 5 17 2 28
SRP_00230 0.61 0.60 0.62 54 39534 35 3 30 2 36
SRP_00240 0.48 0.47 0.50 48 48109 50 5 43 2 54
SRP_00241 0.55 0.59 0.52 48 45963 47 13 32 2 34
SRP_00252 0.62 0.63 0.62 56 49050 38 9 26 3 33
SRP_00253 0.62 0.64 0.60 58 49358 45 9 30 6 32
SRP_00254 0.51 0.54 0.48 33 32316 42 9 27 6 28
SRP_00255 0.59 0.60 0.58 56 47489 47 2 39 6 38
SRP_00257 0.86 0.83 0.88 90 50938 17 1 11 5 18
SRP_00260 0.75 0.77 0.73 77 47789 31 6 23 2 23
SRP_00265 0.54 0.59 0.49 43 41241 53 15 29 9 30
SRP_00276 0.81 0.79 0.84 87 50617 18 0 17 1 23
SRP_00278 0.70 0.69 0.72 72 45956 30 3 25 2 33
SRP_00301 0.86 0.86 0.86 74 32554 18 1 11 6 12
SRP_00302 0.44 0.44 0.44 36 41535 57 2 43 12 45
SRP_00308 0.57 0.56 0.58 49 36231 37 3 32 2 39
SRP_00322 0.57 0.55 0.58 60 48413 44 3 40 1 49
SRP_00323 0.78 0.77 0.80 82 47176 23 0 20 3 25
SRP_00328 0.64 0.65 0.63 55 39533 38 8 25 5 30
SRP_00329 0.67 0.66 0.67 56 39820 28 4 23 1 29
SRP_00330 0.41 0.42 0.40 40 49987 61 5 54 2 55
SRP_00331 0.63 0.61 0.66 53 37595 30 2 25 3 34
SRP_00332 0.80 0.77 0.82 82 50303 19 1 17 1 24
SRP_00339 0.51 0.53 0.49 50 44449 54 8 44 2 44
SRP_00340 0.54 0.57 0.51 47 41236 47 9 36 2 35
SRP_00361 0.54 0.54 0.54 49 41814 50 3 39 8 41
SRP_00371 0.74 0.73 0.74 60 36234 25 3 18 4 22

^top



Performance of Carnac(seed) - scored lower in this pairwise comparison

1. Total counts & total scores for Carnac(seed)

Total Base Pair Counts
Total TP 265
Total TN 5425526
Total FP 62
Total FP CONTRA 3
Total FP INCONS 35
Total FP COMP 24
Total FN 10202
Total Scores
MCC 0.149
Average MCC ± 95% Confidence Intervals 0.056 ± 0.034
Sensitivity 0.025
Positive Predictive Value 0.875
Nr of predictions 118

^top



2. Individual counts for Carnac(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00001 0.00 0.00 0.00 0 34191 0 0 0 0 75
ASE_00002 0.00 0.00 0.00 0 35511 0 0 0 0 73
ASE_00009 0.00 0.00 0.00 0 26106 0 0 0 0 67
ASE_00014 0.00 0.00 0.00 0 54285 0 0 0 0 106
ASE_00116 0.00 0.00 0.00 0 31626 0 0 0 0 66
ASE_00120 0.00 0.00 0.00 0 46971 0 0 0 0 94
ASE_00121 0.00 0.00 0.00 0 45150 0 0 0 0 92
ASE_00145 0.00 0.00 0.00 0 70125 0 0 0 0 111
ASE_00154 0.00 0.00 0.00 0 65341 0 0 0 0 107
ASE_00187 0.00 0.00 0.00 0 68635 0 0 0 0 115
ASE_00191 0.00 0.00 0.00 0 29646 0 0 0 0 61
ASE_00196 0.00 0.00 0.00 0 31626 0 0 0 0 75
ASE_00205 0.00 0.00 0.00 0 46056 0 0 0 0 90
ASE_00209 0.00 0.00 0.00 0 43071 0 0 0 0 88
ASE_00210 0.00 0.00 0.00 0 27261 0 0 0 0 65
ASE_00213 0.00 0.00 0.00 0 27261 0 0 0 0 66
ASE_00346 0.00 0.00 0.00 0 48205 0 0 0 0 97
ASE_00359 0.00 0.00 0.00 0 42778 0 0 0 0 80
ASE_00360 0.00 0.00 0.00 0 29403 0 0 0 0 65
ASE_00417 0.00 0.00 0.00 0 54615 0 0 0 0 100
ASE_00418 0.00 0.00 0.00 0 38781 0 0 0 0 75
CRW_00016 0.00 0.00 0.00 0 77421 0 0 0 0 120
CRW_00610 0.00 0.00 0.00 0 36315 0 0 0 0 81
CRW_00613 0.00 0.00 0.00 0 34980 0 0 0 0 78
CRW_00618 0.00 0.00 0.00 0 56280 0 0 0 0 61
CRW_00633 0.00 0.00 0.00 0 63546 0 0 0 0 106
CRW_00634 0.00 0.00 0.00 0 64620 0 0 0 0 94
CRW_00670 0.00 0.00 0.00 0 70125 0 0 0 0 120
CRW_00671 0.00 0.00 0.00 0 62835 0 0 0 0 117
CRW_00672 0.00 0.00 0.00 0 72390 0 0 0 0 111
CRW_00674 0.00 0.00 0.00 0 83436 0 0 0 0 124
CRW_00676 0.00 0.00 0.00 0 93528 0 0 0 0 115
CRW_00692 0.00 0.00 0.00 0 68635 0 0 0 0 90
PDB_00827 0.00 0.00 0.00 0 27028 0 0 0 0 81
RFA_00598 0.00 0.00 0.00 0 84666 0 0 0 0 101
RFA_00606 0.17 0.08 0.38 3 21313 6 1 4 1 36
RFA_00620 0.13 0.08 0.23 3 21932 11 2 8 1 36
RFA_00621 0.00 0.00 0.00 0 46360 0 0 0 0 76
RFA_00623 0.00 0.00 0.00 0 43956 0 0 0 0 78
RFA_00624 0.00 0.00 0.00 0 30135 0 0 0 0 61
RFA_00626 0.67 0.52 0.87 45 56564 11 0 7 4 42
RFA_00627 0.69 0.55 0.86 48 56897 10 0 8 2 39
RFA_00628 0.66 0.43 1.00 37 57254 6 0 0 6 49
RFA_00639 0.00 0.00 0.00 0 54608 7 0 7 0 87
RFA_00814 0.66 0.44 1.00 18 25182 2 0 0 2 23
RFA_00815 0.75 0.56 1.00 23 24508 1 0 0 1 18
RFA_00816 0.66 0.44 1.00 18 23202 2 0 0 2 23
RFA_00817 0.80 0.66 0.96 27 21917 1 0 1 0 14
RFA_00818 0.70 0.49 1.00 20 20281 3 0 0 3 21
RFA_00819 0.75 0.56 1.00 23 27943 2 0 0 2 18
SRP_00009 0.00 0.00 0.00 0 46665 0 0 0 0 102
SRP_00011 0.00 0.00 0.00 0 46360 0 0 0 0 104
SRP_00012 0.00 0.00 0.00 0 46360 0 0 0 0 101
SRP_00015 0.00 0.00 0.00 0 46360 0 0 0 0 99
SRP_00016 0.00 0.00 0.00 0 47895 0 0 0 0 109
SRP_00018 0.00 0.00 0.00 0 38503 0 0 0 0 81
SRP_00019 0.00 0.00 0.00 0 36046 0 0 0 0 91
SRP_00069 0.00 0.00 0.00 0 34453 0 0 0 0 85
SRP_00073 0.00 0.00 0.00 0 39060 0 0 0 0 71
SRP_00079 0.00 0.00 0.00 0 41905 0 0 0 0 88
SRP_00085 0.00 0.00 0.00 0 45451 0 0 0 0 101
SRP_00087 0.00 0.00 0.00 0 37401 0 0 0 0 58
SRP_00091 0.00 0.00 0.00 0 52975 0 0 0 0 95
SRP_00099 0.00 0.00 0.00 0 44850 0 0 0 0 95
SRP_00104 0.00 0.00 0.00 0 34453 0 0 0 0 69
SRP_00124 0.00 0.00 0.00 0 37128 0 0 0 0 85
SRP_00127 0.00 0.00 0.00 0 72390 0 0 0 0 119
SRP_00130 0.00 0.00 0.00 0 49770 0 0 0 0 97
SRP_00136 0.00 0.00 0.00 0 44253 0 0 0 0 71
SRP_00142 0.00 0.00 0.00 0 46056 0 0 0 0 93
SRP_00143 0.00 0.00 0.00 0 49770 0 0 0 0 101
SRP_00144 0.00 0.00 0.00 0 46665 0 0 0 0 94
SRP_00148 0.00 0.00 0.00 0 50086 0 0 0 0 87
SRP_00152 0.00 0.00 0.00 0 45753 0 0 0 0 103
SRP_00153 0.00 0.00 0.00 0 46971 0 0 0 0 107
SRP_00157 0.00 0.00 0.00 0 44253 0 0 0 0 72
SRP_00160 0.00 0.00 0.00 0 86320 0 0 0 0 130
SRP_00175 0.00 0.00 0.00 0 44551 0 0 0 0 84
SRP_00178 0.00 0.00 0.00 0 45753 0 0 0 0 102
SRP_00179 0.00 0.00 0.00 0 46056 0 0 0 0 105
SRP_00181 0.00 0.00 0.00 0 46056 0 0 0 0 102
SRP_00182 0.00 0.00 0.00 0 46056 0 0 0 0 101
SRP_00184 0.00 0.00 0.00 0 46056 0 0 0 0 100
SRP_00188 0.00 0.00 0.00 0 31125 0 0 0 0 79
SRP_00197 0.00 0.00 0.00 0 49455 0 0 0 0 101
SRP_00198 0.00 0.00 0.00 0 51040 0 0 0 0 103
SRP_00199 0.00 0.00 0.00 0 51040 0 0 0 0 105
SRP_00201 0.00 0.00 0.00 0 47895 0 0 0 0 111
SRP_00202 0.00 0.00 0.00 0 54285 0 0 0 0 117
SRP_00203 0.00 0.00 0.00 0 48828 0 0 0 0 105
SRP_00204 0.00 0.00 0.00 0 49141 0 0 0 0 102
SRP_00206 0.00 0.00 0.00 0 47895 0 0 0 0 108
SRP_00230 0.00 0.00 0.00 0 39621 0 0 0 0 90
SRP_00240 0.00 0.00 0.00 0 48205 0 0 0 0 102
SRP_00241 0.00 0.00 0.00 0 46056 0 0 0 0 82
SRP_00252 0.00 0.00 0.00 0 49141 0 0 0 0 89
SRP_00253 0.00 0.00 0.00 0 49455 0 0 0 0 90
SRP_00254 0.00 0.00 0.00 0 32385 0 0 0 0 61
SRP_00255 0.00 0.00 0.00 0 47586 0 0 0 0 94
SRP_00257 0.00 0.00 0.00 0 51040 0 0 0 0 108
SRP_00260 0.00 0.00 0.00 0 47895 0 0 0 0 100
SRP_00265 0.00 0.00 0.00 0 41328 0 0 0 0 73
SRP_00276 0.00 0.00 0.00 0 50721 0 0 0 0 110
SRP_00278 0.00 0.00 0.00 0 46056 0 0 0 0 105
SRP_00301 0.00 0.00 0.00 0 32640 0 0 0 0 86
SRP_00302 0.00 0.00 0.00 0 41616 0 0 0 0 81
SRP_00308 0.00 0.00 0.00 0 36315 0 0 0 0 88
SRP_00322 0.00 0.00 0.00 0 48516 0 0 0 0 109
SRP_00323 0.00 0.00 0.00 0 47278 0 0 0 0 107
SRP_00328 0.00 0.00 0.00 0 39621 0 0 0 0 85
SRP_00329 0.00 0.00 0.00 0 39903 0 0 0 0 85
SRP_00330 0.00 0.00 0.00 0 50086 0 0 0 0 95
SRP_00331 0.00 0.00 0.00 0 37675 0 0 0 0 87
SRP_00332 0.00 0.00 0.00 0 50403 0 0 0 0 106
SRP_00339 0.00 0.00 0.00 0 44551 0 0 0 0 94
SRP_00340 0.00 0.00 0.00 0 41328 0 0 0 0 82
SRP_00361 0.00 0.00 0.00 0 41905 0 0 0 0 90
SRP_00371 0.00 0.00 0.00 0 36315 0 0 0 0 82

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.