CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of RNAsubopt - scored higher in this pairwise comparison

  4. Performance of NanoFolder - scored lower in this pairwise comparison

  5. Compile and download dataset for RNAsubopt & NanoFolder [.zip] - may take several seconds...


Overview

Metric RNAsubopt NanoFolder
MCC 0.504 > 0.176
Average MCC ± 95% Confidence Intervals 0.499 ± 0.033 > 0.185 ± 0.028
Sensitivity 0.509 > 0.209
Positive Predictive Value 0.501 > 0.152
Total TP 5439 > 2233
Total TN 6514471 > 6510595
Total FP 6104 < 12959
Total FP CONTRA 963 < 2314
Total FP INCONS 4457 < 10188
Total FP COMP 684 > 457
Total FN 5239 < 8445
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of RNAsubopt and NanoFolder. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for RNAsubopt and NanoFolder).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for RNAsubopt and NanoFolder).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for RNAsubopt and NanoFolder. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for RNAsubopt and NanoFolder).

^top





Performance of RNAsubopt - scored higher in this pairwise comparison

1. Total counts & total scores for RNAsubopt

Total Base Pair Counts
Total TP 5439
Total TN 6514471
Total FP 6104
Total FP CONTRA 963
Total FP INCONS 4457
Total FP COMP 684
Total FN 5239
Total Scores
MCC 0.504
Average MCC ± 95% Confidence Intervals 0.499 ± 0.033
Sensitivity 0.509
Positive Predictive Value 0.501
Nr of predictions 110

^top



2. Individual counts for RNAsubopt [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00017 0.30 0.34 0.26 22 50955 83 20 43 20 42
ASE_00083 0.76 0.74 0.78 81 62377 30 2 21 7 29
ASE_00122 0.19 0.19 0.19 17 43274 75 2 72 1 71
ASE_00185 0.69 0.66 0.71 85 73416 38 7 28 3 43
ASE_00190 0.57 0.56 0.58 50 45365 42 7 29 6 40
ASE_00196 0.74 0.73 0.75 55 31553 22 2 16 4 20
ASE_00221 0.54 0.52 0.55 61 64510 53 1 48 4 56
ASE_00227 0.44 0.44 0.45 58 78874 72 11 60 1 75
ASE_00228 0.60 0.63 0.58 54 46878 39 14 25 0 32
ASE_00229 0.73 0.72 0.73 63 42400 28 5 18 5 24
ASE_00231 0.65 0.67 0.64 64 47795 36 5 31 0 32
ASE_00234 0.36 0.34 0.38 44 75350 76 7 65 4 85
ASE_00238 0.42 0.42 0.42 48 64146 70 6 61 3 66
ASE_00248 0.60 0.59 0.62 67 62373 45 1 40 4 47
ASE_00250 0.82 0.82 0.83 107 78874 25 4 18 3 24
ASE_00254 0.41 0.40 0.42 33 36777 50 4 42 4 49
ASE_00255 0.68 0.65 0.71 85 74571 41 3 32 6 45
ASE_00257 0.56 0.56 0.57 54 50945 50 4 37 9 43
ASE_00267 0.60 0.57 0.63 50 45071 37 4 25 8 38
ASE_00270 0.74 0.71 0.76 91 72271 35 1 27 7 37
ASE_00274 0.50 0.50 0.50 51 55510 50 10 40 0 52
ASE_00277 0.46 0.47 0.44 43 48108 55 10 44 1 48
ASE_00279 0.73 0.72 0.74 73 53202 33 3 23 7 28
ASE_00285 0.69 0.66 0.73 80 68896 38 2 28 8 42
ASE_00287 0.72 0.69 0.76 71 54852 32 2 21 9 32
ASE_00294 0.64 0.61 0.68 104 114329 52 0 48 4 67
ASE_00295 0.47 0.45 0.49 59 73800 64 5 56 3 72
ASE_00298 0.47 0.45 0.48 51 67422 60 9 46 5 62
ASE_00318 0.62 0.62 0.61 73 80081 50 14 32 4 45
ASE_00335 0.42 0.42 0.42 49 75348 79 12 57 10 67
ASE_00339 0.52 0.52 0.52 62 80081 69 6 51 12 57
ASE_00340 0.76 0.75 0.76 64 45972 29 6 14 9 21
ASE_00343 0.55 0.57 0.53 64 83315 69 10 47 12 49
ASE_00367 0.31 0.31 0.30 27 42981 68 7 56 5 59
ASE_00370 0.62 0.63 0.61 53 41818 34 1 33 0 31
ASE_00372 0.69 0.69 0.70 69 51904 34 5 25 4 31
ASE_00375 0.47 0.46 0.48 51 60271 59 4 52 3 59
ASE_00376 0.52 0.51 0.52 54 56850 52 4 45 3 51
ASE_00377 0.81 0.80 0.81 86 57524 26 3 17 6 21
ASE_00386 0.54 0.56 0.52 54 50300 54 4 45 5 42
ASE_00393 0.53 0.53 0.54 49 47804 48 1 41 6 44
ASE_00412 0.46 0.45 0.47 47 58210 57 9 45 3 58
ASE_00413 0.56 0.59 0.54 48 44462 44 11 30 3 33
ASE_00416 0.55 0.54 0.58 68 77303 56 9 41 6 59
ASE_00417 0.44 0.45 0.43 45 54510 64 9 51 4 55
ASE_00422 0.49 0.49 0.48 46 46876 55 5 44 6 48
ASE_00428 0.68 0.66 0.70 82 76519 42 3 32 7 43
ASE_00429 - 0.39 0.38 0.40 25 28141 45 3 34 8 41
ASE_00430 0.62 0.62 0.62 60 46874 42 7 30 5 37
ASE_00434 0.56 0.56 0.56 62 68154 57 10 39 8 48
ASE_00437 0.44 0.42 0.46 57 79278 66 4 62 0 78
ASE_00451 0.62 0.59 0.64 74 70761 44 1 40 3 51
CRW_00654 - 0.70 0.69 0.72 85 78092 38 4 29 5 39
CRW_00658 - 0.46 0.48 0.45 49 74581 61 24 37 0 53
CRW_00663 - 0.07 0.08 0.07 5 26035 74 9 57 8 56
CRW_00664 - 0.21 0.24 0.19 19 53529 80 18 62 0 59
CRW_00671 0.65 0.63 0.67 74 62725 43 1 35 7 43
CRW_00672 0.74 0.75 0.74 83 72278 38 6 23 9 28
CRW_00674 0.60 0.61 0.59 76 83308 55 10 42 3 48
CRW_00676 0.56 0.58 0.54 67 93403 71 15 43 13 48
CRW_00688 - 0.75 0.77 0.73 44 21885 17 7 9 1 13
CRW_00721 - 0.52 0.60 0.45 41 58562 65 22 28 15 27
PDB_00078 - 0.64 0.61 0.67 70 49666 34 7 27 0 45
PDB_00278 - 0.41 0.40 0.41 29 24906 41 5 36 0 43
PDB_00571 0.21 0.20 0.23 5 3299 17 6 11 0 20
PDB_00701 - 0.47 0.47 0.48 51 45344 57 5 51 1 58
PDB_00828 0.88 0.78 1.00 21 2464 0 0 0 0 6
PDB_00851 - 0.45 0.39 0.53 16 4723 14 0 14 0 25
PDB_00917 - 0.56 0.54 0.58 51 35157 43 3 34 6 44
PDB_01009 0.46 0.52 0.41 11 2458 16 5 11 0 10
RFA_00770 0.61 0.56 0.67 10 2001 8 0 5 3 8
RFA_00773 0.68 0.56 0.83 10 1941 5 1 1 3 8
RFA_00808 -0.01 0.00 0.00 0 2001 18 2 13 3 16
TMR_00036 0.63 0.65 0.62 66 90418 59 15 26 18 35
TMR_00123 0.46 0.47 0.46 47 66327 57 11 45 1 54
TMR_00180 0.64 0.65 0.63 64 67060 50 9 28 13 35
TMR_00240 0.21 0.24 0.19 20 63082 91 23 65 3 65
TMR_00352 0.47 0.52 0.43 39 95175 94 20 32 42 36
TMR_00353 0.47 0.50 0.44 53 84135 76 22 45 9 52
TMR_00357 0.38 0.40 0.37 37 82927 79 21 43 15 55
TMR_00361 - 0.33 0.40 0.27 29 93420 100 30 49 21 43
TMR_00374 0.23 0.26 0.21 24 81693 99 23 70 6 68
TMR_00384 0.39 0.40 0.39 43 65592 74 12 56 6 65
TMR_00395 0.52 0.50 0.53 55 64517 52 8 40 4 54
TMR_00398 0.67 0.65 0.69 69 65241 37 1 30 6 37
TMR_00399 0.17 0.18 0.15 17 64869 98 27 67 4 77
TMR_00430 0.26 0.27 0.25 29 82504 92 14 74 4 80
TMR_00431 0.64 0.65 0.62 36 51945 72 6 16 50 19
TMR_00443 0.45 0.45 0.45 47 67056 60 15 43 2 57
TMR_00457 0.39 0.40 0.38 41 78497 75 10 58 7 61
TMR_00469 0.48 0.51 0.46 51 64508 66 21 40 5 49
TMR_00502 0.33 0.37 0.31 34 75355 86 18 59 9 59
TMR_00503 0.39 0.40 0.39 44 67783 73 9 60 4 67
TMR_00519 0.20 0.21 0.19 20 63083 97 18 69 10 75
TMR_00528 0.34 0.34 0.34 33 63093 75 13 51 11 64
TMR_00541 0.35 0.35 0.34 37 61668 77 12 59 6 68
TMR_00547 0.58 0.57 0.60 57 65246 43 2 36 5 43
TMR_00562 0.38 0.39 0.37 40 72663 76 13 55 8 62
TMR_00566 0.32 0.33 0.32 33 60622 76 13 58 5 67
TMR_00571 0.28 0.29 0.28 29 60621 84 12 64 8 70
TMR_00584 0.40 0.42 0.38 41 60966 74 9 59 6 56
TMR_00594 0.22 0.23 0.20 24 79283 99 19 75 5 80
TMR_00605 0.44 0.45 0.43 49 74191 67 17 48 2 59
TMR_00609 0.65 0.65 0.65 74 63433 44 8 31 5 39
TMR_00624 0.42 0.44 0.41 47 62366 78 3 65 10 61
TMR_00683 0.16 0.17 0.14 17 82910 111 16 85 10 83
TMR_00698 0.51 0.50 0.51 52 63089 52 11 38 3 52
TMR_00703 0.26 0.26 0.25 26 67426 82 9 67 6 73
TMR_00706 0.52 0.52 0.52 54 66327 53 7 42 4 49
TMR_00711 0.53 0.55 0.51 54 67791 57 17 34 6 44

^top



Performance of NanoFolder - scored lower in this pairwise comparison

1. Total counts & total scores for NanoFolder

Total Base Pair Counts
Total TP 2233
Total TN 6510595
Total FP 12959
Total FP CONTRA 2314
Total FP INCONS 10188
Total FP COMP 457
Total FN 8445
Total Scores
MCC 0.176
Average MCC ± 95% Confidence Intervals 0.185 ± 0.028
Sensitivity 0.209
Positive Predictive Value 0.152
Nr of predictions 110

^top



2. Individual counts for NanoFolder [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00017 0.02 0.03 0.02 2 50914 131 38 86 7 62
ASE_00083 0.22 0.25 0.20 27 62347 108 12 95 1 83
ASE_00122 0.27 0.30 0.24 26 43258 84 13 68 3 62
ASE_00185 0.17 0.19 0.16 24 73382 134 14 116 4 104
ASE_00190 0.24 0.28 0.21 25 45334 93 14 78 1 65
ASE_00196 0.49 0.57 0.42 43 31524 62 16 43 3 32
ASE_00221 0.20 0.21 0.18 25 64484 112 10 101 1 92
ASE_00227 0.28 0.31 0.25 41 78840 122 20 102 0 92
ASE_00228 0.05 0.06 0.04 5 46858 109 17 91 1 81
ASE_00229 0.12 0.14 0.11 12 42380 97 12 82 3 75
ASE_00231 0.37 0.42 0.33 40 47773 84 17 65 2 56
ASE_00234 0.25 0.27 0.23 35 75315 117 10 106 1 94
ASE_00238 0.26 0.30 0.23 34 64111 119 18 98 3 80
ASE_00248 0.13 0.15 0.12 17 62340 124 9 115 0 97
ASE_00250 0.25 0.29 0.23 38 78835 135 17 113 5 93
ASE_00254 0.15 0.16 0.14 13 36761 88 3 79 6 69
ASE_00255 0.11 0.12 0.10 16 74530 146 16 129 1 114
ASE_00257 0.37 0.42 0.33 41 50917 85 15 67 3 56
ASE_00267 0.00 0.00 0.00 0 45036 115 9 105 1 88
ASE_00270 0.19 0.21 0.17 27 72232 132 14 117 1 101
ASE_00274 0.00 0.00 0.00 0 55487 126 12 112 2 103
ASE_00277 0.10 0.12 0.08 11 48071 123 23 100 0 80
ASE_00279 0.14 0.16 0.13 16 53177 109 16 92 1 85
ASE_00285 0.34 0.36 0.31 44 68866 98 14 82 2 78
ASE_00287 0.15 0.17 0.14 17 54824 107 14 91 2 86
ASE_00294 0.12 0.13 0.12 23 114287 172 11 160 1 148
ASE_00295 0.14 0.16 0.13 21 73761 140 19 119 2 110
ASE_00298 0.09 0.10 0.08 11 67389 129 8 120 1 102
ASE_00318 0.03 0.04 0.03 5 80039 156 21 135 0 113
ASE_00335 0.10 0.12 0.09 14 75304 151 22 126 3 102
ASE_00339 0.25 0.29 0.22 34 80042 129 20 104 5 85
ASE_00340 0.06 0.07 0.05 6 45943 110 13 94 3 79
ASE_00343 0.14 0.17 0.12 19 83275 145 31 111 3 94
ASE_00367 0.08 0.09 0.07 8 42960 110 10 93 7 78
ASE_00370 0.03 0.04 0.03 3 41796 109 18 88 3 81
ASE_00372 0.17 0.20 0.15 20 51871 113 19 93 1 80
ASE_00375 0.12 0.14 0.11 15 60246 120 10 107 3 95
ASE_00376 0.00 0.00 0.00 0 56822 136 9 122 5 105
ASE_00377 0.25 0.27 0.23 29 57505 98 10 86 2 78
ASE_00386 0.14 0.16 0.13 15 50284 106 9 95 2 81
ASE_00393 0.31 0.35 0.27 33 47773 90 18 71 1 60
ASE_00412 0.15 0.17 0.14 18 58183 111 14 96 1 87
ASE_00413 0.08 0.10 0.07 8 44438 108 19 86 3 73
ASE_00416 0.39 0.43 0.35 54 77267 103 18 82 3 73
ASE_00417 0.32 0.36 0.28 36 54486 98 18 75 5 64
ASE_00422 0.28 0.31 0.25 29 46856 87 9 77 1 65
ASE_00428 0.24 0.26 0.22 33 76483 121 13 107 1 92
ASE_00429 - 0.08 0.09 0.07 6 28114 84 14 69 1 60
ASE_00430 0.16 0.18 0.14 17 46852 102 10 92 0 80
ASE_00434 0.29 0.34 0.25 37 68116 122 20 92 10 73
ASE_00437 0.00 0.00 0.00 0 79238 164 12 151 1 135
ASE_00451 0.20 0.22 0.19 27 70731 118 8 110 0 98
CRW_00654 - 0.41 0.47 0.36 58 78049 105 18 85 2 66
CRW_00658 - 0.21 0.26 0.16 27 74527 137 28 109 0 75
CRW_00663 - 0.08 0.10 0.07 6 26014 90 17 69 4 55
CRW_00664 - 0.14 0.19 0.11 15 53486 134 42 85 7 63
CRW_00671 0.18 0.21 0.16 24 62682 130 23 106 1 93
CRW_00672 0.36 0.43 0.31 48 72233 114 35 74 5 63
CRW_00674 0.16 0.19 0.14 23 83274 142 19 120 3 101
CRW_00676 0.11 0.14 0.09 16 93347 169 29 136 4 99
CRW_00688 - 0.33 0.40 0.27 23 21859 64 21 42 1 34
CRW_00721 - 0.22 0.31 0.15 21 58515 133 46 71 16 47
PDB_00078 - 0.00 0.00 0.00 0 49644 126 10 116 0 115
PDB_00278 - 0.04 0.06 0.04 4 24875 98 13 84 1 68
PDB_00571 -0.01 0.00 0.00 0 3286 36 6 29 1 25
PDB_00701 - 0.04 0.05 0.04 5 45319 127 14 113 0 104
PDB_00828 0.30 0.33 0.28 9 2453 23 0 23 0 18
PDB_00851 - 0.16 0.17 0.17 7 4711 35 3 32 0 34
PDB_00917 - 0.31 0.34 0.29 32 35136 81 10 67 4 63
PDB_01009 0.27 0.33 0.23 7 2455 25 3 20 2 14
RFA_00770 -0.01 0.00 0.00 0 1996 24 0 20 4 18
RFA_00773 0.93 1.00 0.86 18 1932 9 3 0 6 0
RFA_00808 0.83 1.00 0.70 16 1993 7 7 0 0 0
TMR_00036 0.22 0.29 0.17 29 90359 153 37 100 16 72
TMR_00123 0.19 0.23 0.16 23 66283 126 20 104 2 78
TMR_00180 0.12 0.16 0.10 16 66999 149 35 111 3 83
TMR_00240 0.23 0.31 0.17 26 63041 128 39 84 5 59
TMR_00352 0.07 0.09 0.05 7 95133 171 39 87 45 68
TMR_00353 0.41 0.51 0.32 54 84087 127 49 65 13 51
TMR_00357 0.26 0.34 0.19 31 82869 146 44 84 18 61
TMR_00361 - 0.06 0.10 0.04 7 93368 174 64 89 21 65
TMR_00374 0.09 0.13 0.07 12 81631 174 44 123 7 80
TMR_00384 0.17 0.20 0.14 22 65550 133 26 105 2 86
TMR_00395 0.27 0.31 0.24 34 64478 115 10 98 7 75
TMR_00398 0.05 0.07 0.05 7 65197 139 24 113 2 99
TMR_00399 0.20 0.26 0.16 24 64833 130 38 85 7 70
TMR_00430 0.18 0.23 0.15 25 82450 148 34 112 2 84
TMR_00431 0.24 0.33 0.17 18 51899 127 26 60 41 37
TMR_00443 0.17 0.20 0.15 21 67018 126 24 98 4 83
TMR_00457 0.19 0.25 0.15 25 78443 141 39 99 3 77
TMR_00469 0.10 0.12 0.08 12 64469 145 27 112 6 88
TMR_00502 0.14 0.19 0.11 18 75303 150 45 100 5 75
TMR_00503 0.11 0.14 0.10 15 67741 141 20 120 1 96
TMR_00519 0.04 0.05 0.03 5 63046 141 33 106 2 90
TMR_00528 0.13 0.15 0.11 15 63051 131 20 104 7 82
TMR_00541 0.12 0.14 0.10 15 61630 139 22 109 8 90
TMR_00547 0.28 0.35 0.23 35 65188 122 37 81 4 65
TMR_00562 0.09 0.12 0.08 12 72617 147 27 115 5 90
TMR_00566 0.13 0.16 0.11 16 60582 132 22 106 4 84
TMR_00571 0.05 0.06 0.04 6 60577 149 19 124 6 93
TMR_00584 0.01 0.01 0.01 1 60922 159 33 119 7 96
TMR_00594 0.08 0.10 0.06 10 79238 156 30 123 3 94
TMR_00605 0.28 0.35 0.23 38 74141 129 36 90 3 70
TMR_00609 0.53 0.62 0.45 70 63391 89 28 57 4 43
TMR_00624 0.36 0.44 0.30 47 62325 111 27 82 2 61
TMR_00683 0.01 0.02 0.01 2 82854 176 50 122 4 98
TMR_00698 0.20 0.24 0.17 25 63042 124 27 96 1 79
TMR_00703 0.10 0.12 0.08 12 67376 140 26 114 0 87
TMR_00706 0.14 0.17 0.11 17 66281 133 29 103 1 86
TMR_00711 0.18 0.22 0.14 22 67740 137 41 93 3 76

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.