CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of MXScarna(seed) - scored higher in this pairwise comparison

  4. Performance of NanoFolder - scored lower in this pairwise comparison

  5. Compile and download dataset for MXScarna(seed) & NanoFolder [.zip] - may take several seconds...


Overview

Metric MXScarna(seed) NanoFolder
MCC 0.664 > 0.202
Average MCC ± 95% Confidence Intervals 0.685 ± 0.037 > 0.260 ± 0.038
Sensitivity 0.566 > 0.232
Positive Predictive Value 0.780 > 0.179
Total TP 5159 > 2117
Total TN 4589888 > 4584648
Total FP 1979 < 10017
Total FP CONTRA 172 < 1492
Total FP INCONS 1283 < 8245
Total FP COMP 524 > 280
Total FN 3959 < 7001
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of MXScarna(seed) and NanoFolder. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for MXScarna(seed) and NanoFolder).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for MXScarna(seed) and NanoFolder).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for MXScarna(seed) and NanoFolder. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for MXScarna(seed) and NanoFolder).

^top





Performance of MXScarna(seed) - scored higher in this pairwise comparison

1. Total counts & total scores for MXScarna(seed)

Total Base Pair Counts
Total TP 5159
Total TN 4589888
Total FP 1979
Total FP CONTRA 172
Total FP INCONS 1283
Total FP COMP 524
Total FN 3959
Total Scores
MCC 0.664
Average MCC ± 95% Confidence Intervals 0.685 ± 0.037
Sensitivity 0.566
Positive Predictive Value 0.780
Nr of predictions 126

^top



2. Individual counts for MXScarna(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00017 0.49 0.41 0.59 26 50996 36 2 16 18 38
ASE_00083 0.71 0.56 0.89 62 62411 11 1 7 3 48
ASE_00122 0.66 0.58 0.75 51 43297 20 1 16 3 37
ASE_00185 0.23 0.16 0.32 21 73470 46 8 37 1 107
ASE_00190 0.53 0.42 0.67 38 45394 23 5 14 4 52
ASE_00191 0.82 0.70 0.96 43 29601 4 0 2 2 18
ASE_00196 0.66 0.52 0.85 39 31580 9 0 7 2 36
ASE_00221 0.78 0.69 0.89 81 64529 13 2 8 3 36
ASE_00227 0.28 0.21 0.38 28 78930 47 1 44 2 105
ASE_00228 0.62 0.51 0.76 44 46913 18 2 12 4 42
ASE_00229 0.72 0.62 0.84 54 42422 13 1 9 3 33
ASE_00231 0.49 0.40 0.60 38 47832 27 5 20 2 58
ASE_00234 0.23 0.17 0.32 22 75398 48 3 43 2 107
ASE_00238 0.54 0.39 0.76 44 64203 17 3 11 3 70
ASE_00248 0.81 0.72 0.92 82 62392 11 2 5 4 32
ASE_00250 0.60 0.49 0.73 64 78915 25 4 20 1 67
ASE_00254 0.67 0.59 0.77 48 36794 17 1 13 3 34
ASE_00255 0.61 0.50 0.74 65 74603 26 5 18 3 65
ASE_00257 0.72 0.58 0.90 56 50978 9 4 2 3 41
ASE_00267 0.75 0.65 0.86 57 45084 12 2 7 3 31
ASE_00270 0.39 0.27 0.56 35 72328 32 0 27 5 93
ASE_00274 0.60 0.48 0.77 49 55547 18 1 14 3 54
ASE_00277 0.60 0.48 0.75 44 48146 19 1 14 4 47
ASE_00279 0.78 0.65 0.93 66 53230 8 2 3 3 35
ASE_00285 0.51 0.40 0.64 49 68930 30 2 25 3 73
ASE_00287 0.67 0.54 0.84 56 54879 14 2 9 3 47
ASE_00294 0.67 0.51 0.87 88 114380 16 2 11 3 83
ASE_00295 0.76 0.63 0.92 82 73831 10 2 5 3 49
ASE_00298 0.55 0.42 0.72 47 67463 21 4 14 3 66
ASE_00318 0.63 0.54 0.74 64 80113 37 6 17 14 54
ASE_00321 0.74 0.69 0.79 61 54538 25 4 12 9 27
ASE_00335 0.72 0.66 0.78 76 75369 33 6 15 12 40
ASE_00339 0.44 0.34 0.58 40 80131 37 3 26 8 79
ASE_00340 0.51 0.45 0.59 38 45992 31 3 23 5 47
ASE_00343 0.27 0.20 0.35 23 83371 49 4 38 7 90
ASE_00367 0.79 0.71 0.88 61 43002 11 2 6 3 25
ASE_00370 0.81 0.71 0.92 60 41840 8 1 4 3 24
ASE_00372 0.57 0.45 0.71 45 51940 21 2 16 3 55
ASE_00375 0.50 0.39 0.64 43 60311 26 1 23 2 67
ASE_00376 0.65 0.55 0.76 58 56877 21 2 16 3 47
ASE_00377 0.72 0.58 0.89 62 57560 11 1 7 3 45
ASE_00386 0.75 0.61 0.92 59 50339 8 1 4 3 37
ASE_00393 0.68 0.57 0.82 53 47830 15 1 11 3 40
ASE_00412 0.55 0.42 0.72 44 58250 20 1 16 3 61
ASE_00413 0.68 0.57 0.82 46 44495 14 1 9 4 35
ASE_00416 0.69 0.60 0.80 76 77326 23 3 16 4 51
ASE_00417 0.78 0.63 0.95 63 54549 5 0 3 2 37
ASE_00422 0.58 0.49 0.69 46 46904 24 3 18 3 48
ASE_00427 0.56 0.41 0.76 19 40730 18 1 5 12 27
ASE_00428 0.69 0.60 0.80 75 76542 22 5 14 3 50
ASE_00430 0.55 0.45 0.68 44 46906 24 1 20 3 53
ASE_00434 0.62 0.55 0.71 60 68180 34 5 20 9 50
ASE_00437 0.58 0.50 0.69 67 79304 33 2 28 3 68
ASE_00451 0.77 0.66 0.88 83 70782 14 2 9 3 42
ASE_00454 0.05 0.03 0.09 2 43342 25 1 20 4 61
CRW_00671 0.56 0.37 0.84 43 62784 10 0 8 2 74
CRW_00672 0.43 0.28 0.67 31 72344 17 0 15 2 80
CRW_00674 0.22 0.14 0.37 17 83390 29 5 24 0 107
CRW_00676 0.03 0.02 0.07 2 93500 31 4 22 5 113
CRW_01535 0.89 0.89 0.89 33 7103 8 0 4 4 4
CRW_01590 0.92 0.92 0.92 34 7103 7 0 3 4 3
CRW_01603 0.89 0.89 0.89 33 7103 8 0 4 4 4
PDB_00828 0.82 0.67 1.00 18 2467 0 0 0 0 9
PDB_01051 0.64 0.54 0.78 7 894 5 0 2 3 6
PDB_01236 0.87 0.81 0.94 44 11581 8 0 3 5 10
RFA_00416 0.97 0.93 1.00 14 1471 1 0 0 1 1
RFA_00434 0.97 0.93 1.00 14 1417 1 0 0 1 1
RFA_00436 0.97 0.93 1.00 14 1417 1 0 0 1 1
RFA_00442 0.97 0.93 1.00 14 1417 1 0 0 1 1
RFA_00446 0.97 0.93 1.00 14 1471 1 0 0 1 1
RFA_00604 0.72 0.63 0.81 22 13503 13 3 2 8 13
RFA_00620 0.20 0.18 0.23 7 21915 33 7 16 10 32
RFA_00664 0.69 0.64 0.75 9 978 4 0 3 1 5
RFA_00674 0.69 0.64 0.75 9 1116 4 0 3 1 5
RFA_00675 0.69 0.64 0.75 9 978 4 0 3 1 5
RFA_00677 0.69 0.64 0.75 9 978 4 0 3 1 5
RFA_00684 0.66 0.64 0.69 9 977 5 0 4 1 5
RFA_00703 0.62 0.57 0.67 8 4266 8 1 3 4 6
RFA_00716 0.48 0.43 0.55 6 935 6 0 5 1 8
RFA_00717 0.69 0.64 0.75 9 891 3 0 3 0 5
RFA_00730 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00763 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00765 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00770 0.57 0.56 0.59 10 1999 7 2 5 0 8
RFA_00773 0.55 0.56 0.56 10 1935 8 3 5 0 8
RFA_00808 -0.01 0.00 0.00 0 2000 19 2 14 3 16
RFA_00816 0.96 0.93 1.00 38 23182 15 0 0 15 3
SPR_00314 1.00 1.00 1.00 21 2905 0 0 0 0 0
SPR_00452 0.98 0.95 1.00 20 2830 0 0 0 0 1
SRP_00038 0.73 0.70 0.77 23 5020 9 0 7 2 10
SRP_00046 0.95 0.94 0.97 30 4722 3 0 1 2 2
SRP_00103 0.91 0.89 0.92 91 45352 17 0 8 9 11
SRP_00117 0.81 0.78 0.83 80 45355 24 0 16 8 22
SRP_00144 0.87 0.86 0.87 81 46572 28 0 12 16 13
SRP_00170 0.51 0.48 0.55 12 4073 14 1 9 4 13
SRP_00173 0.78 0.70 0.86 63 38153 15 0 10 5 27
SRP_00194 0.74 0.76 0.73 16 3218 9 0 6 3 5
SRP_00197 0.92 0.90 0.94 91 49358 24 0 6 18 10
SRP_00198 0.90 0.87 0.92 90 50942 24 0 8 16 13
SRP_00200 0.72 0.67 0.77 24 6872 10 1 6 3 12
SRP_00216 0.76 0.70 0.83 19 3380 6 1 3 2 8
SRP_00229 0.40 0.41 0.41 11 5226 20 0 16 4 16
SRP_00233 0.87 0.82 0.93 27 5122 3 0 2 1 6
SRP_00234 0.84 0.76 0.93 65 36245 14 0 5 9 21
SRP_00243 0.78 0.74 0.82 23 5123 6 0 5 1 8
SRP_00244 0.71 0.64 0.78 25 6871 10 1 6 3 14
SRP_00266 0.57 0.59 0.57 17 4626 14 2 11 1 12
SRP_00267 0.89 0.89 0.90 85 44457 21 0 9 12 11
SRP_00269 0.87 0.82 0.93 27 5021 3 0 2 1 6
SRP_00273 0.74 0.65 0.84 26 6755 8 1 4 3 14
SRP_00276 0.93 0.91 0.94 100 50615 13 0 6 7 10
SRP_00278 0.73 0.69 0.77 72 45963 26 0 21 5 33
SRP_00285 0.86 0.80 0.92 24 3715 3 0 2 1 6
SRP_00303 0.71 0.69 0.73 22 6640 13 1 7 5 10
SRP_00308 0.82 0.73 0.91 64 36245 15 0 6 9 24
SRP_00317 0.89 0.87 0.92 85 45058 22 0 7 15 13
SRP_00321 0.76 0.75 0.78 81 47482 30 0 23 7 27
SRP_00322 0.79 0.75 0.83 82 48417 26 0 17 9 27
SRP_00332 0.95 0.93 0.97 99 50301 15 0 3 12 7
SRP_00333 0.92 0.88 0.97 30 5120 2 0 1 1 4
SRP_00337 0.78 0.71 0.84 65 35434 14 1 11 2 26
SRP_00338 0.75 0.72 0.79 26 5427 8 0 7 1 10
SRP_00342 0.71 0.68 0.74 17 3982 8 1 5 2 8
SRP_00358 0.83 0.77 0.90 27 6298 6 1 2 3 8
SRP_00383 0.75 0.67 0.84 16 3221 5 0 3 2 8
TMR_00357 0.19 0.13 0.28 12 82985 38 4 27 7 80

^top



Performance of NanoFolder - scored lower in this pairwise comparison

1. Total counts & total scores for NanoFolder

Total Base Pair Counts
Total TP 2117
Total TN 4584648
Total FP 10017
Total FP CONTRA 1492
Total FP INCONS 8245
Total FP COMP 280
Total FN 7001
Total Scores
MCC 0.202
Average MCC ± 95% Confidence Intervals 0.260 ± 0.038
Sensitivity 0.232
Positive Predictive Value 0.179
Nr of predictions 126

^top



2. Individual counts for NanoFolder [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
ASE_00017 0.02 0.03 0.02 2 50914 131 38 86 7 62
ASE_00083 0.22 0.25 0.20 27 62347 108 12 95 1 83
ASE_00122 0.27 0.30 0.24 26 43258 84 13 68 3 62
ASE_00185 0.17 0.19 0.16 24 73382 134 14 116 4 104
ASE_00190 0.24 0.28 0.21 25 45334 93 14 78 1 65
ASE_00191 0.36 0.43 0.30 26 29559 67 15 46 6 35
ASE_00196 0.49 0.57 0.42 43 31524 62 16 43 3 32
ASE_00221 0.20 0.21 0.18 25 64484 112 10 101 1 92
ASE_00227 0.28 0.31 0.25 41 78840 122 20 102 0 92
ASE_00228 0.05 0.06 0.04 5 46858 109 17 91 1 81
ASE_00229 0.12 0.14 0.11 12 42380 97 12 82 3 75
ASE_00231 0.37 0.42 0.33 40 47773 84 17 65 2 56
ASE_00234 0.25 0.27 0.23 35 75315 117 10 106 1 94
ASE_00238 0.26 0.30 0.23 34 64111 119 18 98 3 80
ASE_00248 0.13 0.15 0.12 17 62340 124 9 115 0 97
ASE_00250 0.25 0.29 0.23 38 78835 135 17 113 5 93
ASE_00254 0.15 0.16 0.14 13 36761 88 3 79 6 69
ASE_00255 0.11 0.12 0.10 16 74530 146 16 129 1 114
ASE_00257 0.37 0.42 0.33 41 50917 85 15 67 3 56
ASE_00267 0.00 0.00 0.00 0 45036 115 9 105 1 88
ASE_00270 0.19 0.21 0.17 27 72232 132 14 117 1 101
ASE_00274 0.00 0.00 0.00 0 55487 126 12 112 2 103
ASE_00277 0.10 0.12 0.08 11 48071 123 23 100 0 80
ASE_00279 0.14 0.16 0.13 16 53177 109 16 92 1 85
ASE_00285 0.34 0.36 0.31 44 68866 98 14 82 2 78
ASE_00287 0.15 0.17 0.14 17 54824 107 14 91 2 86
ASE_00294 0.12 0.13 0.12 23 114287 172 11 160 1 148
ASE_00295 0.14 0.16 0.13 21 73761 140 19 119 2 110
ASE_00298 0.09 0.10 0.08 11 67389 129 8 120 1 102
ASE_00318 0.03 0.04 0.03 5 80039 156 21 135 0 113
ASE_00321 0.11 0.14 0.09 12 54484 122 22 97 3 76
ASE_00335 0.10 0.12 0.09 14 75304 151 22 126 3 102
ASE_00339 0.25 0.29 0.22 34 80042 129 20 104 5 85
ASE_00340 0.06 0.07 0.05 6 45943 110 13 94 3 79
ASE_00343 0.14 0.17 0.12 19 83275 145 31 111 3 94
ASE_00367 0.08 0.09 0.07 8 42960 110 10 93 7 78
ASE_00370 0.03 0.04 0.03 3 41796 109 18 88 3 81
ASE_00372 0.17 0.20 0.15 20 51871 113 19 93 1 80
ASE_00375 0.12 0.14 0.11 15 60246 120 10 107 3 95
ASE_00376 0.00 0.00 0.00 0 56822 136 9 122 5 105
ASE_00377 0.25 0.27 0.23 29 57505 98 10 86 2 78
ASE_00386 0.14 0.16 0.13 15 50284 106 9 95 2 81
ASE_00393 0.31 0.35 0.27 33 47773 90 18 71 1 60
ASE_00412 0.15 0.17 0.14 18 58183 111 14 96 1 87
ASE_00413 0.08 0.10 0.07 8 44438 108 19 86 3 73
ASE_00416 0.39 0.43 0.35 54 77267 103 18 82 3 73
ASE_00417 0.32 0.36 0.28 36 54486 98 18 75 5 64
ASE_00422 0.28 0.31 0.25 29 46856 87 9 77 1 65
ASE_00427 0.06 0.09 0.04 4 40651 105 35 65 5 42
ASE_00428 0.24 0.26 0.22 33 76483 121 13 107 1 92
ASE_00430 0.16 0.18 0.14 17 46852 102 10 92 0 80
ASE_00434 0.29 0.34 0.25 37 68116 122 20 92 10 73
ASE_00437 0.00 0.00 0.00 0 79238 164 12 151 1 135
ASE_00451 0.20 0.22 0.19 27 70731 118 8 110 0 98
ASE_00454 0.24 0.32 0.19 20 43260 103 28 57 18 43
CRW_00671 0.18 0.21 0.16 24 62682 130 23 106 1 93
CRW_00672 0.36 0.43 0.31 48 72233 114 35 74 5 63
CRW_00674 0.16 0.19 0.14 23 83274 142 19 120 3 101
CRW_00676 0.11 0.14 0.09 16 93347 169 29 136 4 99
CRW_01535 0.20 0.24 0.17 9 7088 43 7 36 0 28
CRW_01590 -0.01 0.00 0.00 0 7093 50 7 40 3 37
CRW_01603 0.41 0.46 0.38 17 7095 29 6 22 1 20
PDB_00828 0.30 0.33 0.28 9 2453 23 0 23 0 18
PDB_01051 -0.02 0.00 0.00 0 885 18 1 17 0 13
PDB_01236 0.31 0.33 0.30 18 11568 43 3 39 1 36
RFA_00416 0.94 1.00 0.88 15 1468 6 2 0 4 0
RFA_00434 0.84 1.00 0.71 15 1410 8 6 0 2 0
RFA_00436 0.86 1.00 0.75 15 1411 7 5 0 2 0
RFA_00442 0.94 1.00 0.88 15 1414 4 2 0 2 0
RFA_00446 0.85 0.93 0.78 14 1467 7 3 1 3 1
RFA_00604 0.42 0.54 0.33 19 13472 53 14 25 14 16
RFA_00620 0.16 0.23 0.11 9 21864 80 29 43 8 30
RFA_00664 -0.01 0.00 0.00 0 978 13 0 12 1 14
RFA_00674 0.12 0.14 0.13 2 1112 15 0 14 1 12
RFA_00675 0.28 0.29 0.29 4 976 10 1 9 0 10
RFA_00677 0.62 0.64 0.60 9 975 10 0 6 4 5
RFA_00684 0.74 0.71 0.77 10 977 5 2 1 2 4
RFA_00703 0.26 0.36 0.19 5 4252 32 5 16 11 9
RFA_00716 -0.02 0.00 0.00 0 929 17 0 17 0 14
RFA_00717 0.64 0.64 0.64 9 889 5 2 3 0 5
RFA_00730 0.38 0.42 0.36 5 889 10 3 6 1 7
RFA_00763 0.38 0.42 0.36 5 889 10 3 6 1 7
RFA_00765 0.39 0.42 0.38 5 890 9 2 6 1 7
RFA_00770 -0.01 0.00 0.00 0 1996 24 0 20 4 18
RFA_00773 0.93 1.00 0.86 18 1932 9 3 0 6 0
RFA_00808 0.83 1.00 0.70 16 1993 7 7 0 0 0
RFA_00816 0.00 0.00 0.00 0 23133 89 30 57 2 41
SPR_00314 0.47 0.57 0.39 12 2895 21 9 10 2 9
SPR_00452 0.27 0.33 0.23 7 2820 24 7 16 1 14
SRP_00038 0.23 0.27 0.20 9 5005 36 1 35 0 24
SRP_00046 0.33 0.38 0.29 12 4712 30 3 26 1 20
SRP_00103 0.00 0.00 0.00 0 45319 132 14 118 0 102
SRP_00117 0.20 0.23 0.18 23 45322 106 9 97 0 79
SRP_00144 0.05 0.06 0.05 6 46538 122 17 104 1 88
SRP_00170 0.29 0.36 0.25 9 4059 28 4 23 1 16
SRP_00173 0.00 0.00 0.00 0 38104 124 13 109 2 90
SRP_00194 0.27 0.33 0.23 7 3209 27 5 19 3 14
SRP_00197 0.12 0.14 0.11 14 49324 117 16 101 0 87
SRP_00198 0.07 0.08 0.06 8 50903 129 15 114 0 95
SRP_00200 0.41 0.47 0.35 17 6855 31 4 27 0 19
SRP_00216 0.44 0.48 0.41 13 3371 19 4 15 0 14
SRP_00229 0.42 0.52 0.35 14 5213 26 8 18 0 13
SRP_00233 0.25 0.30 0.22 10 5106 35 4 31 0 23
SRP_00234 0.21 0.24 0.18 21 36200 97 19 75 3 65
SRP_00243 0.47 0.55 0.40 17 5109 25 7 18 0 14
SRP_00244 0.38 0.44 0.34 17 6853 33 2 31 0 22
SRP_00266 0.55 0.66 0.46 19 4615 22 8 14 0 10
SRP_00267 0.04 0.05 0.04 5 44424 122 15 107 0 91
SRP_00269 0.46 0.52 0.43 17 5010 24 6 17 1 16
SRP_00273 0.36 0.40 0.33 16 6737 34 6 27 1 24
SRP_00276 0.19 0.22 0.18 24 50585 112 10 102 0 86
SRP_00278 0.33 0.36 0.30 38 45928 90 14 76 0 67
SRP_00285 0.38 0.43 0.35 13 3704 24 6 18 0 17
SRP_00303 0.36 0.41 0.33 13 6630 27 3 24 0 19
SRP_00308 0.14 0.17 0.12 15 36189 112 18 93 1 73
SRP_00317 0.14 0.16 0.12 16 45018 116 16 100 0 82
SRP_00321 0.44 0.49 0.40 53 47453 81 16 64 1 55
SRP_00322 0.22 0.25 0.21 27 48385 105 8 96 1 82
SRP_00332 0.12 0.14 0.11 15 50269 120 8 111 1 91
SRP_00333 0.20 0.24 0.19 8 5108 35 3 32 0 26
SRP_00337 0.42 0.46 0.38 42 35400 70 15 54 1 49
SRP_00338 0.52 0.56 0.49 20 5419 22 4 17 1 16
SRP_00342 0.20 0.24 0.17 6 3970 29 7 22 0 19
SRP_00358 0.32 0.37 0.28 13 6282 33 4 29 0 22
SRP_00383 -0.01 0.00 0.00 0 3205 35 9 26 0 24
TMR_00357 0.26 0.34 0.19 31 82869 146 44 84 18 61

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.