CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of Vsfold4 - scored higher in this pairwise comparison

  4. Performance of NanoFolder - scored lower in this pairwise comparison

  5. Compile and download dataset for Vsfold4 & NanoFolder [.zip] - may take several seconds...


Overview

Metric Vsfold4 NanoFolder
MCC 0.436 > 0.335
Average MCC ± 95% Confidence Intervals 0.480 ± 0.069 > 0.389 ± 0.054
Sensitivity 0.399 > 0.384
Positive Predictive Value 0.485 > 0.301
Total TP 1106 > 1064
Total TN 414716 > 413459
Total FP 1253 < 2599
Total FP CONTRA 129 < 475
Total FP INCONS 1047 < 2000
Total FP COMP 77 < 124
Total FN 1665 < 1707
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of Vsfold4 and NanoFolder. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for Vsfold4 and NanoFolder).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for Vsfold4 and NanoFolder).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for Vsfold4 and NanoFolder. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for Vsfold4 and NanoFolder).

^top





Performance of Vsfold4 - scored higher in this pairwise comparison

1. Total counts & total scores for Vsfold4

Total Base Pair Counts
Total TP 1106
Total TN 414716
Total FP 1253
Total FP CONTRA 129
Total FP INCONS 1047
Total FP COMP 77
Total FN 1665
Total Scores
MCC 0.436
Average MCC ± 95% Confidence Intervals 0.480 ± 0.069
Sensitivity 0.399
Positive Predictive Value 0.485
Nr of predictions 117

^top



2. Individual counts for Vsfold4 [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_01535 0.26 0.24 0.29 9 7109 23 1 21 1 28
CRW_01539 0.00 0.00 0.00 0 7113 30 1 26 3 37
CRW_01563 0.85 0.73 1.00 27 6876 1 0 0 1 10
CRW_01583 0.42 0.35 0.50 13 7114 16 0 13 3 24
CRW_01590 0.00 0.00 0.00 0 7106 34 4 30 0 37
CRW_01593 0.26 0.24 0.28 9 7108 23 4 19 0 28
CRW_01595 0.00 0.00 0.00 0 8482 33 2 31 0 37
CRW_01603 0.00 0.00 0.00 0 7113 27 5 22 0 37
CRW_01617 0.37 0.35 0.41 13 7108 19 2 17 0 24
CRW_01625 0.32 0.32 0.32 12 7103 25 3 22 0 25
PDB_00011 - 0.86 0.75 1.00 9 267 0 0 0 0 3
PDB_00017 - -0.02 0.00 0.00 0 695 8 0 8 0 15
PDB_00036 - -0.01 0.00 0.00 0 3466 26 0 20 6 27
PDB_00070 0.42 0.38 0.47 9 2831 11 0 10 1 15
PDB_00113 - 0.74 0.56 1.00 5 226 0 0 0 0 4
PDB_00119 - 0.95 0.90 1.00 9 244 0 0 0 0 1
PDB_00136 - 0.85 0.77 0.95 20 2394 1 0 1 0 6
PDB_00137 - 0.87 0.77 1.00 10 396 0 0 0 0 3
PDB_00160 - 0.94 0.89 1.00 8 268 0 0 0 0 1
PDB_00178 - 0.39 0.36 0.44 4 519 5 1 4 0 7
PDB_00221 - 0.62 0.56 0.71 5 246 2 0 2 0 4
PDB_00312 - 1.00 1.00 1.00 9 426 0 0 0 0 0
PDB_00326 - 0.31 0.25 0.39 7 3723 11 0 11 0 21
PDB_00341 - 0.55 0.44 0.70 7 693 3 0 3 0 9
PDB_00349 - -0.01 0.00 0.00 0 940 6 0 6 0 14
PDB_00417 - 0.18 0.16 0.21 9 14154 34 1 32 1 47
PDB_00430 0.78 0.72 0.86 18 3219 3 1 2 0 7
PDB_00527 - -0.02 0.00 0.00 0 294 6 1 5 0 5
PDB_00565 - 0.29 0.26 0.34 13 8090 25 1 24 0 37
PDB_00570 - 0.22 0.18 0.29 4 2264 10 1 9 0 18
PDB_00571 0.87 0.80 0.95 20 3300 1 1 0 0 5
PDB_00573 - 0.26 0.24 0.29 8 6642 24 0 20 4 26
PDB_00584 - 0.27 0.24 0.33 4 1473 8 0 8 0 13
PDB_00698 - -0.02 0.00 0.00 0 848 13 0 13 0 19
PDB_00725 - 0.85 0.73 1.00 11 584 0 0 0 0 4
PDB_00734 - 0.62 0.40 1.00 4 227 0 0 0 0 6
PDB_00823 - -0.03 0.00 0.00 0 186 4 0 4 0 9
PDB_00828 0.86 0.74 1.00 20 2465 0 0 0 0 7
PDB_00851 - 0.56 0.44 0.72 18 4728 7 0 7 0 23
PDB_00890 - 0.92 0.86 1.00 12 691 0 0 0 0 2
PDB_00923 - 1.00 1.00 1.00 6 184 0 0 0 0 0
PDB_00994 - 0.53 0.50 0.57 21 10548 19 0 16 3 21
PDB_01009 0.90 0.90 0.90 19 2464 3 0 2 1 2
PDB_01051 0.36 0.38 0.36 5 889 9 2 7 0 8
PDB_01074 - 0.26 0.24 0.29 6 2754 17 0 15 2 19
PDB_01130 - 0.26 0.25 0.29 5 1753 12 2 10 0 15
PDB_01199 - -0.01 0.00 0.00 0 2130 15 1 14 0 24
PDB_01236 0.50 0.43 0.59 23 11589 17 0 16 1 31
PDB_01275 - 0.35 0.31 0.39 22 18280 35 3 31 1 49
RFA_00416 0.97 0.93 1.00 14 1471 3 0 0 3 1
RFA_00434 0.97 0.93 1.00 14 1417 2 0 0 2 1
RFA_00436 0.97 0.93 1.00 14 1417 2 0 0 2 1
RFA_00442 0.97 0.93 1.00 14 1417 2 0 0 2 1
RFA_00446 0.97 0.93 1.00 14 1471 2 0 0 2 1
RFA_00604 0.52 0.49 0.57 17 13500 18 8 5 5 18
RFA_00664 -0.01 0.00 0.00 0 980 11 0 10 1 14
RFA_00674 -0.01 0.00 0.00 0 1118 10 0 10 0 14
RFA_00675 -0.01 0.00 0.00 0 981 9 1 8 0 14
RFA_00677 0.64 0.57 0.73 8 979 5 0 3 2 6
RFA_00684 0.75 0.57 1.00 8 982 1 0 0 1 6
RFA_00703 0.00 0.00 0.00 0 4270 22 2 6 14 14
RFA_00716 -0.01 0.00 0.00 0 936 10 0 10 0 14
RFA_00717 0.39 0.36 0.45 5 892 6 0 6 0 9
RFA_00730 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00763 0.64 0.42 1.00 5 898 0 0 0 0 7
RFA_00765 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00770 -0.01 0.00 0.00 0 2003 15 2 11 2 18
RFA_00773 -0.01 0.00 0.00 0 1936 19 1 16 2 18
RFA_00808 0.62 0.56 0.69 9 2003 4 3 1 0 7
SPR_00030 - 0.70 0.67 0.74 14 2466 5 0 5 0 7
SPR_00066 0.98 0.95 1.00 20 2830 0 0 0 0 1
SPR_00228 0.95 0.90 1.00 18 3810 1 0 0 1 2
SPR_00314 0.75 0.71 0.79 15 2907 4 2 2 0 6
SPR_00331 0.82 0.80 0.84 16 2756 3 2 1 0 4
SPR_00333 0.63 0.62 0.65 13 2755 7 3 4 0 8
SPR_00353 0.75 0.71 0.79 15 2831 4 2 2 0 6
SPR_00398 - -0.01 0.00 0.00 0 2194 19 2 15 2 13
SPR_00452 0.73 0.76 0.70 16 2827 7 3 4 0 5
SPR_00456 1.00 1.00 1.00 21 2829 1 0 0 1 0
SPR_00462 0.76 0.74 0.78 14 2397 5 3 1 1 5
SPR_00476 1.00 1.00 1.00 21 2754 0 0 0 0 0
SPR_00496 1.00 1.00 1.00 21 2754 0 0 0 0 0
SPR_00497 1.00 1.00 1.00 21 2754 0 0 0 0 0
SPR_00502 1.00 1.00 1.00 21 2905 0 0 0 0 0
SPR_00509 1.00 1.00 1.00 20 2906 0 0 0 0 0
SPR_00512 0.97 0.95 1.00 19 2907 0 0 0 0 1
SPR_00542 0.49 0.52 0.46 11 3546 13 4 9 0 10
SPR_00557 0.98 0.95 1.00 20 2830 0 0 0 0 1
SPR_00832 - 0.69 0.67 0.71 10 1639 4 1 3 0 5
SPR_00868 0.98 0.95 1.00 20 2906 0 0 0 0 1
SPR_00932 0.93 0.90 0.95 19 2830 4 0 1 3 2
SPR_00957 1.00 1.00 1.00 21 2982 0 0 0 0 0
SPR_01040 0.98 0.95 1.00 20 2755 0 0 0 0 1
SPR_01143 0.76 0.76 0.76 16 2535 5 3 2 0 5
SRP_00005 - -0.01 0.00 0.00 0 3984 21 2 19 0 24
SRP_00038 0.28 0.27 0.29 9 5019 22 2 20 0 24
SRP_00046 -0.01 0.00 0.00 0 4730 23 0 23 0 32
SRP_00138 - -0.01 0.00 0.00 0 3985 20 5 15 0 27
SRP_00170 0.37 0.36 0.39 9 4072 15 0 14 1 16
SRP_00193 - 0.20 0.19 0.21 7 6870 26 0 26 0 29
SRP_00194 0.33 0.33 0.33 7 3219 14 3 11 0 14
SRP_00200 0.00 0.00 0.00 0 6874 29 2 27 0 36
SRP_00216 0.36 0.33 0.41 9 3381 13 2 11 0 18
SRP_00229 0.36 0.37 0.36 10 5225 18 5 13 0 17
SRP_00233 0.63 0.61 0.67 20 5121 10 2 8 0 13
SRP_00243 0.45 0.39 0.52 12 5128 11 1 10 0 19
SRP_00244 0.12 0.10 0.16 4 6878 21 3 18 0 35
SRP_00266 0.32 0.31 0.33 9 4629 18 2 16 0 20
SRP_00269 -0.01 0.00 0.00 0 5022 28 4 24 0 33
SRP_00273 0.40 0.38 0.44 15 6752 19 3 16 0 25
SRP_00285 -0.01 0.00 0.00 0 3719 22 2 20 0 30
SRP_00303 0.00 0.00 0.00 0 6641 29 3 26 0 32
SRP_00333 -0.01 0.00 0.00 0 5127 24 1 23 0 34
SRP_00338 0.24 0.22 0.28 8 5431 21 2 19 0 28
SRP_00342 0.00 0.00 0.00 0 3993 12 2 10 0 25
SRP_00358 0.00 0.00 0.00 0 6302 26 2 24 0 35
SRP_00383 -0.01 0.00 0.00 0 3220 20 2 18 0 24

^top



Performance of NanoFolder - scored lower in this pairwise comparison

1. Total counts & total scores for NanoFolder

Total Base Pair Counts
Total TP 1064
Total TN 413459
Total FP 2599
Total FP CONTRA 475
Total FP INCONS 2000
Total FP COMP 124
Total FN 1707
Total Scores
MCC 0.335
Average MCC ± 95% Confidence Intervals 0.389 ± 0.054
Sensitivity 0.384
Positive Predictive Value 0.301
Nr of predictions 117

^top



2. Individual counts for NanoFolder [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_01535 0.20 0.24 0.17 9 7088 43 7 36 0 28
CRW_01539 0.10 0.11 0.10 4 7099 40 1 36 3 33
CRW_01563 0.11 0.14 0.11 5 6856 43 5 37 1 32
CRW_01583 0.36 0.41 0.33 15 7095 31 8 22 1 22
CRW_01590 -0.01 0.00 0.00 0 7093 50 7 40 3 37
CRW_01593 0.32 0.38 0.29 14 7091 36 5 30 1 23
CRW_01595 -0.01 0.00 0.00 0 8459 57 10 46 1 37
CRW_01603 0.41 0.46 0.38 17 7095 29 6 22 1 20
CRW_01617 0.20 0.24 0.17 9 7087 45 7 37 1 28
CRW_01625 0.43 0.49 0.38 18 7093 31 7 22 2 19
PDB_00011 - 0.96 0.92 1.00 11 265 0 0 0 0 1
PDB_00017 - 0.66 0.67 0.67 10 688 5 1 4 0 5
PDB_00036 - -0.01 0.00 0.00 0 3457 29 1 28 0 27
PDB_00070 0.25 0.29 0.23 7 2819 24 4 20 0 17
PDB_00113 - 0.94 0.89 1.00 8 223 0 0 0 0 1
PDB_00119 - 0.95 0.90 1.00 9 244 0 0 0 0 1
PDB_00136 - 0.70 0.73 0.68 19 2387 9 1 8 0 7
PDB_00137 - 0.87 0.77 1.00 10 396 0 0 0 0 3
PDB_00160 - 0.89 0.89 0.89 8 267 2 0 1 1 1
PDB_00178 - -0.02 0.00 0.00 0 517 11 1 10 0 11
PDB_00221 - 0.94 0.89 1.00 8 245 0 0 0 0 1
PDB_00312 - 1.00 1.00 1.00 9 426 1 0 0 1 0
PDB_00326 - 0.58 0.64 0.53 18 3707 16 4 12 0 10
PDB_00341 - 0.47 0.44 0.54 7 690 6 0 6 0 9
PDB_00349 - 0.10 0.14 0.10 2 926 18 0 18 0 12
PDB_00417 - 0.00 0.00 0.00 0 14138 58 2 56 0 56
PDB_00430 0.38 0.44 0.34 11 3208 21 6 15 0 14
PDB_00527 - -0.02 0.00 0.00 0 292 8 2 6 0 5
PDB_00565 - 0.24 0.24 0.24 12 8079 37 1 36 0 38
PDB_00570 - -0.01 0.00 0.00 0 2257 21 0 21 0 22
PDB_00571 -0.01 0.00 0.00 0 3286 36 6 29 1 25
PDB_00573 - -0.01 0.00 0.00 0 6622 48 5 43 0 34
PDB_00584 - 0.47 0.53 0.43 9 1464 12 1 11 0 8
PDB_00698 - 0.97 0.95 1.00 18 843 0 0 0 0 1
PDB_00725 - 0.82 0.80 0.86 12 581 2 0 2 0 3
PDB_00734 - 0.83 0.70 1.00 7 224 0 0 0 0 3
PDB_00823 - -0.04 0.00 0.00 0 183 7 0 7 0 9
PDB_00828 0.30 0.33 0.28 9 2453 23 0 23 0 18
PDB_00851 - 0.16 0.17 0.17 7 4711 35 3 32 0 34
PDB_00890 - -0.02 0.00 0.00 0 690 13 2 11 0 14
PDB_00923 - 1.00 1.00 1.00 6 184 2 0 0 2 0
PDB_00994 - 0.40 0.50 0.33 21 10521 44 13 30 1 21
PDB_01009 0.27 0.33 0.23 7 2455 25 3 20 2 14
PDB_01051 -0.02 0.00 0.00 0 885 18 1 17 0 13
PDB_01074 - -0.01 0.00 0.00 0 2747 28 3 25 0 25
PDB_01130 - -0.01 0.00 0.00 0 1742 28 5 23 0 20
PDB_01199 - -0.01 0.00 0.00 0 2116 29 2 27 0 24
PDB_01236 0.31 0.33 0.30 18 11568 43 3 39 1 36
PDB_01275 - 0.04 0.04 0.04 3 18258 75 6 69 0 68
RFA_00416 0.94 1.00 0.88 15 1468 6 2 0 4 0
RFA_00434 0.84 1.00 0.71 15 1410 8 6 0 2 0
RFA_00436 0.86 1.00 0.75 15 1411 7 5 0 2 0
RFA_00442 0.94 1.00 0.88 15 1414 4 2 0 2 0
RFA_00446 0.85 0.93 0.78 14 1467 7 3 1 3 1
RFA_00604 0.42 0.54 0.33 19 13472 53 14 25 14 16
RFA_00664 -0.01 0.00 0.00 0 978 13 0 12 1 14
RFA_00674 0.12 0.14 0.13 2 1112 15 0 14 1 12
RFA_00675 0.28 0.29 0.29 4 976 10 1 9 0 10
RFA_00677 0.62 0.64 0.60 9 975 10 0 6 4 5
RFA_00684 0.74 0.71 0.77 10 977 5 2 1 2 4
RFA_00703 0.26 0.36 0.19 5 4252 32 5 16 11 9
RFA_00716 -0.02 0.00 0.00 0 929 17 0 17 0 14
RFA_00717 0.64 0.64 0.64 9 889 5 2 3 0 5
RFA_00730 0.38 0.42 0.36 5 889 10 3 6 1 7
RFA_00763 0.38 0.42 0.36 5 889 10 3 6 1 7
RFA_00765 0.39 0.42 0.38 5 890 9 2 6 1 7
RFA_00770 -0.01 0.00 0.00 0 1996 24 0 20 4 18
RFA_00773 0.93 1.00 0.86 18 1932 9 3 0 6 0
RFA_00808 0.83 1.00 0.70 16 1993 7 7 0 0 0
SPR_00030 - 0.47 0.52 0.42 11 2459 16 2 13 1 10
SPR_00066 0.47 0.57 0.39 12 2819 20 4 15 1 9
SPR_00228 0.44 0.60 0.32 12 3791 26 15 10 1 8
SPR_00314 0.47 0.57 0.39 12 2895 21 9 10 2 9
SPR_00331 0.48 0.60 0.39 12 2744 19 6 13 0 8
SPR_00333 0.25 0.33 0.21 7 2741 27 10 17 0 14
SPR_00353 0.64 0.81 0.52 17 2817 16 8 8 0 4
SPR_00398 - -0.01 0.00 0.00 0 2185 28 10 16 2 13
SPR_00452 0.27 0.33 0.23 7 2820 24 7 16 1 14
SPR_00456 0.49 0.57 0.43 12 2822 19 5 11 3 9
SPR_00462 0.65 0.74 0.58 14 2391 11 4 6 1 5
SPR_00476 0.47 0.52 0.42 11 2749 17 4 11 2 10
SPR_00496 0.47 0.57 0.40 12 2745 22 7 11 4 9
SPR_00497 0.27 0.33 0.23 7 2745 23 5 18 0 14
SPR_00502 0.27 0.33 0.23 7 2896 26 2 21 3 14
SPR_00509 0.23 0.30 0.19 6 2895 25 8 17 0 14
SPR_00512 0.61 0.75 0.50 15 2896 15 6 9 0 5
SPR_00542 0.26 0.33 0.22 7 3538 28 4 21 3 14
SPR_00557 0.42 0.52 0.34 11 2818 22 8 13 1 10
SPR_00832 - 0.53 0.67 0.43 10 1630 13 5 8 0 5
SPR_00868 0.49 0.57 0.43 12 2898 19 5 11 3 9
SPR_00932 0.43 0.52 0.35 11 2819 21 7 13 1 10
SPR_00957 0.63 0.76 0.53 16 2973 15 7 7 1 5
SPR_01040 0.15 0.19 0.13 4 2744 27 4 23 0 17
SPR_01143 0.41 0.48 0.36 10 2528 18 8 10 0 11
SRP_00005 - -0.01 0.00 0.00 0 3966 39 6 33 0 24
SRP_00038 0.23 0.27 0.20 9 5005 36 1 35 0 24
SRP_00046 0.33 0.38 0.29 12 4712 30 3 26 1 20
SRP_00138 - 0.44 0.52 0.39 14 3969 24 8 14 2 13
SRP_00170 0.29 0.36 0.25 9 4059 28 4 23 1 16
SRP_00193 - 0.16 0.19 0.15 7 6855 43 4 37 2 29
SRP_00194 0.27 0.33 0.23 7 3209 27 5 19 3 14
SRP_00200 0.41 0.47 0.35 17 6855 31 4 27 0 19
SRP_00216 0.44 0.48 0.41 13 3371 19 4 15 0 14
SRP_00229 0.42 0.52 0.35 14 5213 26 8 18 0 13
SRP_00233 0.25 0.30 0.22 10 5106 35 4 31 0 23
SRP_00243 0.47 0.55 0.40 17 5109 25 7 18 0 14
SRP_00244 0.38 0.44 0.34 17 6853 33 2 31 0 22
SRP_00266 0.55 0.66 0.46 19 4615 22 8 14 0 10
SRP_00269 0.46 0.52 0.43 17 5010 24 6 17 1 16
SRP_00273 0.36 0.40 0.33 16 6737 34 6 27 1 24
SRP_00285 0.38 0.43 0.35 13 3704 24 6 18 0 17
SRP_00303 0.36 0.41 0.33 13 6630 27 3 24 0 19
SRP_00333 0.20 0.24 0.19 8 5108 35 3 32 0 26
SRP_00338 0.52 0.56 0.49 20 5419 22 4 17 1 16
SRP_00342 0.20 0.24 0.17 6 3970 29 7 22 0 19
SRP_00358 0.32 0.37 0.28 13 6282 33 4 29 0 22
SRP_00383 -0.01 0.00 0.00 0 3205 35 9 26 0 24

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.