CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of Afold - scored higher in this pairwise comparison

  4. Performance of CMfinder(seed) - scored lower in this pairwise comparison

  5. Compile and download dataset for Afold & CMfinder(seed) [.zip] - may take several seconds...


Overview

Metric Afold CMfinder(seed)
MCC 0.616 > 0.489
Average MCC ± 95% Confidence Intervals 0.582 ± 0.046 > 0.515 ± 0.046
Sensitivity 0.610 > 0.317
Positive Predictive Value 0.624 < 0.755
Total TP 3397 > 1768
Total TN 2610547 < 2613649
Total FP 2592 > 649
Total FP CONTRA 352 > 40
Total FP INCONS 1696 > 535
Total FP COMP 544 > 74
Total FN 2172 < 3801
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of Afold and CMfinder(seed). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for Afold and CMfinder(seed)).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for Afold and CMfinder(seed)).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for Afold and CMfinder(seed). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for Afold and CMfinder(seed)).

^top





Performance of Afold - scored higher in this pairwise comparison

1. Total counts & total scores for Afold

Total Base Pair Counts
Total TP 3397
Total TN 2610547
Total FP 2592
Total FP CONTRA 352
Total FP INCONS 1696
Total FP COMP 544
Total FN 2172
Total Scores
MCC 0.616
Average MCC ± 95% Confidence Intervals 0.582 ± 0.046
Sensitivity 0.610
Positive Predictive Value 0.624
Nr of predictions 117

^top



2. Individual counts for Afold [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_00013 0.22 0.22 0.22 25 103625 113 13 77 23 90
CRW_00016 0.65 0.62 0.69 74 77313 47 5 29 13 46
CRW_00610 0.42 0.42 0.43 34 36235 46 6 40 0 47
CRW_00613 0.46 0.42 0.51 33 34915 41 2 30 9 45
CRW_00614 0.07 0.09 0.06 5 121689 141 24 53 64 52
CRW_00633 0.45 0.44 0.47 47 63445 58 10 44 4 59
CRW_00670 0.77 0.74 0.81 89 70015 26 0 21 5 31
CRW_00671 0.66 0.64 0.68 75 62725 41 8 27 6 42
CRW_00672 0.74 0.73 0.74 81 72281 36 10 18 8 30
CRW_00674 0.55 0.55 0.55 68 83312 61 8 48 5 56
CRW_00676 0.54 0.56 0.52 64 93406 75 17 41 17 51
CRW_00692 0.57 0.57 0.57 51 68545 54 15 24 15 39
PDB_00012 1.00 1.00 1.00 7 399 3 0 0 3 0
PDB_00213 0.87 0.80 0.94 33 5015 2 0 2 0 8
PDB_00553 1.00 1.00 1.00 11 454 0 0 0 0 0
PDB_00716 -0.01 0.00 0.00 0 2677 25 1 23 1 23
PDB_00810 0.77 0.71 0.86 12 1067 2 0 2 0 5
PDB_00827 0.67 0.59 0.76 48 26965 16 0 15 1 33
PDB_01050 0.96 0.92 1.00 12 618 2 0 0 2 1
PDB_01051 0.75 0.69 0.82 9 892 6 0 2 4 4
PDB_01092 0.69 0.60 0.79 31 10114 11 1 7 3 21
PDB_01152 0.96 0.93 1.00 13 548 0 0 0 0 1
RFA_00603 0.64 0.69 0.60 24 13490 20 4 12 4 11
RFA_00604 0.72 0.74 0.70 26 13493 24 2 9 13 9
RFA_00606 0.67 0.67 0.68 26 21283 26 7 5 14 13
RFA_00607 0.57 0.61 0.54 22 17725 25 6 13 6 14
RFA_00609 0.75 0.76 0.74 28 17917 28 2 8 18 9
RFA_00610 0.77 0.76 0.78 28 17169 23 2 6 15 9
RFA_00611 0.79 0.77 0.82 27 13008 13 0 6 7 8
RFA_00613 0.61 0.60 0.62 21 12846 17 4 9 4 14
RFA_00615 0.51 0.51 0.51 18 13331 18 8 9 1 17
RFA_00620 0.64 0.62 0.67 24 21909 26 3 9 14 15
RFA_00626 0.89 0.86 0.93 75 56535 31 2 4 25 12
RFA_00627 0.48 0.47 0.48 41 56868 50 10 34 6 46
RFA_00628 0.83 0.80 0.85 69 57210 38 1 11 26 17
RFA_00630 0.60 0.60 0.60 52 56866 46 8 27 11 35
RFA_00632 0.40 0.39 0.41 11 4068 18 0 16 2 17
RFA_00636 0.42 0.43 0.43 12 3977 17 2 14 1 16
RFA_00639 0.49 0.49 0.49 43 54528 61 8 36 17 44
RFA_00644 -0.01 0.00 0.00 0 2681 21 4 16 1 18
RFA_00645 -0.01 0.00 0.00 0 2401 15 3 11 1 18
RFA_00649 0.65 0.61 0.69 11 2129 8 1 4 3 7
RFA_00653 0.34 0.33 0.35 6 2128 12 3 8 1 12
RFA_00654 0.27 0.28 0.28 5 2397 14 2 11 1 13
RFA_00658 0.69 0.64 0.75 9 1116 5 0 3 2 5
RFA_00659 0.30 0.29 0.33 4 1116 10 1 7 2 10
RFA_00664 0.07 0.07 0.09 1 979 11 0 10 1 13
RFA_00667 0.30 0.29 0.33 4 978 8 1 7 0 10
RFA_00668 0.41 0.43 0.40 6 975 9 0 9 0 8
RFA_00672 0.17 0.15 0.20 2 893 9 0 8 1 11
RFA_00673 0.42 0.36 0.50 5 1118 6 0 5 1 9
RFA_00674 0.69 0.71 0.67 10 1113 6 0 5 1 4
RFA_00675 0.29 0.29 0.31 4 977 9 1 8 0 10
RFA_00677 0.42 0.43 0.43 6 976 9 0 8 1 8
RFA_00678 0.66 0.64 0.69 9 933 4 0 4 0 5
RFA_00680 0.74 0.71 0.77 10 1115 6 0 3 3 4
RFA_00684 0.66 0.64 0.69 9 977 5 0 4 1 5
RFA_00685 0.42 0.36 0.50 5 980 6 0 5 1 9
RFA_00704 0.48 0.43 0.55 6 979 7 0 5 2 8
RFA_00705 0.69 0.71 0.67 10 1020 5 0 5 0 4
RFA_00706 0.44 0.36 0.56 5 1026 4 0 4 0 9
RFA_00707 0.07 0.07 0.10 1 1025 9 3 6 0 13
RFA_00708 0.47 0.36 0.63 5 1027 3 0 3 0 9
RFA_00709 0.16 0.14 0.20 2 980 9 0 8 1 12
RFA_00710 0.23 0.21 0.27 3 979 8 0 8 0 11
RFA_00711 0.40 0.36 0.45 5 1024 6 0 6 0 9
RFA_00716 0.06 0.07 0.08 1 933 12 0 12 0 13
RFA_00717 0.64 0.64 0.64 9 889 5 0 5 0 5
RFA_00730 0.69 0.67 0.73 8 892 4 1 2 1 4
RFA_00731 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00733 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00734 0.61 0.67 0.57 8 889 8 1 5 2 4
RFA_00736 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00737 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00745 0.66 0.67 0.67 8 934 5 1 3 1 4
RFA_00749 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00758 0.96 0.92 1.00 11 892 1 0 0 1 1
RFA_00762 0.64 0.67 0.62 8 890 6 1 4 1 4
RFA_00763 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00764 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00765 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00769 0.55 0.56 0.56 10 1935 10 3 5 2 8
RFA_00770 0.68 0.56 0.83 10 2004 6 0 2 4 8
RFA_00773 0.71 0.56 0.91 10 1942 5 1 0 4 8
RFA_00781 0.55 0.56 0.55 18 5017 17 2 13 2 14
RFA_00786 0.71 0.72 0.70 23 5017 12 2 8 2 9
RFA_00791 0.50 0.50 0.52 16 5120 16 3 12 1 16
RFA_00792 0.90 0.88 0.93 28 5020 3 0 2 1 4
RFA_00801 0.59 0.59 0.59 19 5018 14 0 13 1 13
RFA_00808 -0.01 0.00 0.00 0 2001 16 2 13 1 16
RFA_00809 0.36 0.38 0.35 6 2128 12 1 10 1 10
RFA_00814 0.35 0.37 0.33 15 25155 42 7 23 12 26
RFA_00815 0.56 0.61 0.51 25 24482 35 8 16 11 16
RFA_00816 0.59 0.66 0.53 27 23169 33 13 11 9 14
RFA_00818 0.19 0.22 0.17 9 20247 48 19 26 3 32
RFA_00819 0.96 0.95 0.98 39 27926 32 0 1 31 2
SRP_00016 0.92 0.89 0.94 97 47792 8 0 6 2 12
SRP_00130 0.24 0.24 0.24 23 49676 72 10 61 1 74
SRP_00142 0.57 0.58 0.57 54 45961 43 5 36 2 39
SRP_00143 0.36 0.38 0.35 38 49662 72 8 62 2 63
SRP_00144 0.74 0.74 0.74 70 46571 31 1 23 7 24
SRP_00197 0.24 0.24 0.25 24 49359 72 7 65 0 77
SRP_00199 0.67 0.66 0.68 69 50939 34 2 30 2 36
SRP_00201 0.84 0.84 0.84 93 47784 20 5 13 2 18
SRP_00202 0.95 0.93 0.96 109 54172 6 0 4 2 8
SRP_00203 0.73 0.75 0.72 79 48718 37 3 28 6 26
SRP_00204 0.68 0.68 0.69 69 49041 33 4 27 2 33
SRP_00206 0.84 0.83 0.84 90 47788 18 5 12 1 18
SRP_00208 0.83 0.82 0.85 83 46567 18 2 13 3 18
SRP_00276 0.86 0.85 0.86 94 50612 16 5 10 1 16
SRP_00278 0.72 0.72 0.71 76 45949 33 4 27 2 29
SRP_00321 0.77 0.77 0.78 83 47479 27 3 21 3 25
SRP_00322 0.60 0.60 0.61 65 48409 43 2 40 1 44
SRP_00323 0.90 0.89 0.90 95 47173 14 0 10 4 12
SRP_00330 0.44 0.44 0.45 42 49992 54 2 50 2 53
SRP_00332 0.90 0.90 0.91 95 50299 15 1 8 6 11
SRP_00339 0.56 0.56 0.56 53 44456 45 5 37 3 41

^top



Performance of CMfinder(seed) - scored lower in this pairwise comparison

1. Total counts & total scores for CMfinder(seed)

Total Base Pair Counts
Total TP 1768
Total TN 2613649
Total FP 649
Total FP CONTRA 40
Total FP INCONS 535
Total FP COMP 74
Total FN 3801
Total Scores
MCC 0.489
Average MCC ± 95% Confidence Intervals 0.515 ± 0.046
Sensitivity 0.317
Positive Predictive Value 0.755
Nr of predictions 117

^top



2. Individual counts for CMfinder(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_00013 0.37 0.14 1.00 16 103724 0 0 0 0 99
CRW_00016 0.36 0.14 0.89 17 77402 2 0 2 0 103
CRW_00610 0.49 0.31 0.78 25 36283 7 1 6 0 56
CRW_00613 0.54 0.33 0.87 26 34950 4 0 4 0 52
CRW_00614 0.46 0.25 0.88 14 121755 9 0 2 7 43
CRW_00633 0.32 0.12 0.81 13 63530 3 1 2 0 93
CRW_00670 0.32 0.18 0.55 22 70085 18 0 18 0 98
CRW_00671 0.20 0.09 0.41 11 62808 16 0 16 0 106
CRW_00672 0.38 0.15 0.94 17 72372 1 0 1 0 94
CRW_00674 0.40 0.17 0.95 21 83414 1 0 1 0 103
CRW_00676 0.12 0.04 0.33 5 93513 10 1 9 0 110
CRW_00692 0.41 0.17 1.00 15 68620 0 0 0 0 75
PDB_00012 1.00 1.00 1.00 7 399 2 0 0 2 0
PDB_00213 0.85 0.73 1.00 30 5020 0 0 0 0 11
PDB_00553 0.00 0.00 0.00 0 465 0 0 0 0 11
PDB_00716 -0.01 0.00 0.00 0 2688 13 0 13 0 23
PDB_00810 0.76 0.59 1.00 10 1071 0 0 0 0 7
PDB_00827 0.25 0.10 0.62 8 27015 5 0 5 0 73
PDB_01050 0.68 0.46 1.00 6 624 1 0 0 1 7
PDB_01051 0.83 0.69 1.00 9 894 4 0 0 4 4
PDB_01092 0.38 0.21 0.69 11 10137 6 0 5 1 41
PDB_01152 0.92 0.86 1.00 12 549 0 0 0 0 2
RFA_00603 0.70 0.60 0.81 21 13504 7 0 5 2 14
RFA_00604 0.63 0.51 0.78 18 13507 8 0 5 3 17
RFA_00606 0.34 0.21 0.57 8 21307 8 2 4 2 31
RFA_00607 0.08 0.06 0.13 2 17750 15 3 11 1 34
RFA_00609 0.32 0.22 0.47 8 17938 12 2 7 3 29
RFA_00610 0.39 0.30 0.52 11 17184 13 2 8 3 26
RFA_00611 0.65 0.51 0.82 18 13019 5 0 4 1 17
RFA_00613 0.70 0.57 0.87 20 12857 4 0 3 1 15
RFA_00615 0.57 0.49 0.68 17 13341 10 0 8 2 18
RFA_00620 0.00 0.00 0.00 0 21939 6 0 6 0 39
RFA_00626 0.44 0.32 0.60 28 56569 22 2 17 3 59
RFA_00627 0.53 0.39 0.72 34 56906 15 3 10 2 53
RFA_00628 0.58 0.43 0.77 37 57243 14 3 8 3 49
RFA_00630 0.45 0.32 0.64 28 56909 17 3 13 1 59
RFA_00632 0.27 0.25 0.29 7 4071 17 0 17 0 21
RFA_00636 0.71 0.64 0.78 18 3982 5 2 3 0 10
RFA_00639 0.00 0.00 0.00 0 54575 44 6 34 4 87
RFA_00644 0.00 0.00 0.00 0 2693 8 0 8 0 18
RFA_00645 0.29 0.17 0.50 3 2409 3 0 3 0 15
RFA_00649 0.35 0.22 0.57 4 2138 3 0 3 0 14
RFA_00653 -0.01 0.00 0.00 0 2137 8 0 8 0 18
RFA_00654 0.38 0.22 0.67 4 2409 2 0 2 0 14
RFA_00658 0.40 0.29 0.57 4 1121 4 0 3 1 10
RFA_00659 0.23 0.14 0.40 2 1123 4 0 3 1 12
RFA_00664 0.40 0.29 0.57 4 983 4 0 3 1 10
RFA_00667 0.40 0.29 0.57 4 983 4 0 3 1 10
RFA_00668 0.35 0.21 0.60 3 985 3 0 2 1 11
RFA_00672 0.41 0.31 0.57 4 896 3 0 3 0 9
RFA_00673 0.23 0.14 0.40 2 1123 4 0 3 1 12
RFA_00674 0.40 0.29 0.57 4 1121 4 0 3 1 10
RFA_00675 0.35 0.21 0.60 3 985 3 0 2 1 11
RFA_00677 0.40 0.29 0.57 4 983 4 0 3 1 10
RFA_00678 0.43 0.29 0.67 4 940 3 0 2 1 10
RFA_00680 0.40 0.29 0.57 4 1121 4 0 3 1 10
RFA_00684 0.43 0.29 0.67 4 984 3 0 2 1 10
RFA_00685 0.32 0.21 0.50 3 984 4 0 3 1 11
RFA_00704 0.32 0.21 0.50 3 984 4 0 3 1 11
RFA_00705 0.40 0.29 0.57 4 1028 4 0 3 1 10
RFA_00706 0.43 0.29 0.67 4 1029 3 0 2 1 10
RFA_00707 0.43 0.29 0.67 4 1029 3 0 2 1 10
RFA_00708 0.40 0.29 0.57 4 1028 4 0 3 1 10
RFA_00709 0.32 0.21 0.50 3 984 4 0 3 1 11
RFA_00710 0.32 0.21 0.50 3 984 3 0 3 0 11
RFA_00711 0.43 0.29 0.67 4 1029 3 0 2 1 10
RFA_00716 0.43 0.29 0.67 4 940 3 0 2 1 10
RFA_00717 0.30 0.21 0.43 3 896 4 0 4 0 11
RFA_00730 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00731 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00733 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00734 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00736 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00737 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00745 0.91 0.83 1.00 10 936 0 0 0 0 2
RFA_00749 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00758 0.86 0.75 1.00 9 894 0 0 0 0 3
RFA_00762 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00763 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00764 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00765 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00769 0.71 0.50 1.00 9 1944 0 0 0 0 9
RFA_00770 0.71 0.50 1.00 9 2007 0 0 0 0 9
RFA_00773 0.00 0.00 0.00 0 1949 4 0 4 0 18
RFA_00781 0.97 0.94 1.00 30 5020 0 0 0 0 2
RFA_00786 0.95 0.91 1.00 29 5021 0 0 0 0 3
RFA_00791 0.92 0.88 0.97 28 5122 1 0 1 0 4
RFA_00792 0.97 0.94 1.00 30 5020 0 0 0 0 2
RFA_00801 0.95 0.91 1.00 29 5021 0 0 0 0 3
RFA_00808 0.68 0.56 0.82 9 2005 2 0 2 0 7
RFA_00809 0.41 0.38 0.46 6 2132 7 1 6 0 10
RFA_00814 0.68 0.51 0.91 21 25177 2 0 2 0 20
RFA_00815 0.68 0.51 0.91 21 24508 2 0 2 0 20
RFA_00816 0.68 0.51 0.91 21 23197 2 0 2 0 20
RFA_00818 0.60 0.49 0.74 20 20274 8 1 6 1 21
RFA_00819 0.62 0.51 0.75 21 27938 7 0 7 0 20
SRP_00016 0.54 0.35 0.83 38 47849 8 0 8 0 71
SRP_00130 0.50 0.32 0.78 31 49730 10 1 8 1 66
SRP_00142 0.46 0.37 0.58 34 45997 26 1 24 1 59
SRP_00143 0.50 0.30 0.86 30 49735 5 0 5 0 71
SRP_00144 0.54 0.37 0.80 35 46621 11 0 9 2 59
SRP_00197 0.46 0.26 0.84 26 49424 5 0 5 0 75
SRP_00199 0.54 0.35 0.82 37 50995 8 0 8 0 68
SRP_00201 0.54 0.35 0.83 39 47848 8 0 8 0 72
SRP_00202 0.43 0.21 0.86 25 54256 4 0 4 0 92
SRP_00203 0.51 0.34 0.77 36 48781 12 0 11 1 69
SRP_00204 0.52 0.34 0.80 35 49097 9 1 8 0 67
SRP_00206 0.55 0.36 0.83 39 47848 8 0 8 0 69
SRP_00208 0.52 0.33 0.83 33 46625 7 1 6 0 68
SRP_00276 0.51 0.33 0.80 36 50676 9 0 9 0 74
SRP_00278 0.52 0.33 0.81 35 46013 8 0 8 0 70
SRP_00321 0.42 0.21 0.82 23 47558 5 1 4 0 85
SRP_00322 0.52 0.33 0.82 36 48472 8 0 8 0 73
SRP_00323 0.54 0.34 0.88 36 47237 5 0 5 0 71
SRP_00330 0.44 0.23 0.85 22 50060 5 0 4 1 73
SRP_00332 0.44 0.25 0.75 27 50367 9 0 9 0 79
SRP_00339 0.56 0.39 0.79 37 44504 10 2 8 0 57

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.