CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of Afold - scored higher in this pairwise comparison

  4. Performance of Carnac(seed) - scored lower in this pairwise comparison

  5. Compile and download dataset for Afold & Carnac(seed) [.zip] - may take several seconds...


Overview

Metric Afold Carnac(seed)
MCC 0.635 > 0.447
Average MCC ± 95% Confidence Intervals 0.629 ± 0.057 > 0.370 ± 0.074
Sensitivity 0.622 > 0.227
Positive Predictive Value 0.653 < 0.883
Total TP 1329 > 485
Total TN 356962 < 358448
Total FP 938 > 140
Total FP CONTRA 99 > 13
Total FP INCONS 607 > 51
Total FP COMP 232 > 76
Total FN 806 < 1650
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of Afold and Carnac(seed). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for Afold and Carnac(seed)).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for Afold and Carnac(seed)).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for Afold and Carnac(seed). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for Afold and Carnac(seed)).

^top





Performance of Afold - scored higher in this pairwise comparison

1. Total counts & total scores for Afold

Total Base Pair Counts
Total TP 1329
Total TN 356962
Total FP 938
Total FP CONTRA 99
Total FP INCONS 607
Total FP COMP 232
Total FN 806
Total Scores
MCC 0.635
Average MCC ± 95% Confidence Intervals 0.629 ± 0.057
Sensitivity 0.622
Positive Predictive Value 0.653
Nr of predictions 102

^top



2. Individual counts for Afold [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_01484 0.18 0.18 0.18 7 7464 34 4 28 2 32
CRW_01499 0.45 0.44 0.47 18 7963 21 3 17 1 23
CRW_01603 0.82 0.78 0.85 29 7106 8 0 5 3 8
PDB_00012 1.00 1.00 1.00 7 399 3 0 0 3 0
PDB_00213 0.87 0.80 0.94 33 5015 2 0 2 0 8
PDB_00553 1.00 1.00 1.00 11 454 0 0 0 0 0
PDB_00716 -0.01 0.00 0.00 0 2677 25 1 23 1 23
PDB_00810 0.77 0.71 0.86 12 1067 2 0 2 0 5
PDB_01050 0.96 0.92 1.00 12 618 2 0 0 2 1
PDB_01051 0.75 0.69 0.82 9 892 6 0 2 4 4
PDB_01092 0.69 0.60 0.79 31 10114 11 1 7 3 21
PDB_01152 0.96 0.93 1.00 13 548 0 0 0 0 1
RFA_00389 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00390 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00391 0.93 0.87 1.00 13 1418 2 0 0 2 2
RFA_00409 0.97 0.93 1.00 14 1417 2 0 0 2 1
RFA_00416 0.93 0.93 0.93 14 1470 4 0 1 3 1
RFA_00433 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00434 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00436 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00440 1.00 1.00 1.00 15 1470 2 0 0 2 0
RFA_00442 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00446 0.97 0.93 1.00 14 1471 3 0 0 3 1
RFA_00449 0.97 0.93 1.00 14 1471 3 0 0 3 1
RFA_00587 0.87 0.87 0.87 13 4836 14 0 2 12 2
RFA_00603 0.64 0.69 0.60 24 13490 20 4 12 4 11
RFA_00604 0.72 0.74 0.70 26 13493 24 2 9 13 9
RFA_00607 0.57 0.61 0.54 22 17725 25 6 13 6 14
RFA_00609 0.75 0.76 0.74 28 17917 28 2 8 18 9
RFA_00610 0.77 0.76 0.78 28 17169 23 2 6 15 9
RFA_00611 0.79 0.77 0.82 27 13008 13 0 6 7 8
RFA_00613 0.61 0.60 0.62 21 12846 17 4 9 4 14
RFA_00615 0.51 0.51 0.51 18 13331 18 8 9 1 17
RFA_00632 0.40 0.39 0.41 11 4068 18 0 16 2 17
RFA_00636 0.42 0.43 0.43 12 3977 17 2 14 1 16
RFA_00644 -0.01 0.00 0.00 0 2681 21 4 16 1 18
RFA_00645 -0.01 0.00 0.00 0 2401 15 3 11 1 18
RFA_00649 0.65 0.61 0.69 11 2129 8 1 4 3 7
RFA_00653 0.34 0.33 0.35 6 2128 12 3 8 1 12
RFA_00654 0.27 0.28 0.28 5 2397 14 2 11 1 13
RFA_00658 0.69 0.64 0.75 9 1116 5 0 3 2 5
RFA_00659 0.30 0.29 0.33 4 1116 10 1 7 2 10
RFA_00664 0.07 0.07 0.09 1 979 11 0 10 1 13
RFA_00667 0.30 0.29 0.33 4 978 8 1 7 0 10
RFA_00668 0.41 0.43 0.40 6 975 9 0 9 0 8
RFA_00672 0.17 0.15 0.20 2 893 9 0 8 1 11
RFA_00673 0.42 0.36 0.50 5 1118 6 0 5 1 9
RFA_00674 0.69 0.71 0.67 10 1113 6 0 5 1 4
RFA_00675 0.29 0.29 0.31 4 977 9 1 8 0 10
RFA_00677 0.42 0.43 0.43 6 976 9 0 8 1 8
RFA_00678 0.66 0.64 0.69 9 933 4 0 4 0 5
RFA_00680 0.74 0.71 0.77 10 1115 6 0 3 3 4
RFA_00684 0.66 0.64 0.69 9 977 5 0 4 1 5
RFA_00685 0.42 0.36 0.50 5 980 6 0 5 1 9
RFA_00695 0.67 0.64 0.69 9 7008 29 1 3 25 5
RFA_00703 0.67 0.64 0.69 9 4265 18 1 3 14 5
RFA_00704 0.48 0.43 0.55 6 979 7 0 5 2 8
RFA_00705 0.69 0.71 0.67 10 1020 5 0 5 0 4
RFA_00706 0.44 0.36 0.56 5 1026 4 0 4 0 9
RFA_00707 0.07 0.07 0.10 1 1025 9 3 6 0 13
RFA_00708 0.47 0.36 0.63 5 1027 3 0 3 0 9
RFA_00709 0.16 0.14 0.20 2 980 9 0 8 1 12
RFA_00710 0.23 0.21 0.27 3 979 8 0 8 0 11
RFA_00711 0.40 0.36 0.45 5 1024 6 0 6 0 9
RFA_00716 0.06 0.07 0.08 1 933 12 0 12 0 13
RFA_00717 0.64 0.64 0.64 9 889 5 0 5 0 5
RFA_00730 0.69 0.67 0.73 8 892 4 1 2 1 4
RFA_00731 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00733 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00734 0.61 0.67 0.57 8 889 8 1 5 2 4
RFA_00736 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00737 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00745 0.66 0.67 0.67 8 934 5 1 3 1 4
RFA_00749 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00758 0.96 0.92 1.00 11 892 1 0 0 1 1
RFA_00762 0.64 0.67 0.62 8 890 6 1 4 1 4
RFA_00763 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00764 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00765 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00767 0.63 0.56 0.71 10 1877 5 0 4 1 8
RFA_00768 0.48 0.44 0.53 8 1876 8 1 6 1 10
RFA_00769 0.55 0.56 0.56 10 1935 10 3 5 2 8
RFA_00770 0.68 0.56 0.83 10 2004 6 0 2 4 8
RFA_00773 0.71 0.56 0.91 10 1942 5 1 0 4 8
RFA_00781 0.55 0.56 0.55 18 5017 17 2 13 2 14
RFA_00786 0.71 0.72 0.70 23 5017 12 2 8 2 9
RFA_00791 0.50 0.50 0.52 16 5120 16 3 12 1 16
RFA_00792 0.90 0.88 0.93 28 5020 3 0 2 1 4
RFA_00801 0.59 0.59 0.59 19 5018 14 0 13 1 13
RFA_00808 -0.01 0.00 0.00 0 2001 16 2 13 1 16
RFA_00809 0.36 0.38 0.35 6 2128 12 1 10 1 10
SRP_00084 0.81 0.82 0.80 28 5425 7 1 6 0 6
SRP_00105 -0.01 0.00 0.00 0 5532 33 4 29 0 34
SRP_00141 0.86 0.85 0.88 35 6288 5 1 4 0 6
SRP_00146 0.93 0.94 0.92 34 5216 3 1 2 0 2
SRP_00200 0.44 0.44 0.43 16 6866 21 2 19 0 20
SRP_00231 0.46 0.47 0.46 16 5425 19 1 18 0 18
SRP_00273 0.86 0.88 0.85 35 6745 8 1 5 2 5
SRP_00274 0.91 0.92 0.90 35 6864 5 1 3 1 3
SRP_00338 0.92 0.92 0.92 33 5424 4 0 3 1 3
SRP_00357 0.93 0.92 0.94 33 5321 3 1 1 1 3
SRP_00367 0.83 0.86 0.81 30 6749 7 2 5 0 5

^top



Performance of Carnac(seed) - scored lower in this pairwise comparison

1. Total counts & total scores for Carnac(seed)

Total Base Pair Counts
Total TP 485
Total TN 358448
Total FP 140
Total FP CONTRA 13
Total FP INCONS 51
Total FP COMP 76
Total FN 1650
Total Scores
MCC 0.447
Average MCC ± 95% Confidence Intervals 0.370 ± 0.074
Sensitivity 0.227
Positive Predictive Value 0.883
Nr of predictions 102

^top



2. Individual counts for Carnac(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_01484 0.00 0.00 0.00 0 7503 0 0 0 0 39
CRW_01499 0.00 0.00 0.00 0 8001 0 0 0 0 41
CRW_01603 0.00 0.00 0.00 0 7140 0 0 0 0 37
PDB_00012 1.00 1.00 1.00 7 399 3 0 0 3 0
PDB_00213 0.58 0.34 1.00 14 5036 0 0 0 0 27
PDB_00553 0.79 0.64 1.00 7 458 0 0 0 0 4
PDB_00716 0.00 0.00 0.00 0 2701 0 0 0 0 23
PDB_00810 0.64 0.41 1.00 7 1074 0 0 0 0 10
PDB_01050 0.62 0.38 1.00 5 625 1 0 0 1 8
PDB_01051 0.00 0.00 0.00 0 903 0 0 0 0 13
PDB_01092 0.77 0.63 0.94 33 10118 3 1 1 1 19
PDB_01152 0.88 0.79 1.00 11 550 0 0 0 0 3
RFA_00389 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00390 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00391 0.93 0.87 1.00 13 1418 2 0 0 2 2
RFA_00409 0.77 0.60 1.00 9 1422 1 0 0 1 6
RFA_00416 0.93 0.93 0.93 14 1470 4 0 1 3 1
RFA_00433 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00434 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00436 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00440 1.00 1.00 1.00 15 1470 2 0 0 2 0
RFA_00442 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00446 0.97 0.93 1.00 14 1471 2 0 0 2 1
RFA_00449 0.97 0.93 1.00 14 1471 2 0 0 2 1
RFA_00587 0.00 0.00 0.00 0 4851 0 0 0 0 15
RFA_00603 0.56 0.31 1.00 11 13519 2 0 0 2 24
RFA_00604 0.56 0.31 1.00 11 13519 6 0 0 6 24
RFA_00607 0.25 0.08 0.75 3 17762 10 0 1 9 33
RFA_00609 0.27 0.14 0.56 5 17946 10 2 2 6 32
RFA_00610 0.37 0.14 1.00 5 17200 5 0 0 5 32
RFA_00611 0.38 0.14 1.00 5 13036 0 0 0 0 30
RFA_00613 0.24 0.14 0.42 5 12868 7 4 3 0 30
RFA_00615 0.48 0.23 1.00 8 13358 1 0 0 1 27
RFA_00632 0.00 0.00 0.00 0 4095 0 0 0 0 28
RFA_00636 0.00 0.00 0.00 0 4005 0 0 0 0 28
RFA_00644 0.00 0.00 0.00 0 2701 0 0 0 0 18
RFA_00645 0.00 0.00 0.00 0 2415 0 0 0 0 18
RFA_00649 0.00 0.00 0.00 0 2145 0 0 0 0 18
RFA_00653 0.00 0.00 0.00 0 2145 0 0 0 0 18
RFA_00654 0.00 0.00 0.00 0 2415 0 0 0 0 18
RFA_00658 0.40 0.29 0.57 4 1121 5 0 3 2 10
RFA_00659 -0.01 0.00 0.00 0 1124 4 0 4 0 14
RFA_00664 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00667 -0.01 0.00 0.00 0 985 5 1 4 0 14
RFA_00668 0.50 0.43 0.60 6 980 4 0 4 0 8
RFA_00672 0.00 0.00 0.00 0 903 0 0 0 0 13
RFA_00673 0.47 0.36 0.63 5 1120 4 0 3 1 9
RFA_00674 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00675 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00677 0.53 0.43 0.67 6 981 4 0 3 1 8
RFA_00678 0.00 0.00 0.00 0 946 0 0 0 0 14
RFA_00680 0.84 0.71 1.00 10 1118 2 0 0 2 4
RFA_00684 0.84 0.71 1.00 10 980 1 0 0 1 4
RFA_00685 0.59 0.36 1.00 5 985 0 0 0 0 9
RFA_00695 0.00 0.00 0.00 0 7021 0 0 0 0 14
RFA_00703 0.00 0.00 0.00 0 4278 0 0 0 0 14
RFA_00704 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00705 0.84 0.71 1.00 10 1025 0 0 0 0 4
RFA_00706 0.53 0.29 1.00 4 1031 0 0 0 0 10
RFA_00707 0.53 0.29 1.00 4 1031 0 0 0 0 10
RFA_00708 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00709 -0.01 0.00 0.00 0 984 7 0 6 1 14
RFA_00710 -0.01 0.00 0.00 0 987 3 0 3 0 14
RFA_00711 -0.01 0.00 0.00 0 1031 4 0 4 0 14
RFA_00716 0.00 0.00 0.00 0 946 0 0 0 0 14
RFA_00717 0.00 0.00 0.00 0 903 0 0 0 0 14
RFA_00730 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00731 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00733 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00734 0.64 0.42 1.00 5 898 0 0 0 0 7
RFA_00736 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00737 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00745 0.64 0.42 1.00 5 941 1 0 0 1 7
RFA_00749 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00758 0.00 0.00 0.00 0 903 0 0 0 0 12
RFA_00762 0.64 0.42 1.00 5 898 0 0 0 0 7
RFA_00763 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00764 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00765 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00767 0.74 0.56 1.00 10 1881 0 0 0 0 8
RFA_00768 0.74 0.56 1.00 10 1881 0 0 0 0 8
RFA_00769 0.71 0.56 0.91 10 1942 1 1 0 0 8
RFA_00770 0.62 0.39 1.00 7 2009 0 0 0 0 11
RFA_00773 0.71 0.56 0.91 10 1942 1 1 0 0 8
RFA_00781 0.25 0.13 0.50 4 5042 4 1 3 0 28
RFA_00786 0.25 0.13 0.50 4 5042 4 1 3 0 28
RFA_00791 0.00 0.00 0.00 0 5151 0 0 0 0 32
RFA_00792 0.00 0.00 0.00 0 5046 4 1 3 0 32
RFA_00801 0.00 0.00 0.00 0 5050 0 0 0 0 32
RFA_00808 0.00 0.00 0.00 0 2016 0 0 0 0 16
RFA_00809 0.00 0.00 0.00 0 2145 0 0 0 0 16
SRP_00084 0.00 0.00 0.00 0 5460 0 0 0 0 34
SRP_00105 0.00 0.00 0.00 0 5565 0 0 0 0 34
SRP_00141 0.00 0.00 0.00 0 6328 0 0 0 0 41
SRP_00146 0.00 0.00 0.00 0 5253 0 0 0 0 36
SRP_00200 0.00 0.00 0.00 0 6903 0 0 0 0 36
SRP_00231 0.00 0.00 0.00 0 5460 0 0 0 0 34
SRP_00273 0.00 0.00 0.00 0 6786 0 0 0 0 40
SRP_00274 0.00 0.00 0.00 0 6903 0 0 0 0 38
SRP_00338 0.00 0.00 0.00 0 5460 0 0 0 0 36
SRP_00357 0.00 0.00 0.00 0 5356 0 0 0 0 36
SRP_00367 0.00 0.00 0.00 0 6786 0 0 0 0 35

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.