CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of RNAwolf - scored higher in this pairwise comparison

  4. Performance of Carnac(seed) - scored lower in this pairwise comparison

  5. Compile and download dataset for RNAwolf & Carnac(seed) [.zip] - may take several seconds...


Overview

Metric RNAwolf Carnac(seed)
MCC 0.472 > 0.415
Average MCC ± 95% Confidence Intervals 0.460 ± 0.054 > 0.323 ± 0.066
Sensitivity 0.473 > 0.196
Positive Predictive Value 0.478 < 0.884
Total TP 1236 > 512
Total TN 435959 < 437968
Total FP 1777 > 145
Total FP CONTRA 194 > 14
Total FP INCONS 1158 > 53
Total FP COMP 425 > 78
Total FN 1379 < 2103
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of RNAwolf and Carnac(seed). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for RNAwolf and Carnac(seed)).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for RNAwolf and Carnac(seed)).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for RNAwolf and Carnac(seed). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for RNAwolf and Carnac(seed)).

^top





Performance of RNAwolf - scored higher in this pairwise comparison

1. Total counts & total scores for RNAwolf

Total Base Pair Counts
Total TP 1236
Total TN 435959
Total FP 1777
Total FP CONTRA 194
Total FP INCONS 1158
Total FP COMP 425
Total FN 1379
Total Scores
MCC 0.472
Average MCC ± 95% Confidence Intervals 0.460 ± 0.054
Sensitivity 0.473
Positive Predictive Value 0.478
Nr of predictions 123

^top



2. Individual counts for RNAwolf [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_01484 0.08 0.08 0.09 3 7469 33 1 30 2 36
CRW_01499 0.56 0.54 0.59 22 7964 21 0 15 6 19
CRW_01603 0.10 0.11 0.11 4 7102 34 8 26 0 33
PDB_00005 0.42 0.36 0.50 5 936 6 0 5 1 9
PDB_00012 0.92 0.86 1.00 6 400 2 0 0 2 1
PDB_00213 0.84 0.83 0.85 34 5010 7 0 6 1 7
PDB_00553 0.90 0.82 1.00 9 456 0 0 0 0 2
PDB_00716 0.68 0.57 0.81 13 2685 3 0 3 0 10
PDB_00810 0.55 0.47 0.67 8 1069 4 0 4 0 9
PDB_01001 0.70 0.78 0.64 14 2123 10 1 7 2 4
PDB_01050 0.80 0.77 0.83 10 618 2 0 2 0 3
PDB_01051 1.00 1.00 1.00 13 890 5 0 0 5 0
PDB_01092 0.43 0.38 0.49 20 10112 23 2 19 2 32
PDB_01152 0.92 0.86 1.00 12 549 1 0 0 1 2
RFA_00389 0.69 0.67 0.71 10 1417 7 1 3 3 5
RFA_00390 0.74 0.67 0.83 10 1419 5 0 2 3 5
RFA_00391 0.70 0.60 0.82 9 1420 4 1 1 2 6
RFA_00409 0.80 0.80 0.80 12 1416 9 0 3 6 3
RFA_00416 0.80 0.80 0.80 12 1470 8 0 3 5 3
RFA_00433 0.71 0.67 0.77 10 1418 7 2 1 4 5
RFA_00434 0.80 0.80 0.80 12 1416 8 0 3 5 3
RFA_00436 1.00 1.00 1.00 15 1416 3 0 0 3 0
RFA_00440 0.69 0.67 0.71 10 1471 8 1 3 4 5
RFA_00442 0.87 0.87 0.87 13 1416 6 0 2 4 2
RFA_00446 0.93 0.93 0.93 14 1470 4 0 1 3 1
RFA_00449 0.27 0.27 0.29 4 1471 12 1 9 2 11
RFA_00587 0.93 0.87 1.00 13 4838 16 0 0 16 2
RFA_00603 0.55 0.60 0.50 21 13488 33 5 16 12 14
RFA_00604 0.73 0.77 0.69 27 13491 29 5 7 17 8
RFA_00605 0.15 0.17 0.14 6 15181 46 6 32 8 29
RFA_00607 0.44 0.44 0.44 16 17730 39 1 19 19 20
RFA_00609 0.59 0.62 0.56 23 17914 37 1 17 19 14
RFA_00610 0.09 0.11 0.08 4 17154 53 17 30 6 33
RFA_00611 0.65 0.69 0.62 24 13002 29 3 12 14 11
RFA_00613 0.16 0.17 0.15 6 12839 41 8 27 6 29
RFA_00615 0.10 0.11 0.09 4 13321 42 10 31 1 31
RFA_00632 0.25 0.25 0.27 7 4069 22 1 18 3 21
RFA_00636 0.40 0.39 0.41 11 3978 19 1 15 3 17
RFA_00642 -0.01 0.00 0.00 0 2906 20 2 18 0 18
RFA_00643 -0.01 0.00 0.00 0 2190 21 4 17 0 18
RFA_00644 -0.01 0.00 0.00 0 2682 21 0 19 2 18
RFA_00645 -0.01 0.00 0.00 0 2396 19 4 15 0 18
RFA_00649 0.42 0.44 0.40 8 2125 14 1 11 2 10
RFA_00651 0.10 0.11 0.11 2 2061 17 1 16 0 16
RFA_00653 -0.01 0.00 0.00 0 2123 22 4 18 0 18
RFA_00654 -0.01 0.00 0.00 0 2394 21 3 18 0 18
RFA_00658 0.56 0.50 0.64 7 1117 8 0 4 4 7
RFA_00659 0.20 0.21 0.21 3 1114 14 1 10 3 11
RFA_00664 -0.02 0.00 0.00 0 974 16 0 16 0 14
RFA_00667 0.27 0.29 0.27 4 975 12 1 10 1 10
RFA_00668 0.44 0.43 0.46 6 977 7 0 7 0 8
RFA_00672 -0.01 0.00 0.00 0 893 12 0 10 2 13
RFA_00673 0.36 0.36 0.38 5 1115 10 0 8 2 9
RFA_00674 0.74 0.71 0.77 10 1115 5 0 3 2 4
RFA_00675 -0.01 0.00 0.00 0 979 13 0 11 2 14
RFA_00677 0.55 0.57 0.53 8 975 10 0 7 3 6
RFA_00678 0.30 0.29 0.33 4 934 10 0 8 2 10
RFA_00680 0.42 0.43 0.43 6 1114 12 0 8 4 8
RFA_00684 0.37 0.29 0.50 4 982 4 1 3 0 10
RFA_00685 0.46 0.43 0.50 6 978 6 2 4 0 8
RFA_00695 0.00 0.00 0.00 0 7002 39 2 17 20 14
RFA_00703 0.39 0.43 0.35 6 4261 25 6 5 14 8
RFA_00704 -0.01 0.00 0.00 0 979 11 2 9 0 14
RFA_00705 0.69 0.71 0.67 10 1020 6 0 5 1 4
RFA_00706 -0.01 0.00 0.00 0 1023 12 2 10 0 14
RFA_00707 -0.01 0.00 0.00 0 1025 11 0 10 1 14
RFA_00708 -0.01 0.00 0.00 0 1025 10 1 9 0 14
RFA_00709 -0.01 0.00 0.00 0 978 12 0 12 0 14
RFA_00710 -0.01 0.00 0.00 0 978 12 1 11 0 14
RFA_00711 -0.01 0.00 0.00 0 1022 14 2 11 1 14
RFA_00715 -0.01 0.00 0.00 0 936 10 0 10 0 14
RFA_00716 0.06 0.07 0.07 1 932 13 0 13 0 13
RFA_00717 0.69 0.64 0.75 9 891 4 0 3 1 5
RFA_00730 0.48 0.42 0.56 5 894 6 0 4 2 7
RFA_00731 0.55 0.58 0.54 7 890 9 1 5 3 5
RFA_00733 0.23 0.25 0.23 3 890 12 1 9 2 9
RFA_00734 0.52 0.50 0.55 6 892 7 0 5 2 6
RFA_00736 0.48 0.42 0.56 5 894 7 0 4 3 7
RFA_00737 0.34 0.33 0.36 4 892 9 2 5 2 8
RFA_00745 0.66 0.67 0.67 8 934 8 0 4 4 4
RFA_00749 0.55 0.58 0.54 7 890 9 1 5 3 5
RFA_00758 0.45 0.42 0.50 5 893 5 0 5 0 7
RFA_00762 0.55 0.58 0.54 7 890 9 1 5 3 5
RFA_00763 0.58 0.58 0.58 7 891 8 0 5 3 5
RFA_00764 0.48 0.42 0.56 5 894 6 0 4 2 7
RFA_00765 0.36 0.42 0.33 5 888 13 0 10 3 7
RFA_00767 0.48 0.44 0.53 8 1876 8 1 6 1 10
RFA_00768 0.50 0.44 0.57 8 1877 8 1 5 2 10
RFA_00769 0.41 0.44 0.38 8 1932 13 3 10 0 10
RFA_00770 0.51 0.50 0.53 9 1999 12 2 6 4 9
RFA_00773 0.42 0.39 0.47 7 1938 12 1 7 4 11
RFA_00779 0.26 0.28 0.26 5 1934 14 6 8 0 13
RFA_00781 0.72 0.72 0.72 23 5018 13 0 9 4 9
RFA_00786 0.32 0.31 0.34 10 5021 22 2 17 3 22
RFA_00791 0.41 0.44 0.39 14 5115 23 5 17 1 18
RFA_00792 0.18 0.16 0.21 5 5026 20 2 17 1 27
RFA_00801 0.31 0.31 0.32 10 5019 22 2 19 1 22
RFA_00808 0.68 0.56 0.82 9 2005 2 2 0 0 7
RFA_00809 0.47 0.38 0.60 6 2135 6 0 4 2 10
SPR_00020 0.31 0.35 0.28 7 2676 18 5 13 0 13
SPR_00137 0.49 0.57 0.43 12 2898 18 7 9 2 9
SPR_00273 0.13 0.14 0.14 3 2828 20 3 16 1 18
SPR_00394 0.50 0.52 0.48 11 3632 16 6 6 4 10
SPR_00402 0.73 0.76 0.70 16 2533 10 2 5 3 5
SPR_00721 0.45 0.48 0.43 10 2903 14 0 13 1 11
SPR_00816 0.95 0.95 0.95 19 3466 9 0 1 8 1
SRP_00020 0.93 0.94 0.91 31 5531 12 1 2 9 2
SRP_00058 0.93 0.94 0.91 32 5743 12 1 2 9 2
SRP_00084 0.75 0.76 0.74 26 5425 15 1 8 6 8
SRP_00105 0.18 0.18 0.19 6 5533 26 3 23 0 28
SRP_00134 0.71 0.76 0.67 22 5962 19 3 8 8 7
SRP_00137 0.35 0.40 0.31 10 4154 22 7 15 0 15
SRP_00141 0.89 0.90 0.88 37 6286 11 1 4 6 4
SRP_00146 0.24 0.22 0.28 8 5224 21 1 20 0 28
SRP_00200 0.34 0.33 0.35 12 6869 30 0 22 8 24
SRP_00231 0.64 0.65 0.65 22 5426 15 1 11 3 12
SRP_00273 0.85 0.83 0.87 33 6748 13 1 4 8 7
SRP_00274 0.68 0.68 0.68 26 6865 17 0 12 5 12
SRP_00285 0.61 0.53 0.70 16 3718 12 0 7 5 14
SRP_00338 0.94 0.94 0.94 34 5424 9 0 2 7 2
SRP_00341 0.43 0.45 0.41 14 4719 25 0 20 5 17
SRP_00357 0.94 0.94 0.94 34 5320 8 1 1 6 2
SRP_00367 0.96 0.97 0.94 34 6750 11 1 1 9 1

^top



Performance of Carnac(seed) - scored lower in this pairwise comparison

1. Total counts & total scores for Carnac(seed)

Total Base Pair Counts
Total TP 512
Total TN 437968
Total FP 145
Total FP CONTRA 14
Total FP INCONS 53
Total FP COMP 78
Total FN 2103
Total Scores
MCC 0.415
Average MCC ± 95% Confidence Intervals 0.323 ± 0.066
Sensitivity 0.196
Positive Predictive Value 0.884
Nr of predictions 123

^top



2. Individual counts for Carnac(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_01484 0.00 0.00 0.00 0 7503 0 0 0 0 39
CRW_01499 0.00 0.00 0.00 0 8001 0 0 0 0 41
CRW_01603 0.00 0.00 0.00 0 7140 0 0 0 0 37
PDB_00005 0.88 0.79 1.00 11 935 0 0 0 0 3
PDB_00012 1.00 1.00 1.00 7 399 3 0 0 3 0
PDB_00213 0.58 0.34 1.00 14 5036 0 0 0 0 27
PDB_00553 0.79 0.64 1.00 7 458 0 0 0 0 4
PDB_00716 0.00 0.00 0.00 0 2701 0 0 0 0 23
PDB_00810 0.64 0.41 1.00 7 1074 0 0 0 0 10
PDB_01001 0.00 0.00 0.00 0 2145 0 0 0 0 18
PDB_01050 0.62 0.38 1.00 5 625 1 0 0 1 8
PDB_01051 0.00 0.00 0.00 0 903 0 0 0 0 13
PDB_01092 0.77 0.63 0.94 33 10118 3 1 1 1 19
PDB_01152 0.88 0.79 1.00 11 550 0 0 0 0 3
RFA_00389 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00390 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00391 0.93 0.87 1.00 13 1418 2 0 0 2 2
RFA_00409 0.77 0.60 1.00 9 1422 1 0 0 1 6
RFA_00416 0.93 0.93 0.93 14 1470 4 0 1 3 1
RFA_00433 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00434 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00436 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00440 1.00 1.00 1.00 15 1470 2 0 0 2 0
RFA_00442 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00446 0.97 0.93 1.00 14 1471 2 0 0 2 1
RFA_00449 0.97 0.93 1.00 14 1471 2 0 0 2 1
RFA_00587 0.00 0.00 0.00 0 4851 0 0 0 0 15
RFA_00603 0.56 0.31 1.00 11 13519 2 0 0 2 24
RFA_00604 0.56 0.31 1.00 11 13519 6 0 0 6 24
RFA_00605 0.41 0.17 1.00 6 15219 2 0 0 2 29
RFA_00607 0.25 0.08 0.75 3 17762 10 0 1 9 33
RFA_00609 0.27 0.14 0.56 5 17946 10 2 2 6 32
RFA_00610 0.37 0.14 1.00 5 17200 5 0 0 5 32
RFA_00611 0.38 0.14 1.00 5 13036 0 0 0 0 30
RFA_00613 0.24 0.14 0.42 5 12868 7 4 3 0 30
RFA_00615 0.48 0.23 1.00 8 13358 1 0 0 1 27
RFA_00632 0.00 0.00 0.00 0 4095 0 0 0 0 28
RFA_00636 0.00 0.00 0.00 0 4005 0 0 0 0 28
RFA_00642 0.00 0.00 0.00 0 2926 0 0 0 0 18
RFA_00643 0.00 0.00 0.00 0 2211 0 0 0 0 18
RFA_00644 0.00 0.00 0.00 0 2701 0 0 0 0 18
RFA_00645 0.00 0.00 0.00 0 2415 0 0 0 0 18
RFA_00649 0.00 0.00 0.00 0 2145 0 0 0 0 18
RFA_00651 0.00 0.00 0.00 0 2080 0 0 0 0 18
RFA_00653 0.00 0.00 0.00 0 2145 0 0 0 0 18
RFA_00654 0.00 0.00 0.00 0 2415 0 0 0 0 18
RFA_00658 0.40 0.29 0.57 4 1121 5 0 3 2 10
RFA_00659 -0.01 0.00 0.00 0 1124 4 0 4 0 14
RFA_00664 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00667 -0.01 0.00 0.00 0 985 5 1 4 0 14
RFA_00668 0.50 0.43 0.60 6 980 4 0 4 0 8
RFA_00672 0.00 0.00 0.00 0 903 0 0 0 0 13
RFA_00673 0.47 0.36 0.63 5 1120 4 0 3 1 9
RFA_00674 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00675 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00677 0.53 0.43 0.67 6 981 4 0 3 1 8
RFA_00678 0.00 0.00 0.00 0 946 0 0 0 0 14
RFA_00680 0.84 0.71 1.00 10 1118 2 0 0 2 4
RFA_00684 0.84 0.71 1.00 10 980 1 0 0 1 4
RFA_00685 0.59 0.36 1.00 5 985 0 0 0 0 9
RFA_00695 0.00 0.00 0.00 0 7021 0 0 0 0 14
RFA_00703 0.00 0.00 0.00 0 4278 0 0 0 0 14
RFA_00704 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00705 0.84 0.71 1.00 10 1025 0 0 0 0 4
RFA_00706 0.53 0.29 1.00 4 1031 0 0 0 0 10
RFA_00707 0.53 0.29 1.00 4 1031 0 0 0 0 10
RFA_00708 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00709 -0.01 0.00 0.00 0 984 7 0 6 1 14
RFA_00710 -0.01 0.00 0.00 0 987 3 0 3 0 14
RFA_00711 -0.01 0.00 0.00 0 1031 4 0 4 0 14
RFA_00715 -0.01 0.00 0.00 0 943 3 1 2 0 14
RFA_00716 0.00 0.00 0.00 0 946 0 0 0 0 14
RFA_00717 0.00 0.00 0.00 0 903 0 0 0 0 14
RFA_00730 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00731 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00733 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00734 0.64 0.42 1.00 5 898 0 0 0 0 7
RFA_00736 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00737 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00745 0.64 0.42 1.00 5 941 1 0 0 1 7
RFA_00749 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00758 0.00 0.00 0.00 0 903 0 0 0 0 12
RFA_00762 0.64 0.42 1.00 5 898 0 0 0 0 7
RFA_00763 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00764 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00765 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00767 0.74 0.56 1.00 10 1881 0 0 0 0 8
RFA_00768 0.74 0.56 1.00 10 1881 0 0 0 0 8
RFA_00769 0.71 0.56 0.91 10 1942 1 1 0 0 8
RFA_00770 0.62 0.39 1.00 7 2009 0 0 0 0 11
RFA_00773 0.71 0.56 0.91 10 1942 1 1 0 0 8
RFA_00779 0.74 0.56 1.00 10 1943 0 0 0 0 8
RFA_00781 0.25 0.13 0.50 4 5042 4 1 3 0 28
RFA_00786 0.25 0.13 0.50 4 5042 4 1 3 0 28
RFA_00791 0.00 0.00 0.00 0 5151 0 0 0 0 32
RFA_00792 0.00 0.00 0.00 0 5046 4 1 3 0 32
RFA_00801 0.00 0.00 0.00 0 5050 0 0 0 0 32
RFA_00808 0.00 0.00 0.00 0 2016 0 0 0 0 16
RFA_00809 0.00 0.00 0.00 0 2145 0 0 0 0 16
SPR_00020 0.00 0.00 0.00 0 2701 0 0 0 0 20
SPR_00137 0.00 0.00 0.00 0 2926 0 0 0 0 21
SPR_00273 0.00 0.00 0.00 0 2850 0 0 0 0 21
SPR_00394 0.00 0.00 0.00 0 3655 0 0 0 0 21
SPR_00402 0.00 0.00 0.00 0 2556 0 0 0 0 21
SPR_00721 0.00 0.00 0.00 0 2926 0 0 0 0 21
SPR_00816 0.00 0.00 0.00 0 3486 0 0 0 0 20
SRP_00020 0.00 0.00 0.00 0 5565 0 0 0 0 33
SRP_00058 0.00 0.00 0.00 0 5778 0 0 0 0 34
SRP_00084 0.00 0.00 0.00 0 5460 0 0 0 0 34
SRP_00105 0.00 0.00 0.00 0 5565 0 0 0 0 34
SRP_00134 0.00 0.00 0.00 0 5995 0 0 0 0 29
SRP_00137 0.00 0.00 0.00 0 4186 0 0 0 0 25
SRP_00141 0.00 0.00 0.00 0 6328 0 0 0 0 41
SRP_00146 0.00 0.00 0.00 0 5253 0 0 0 0 36
SRP_00200 0.00 0.00 0.00 0 6903 0 0 0 0 36
SRP_00231 0.00 0.00 0.00 0 5460 0 0 0 0 34
SRP_00273 0.00 0.00 0.00 0 6786 0 0 0 0 40
SRP_00274 0.00 0.00 0.00 0 6903 0 0 0 0 38
SRP_00285 0.00 0.00 0.00 0 3741 0 0 0 0 30
SRP_00338 0.00 0.00 0.00 0 5460 0 0 0 0 36
SRP_00341 0.00 0.00 0.00 0 4753 0 0 0 0 31
SRP_00357 0.00 0.00 0.00 0 5356 0 0 0 0 36
SRP_00367 0.00 0.00 0.00 0 6786 0 0 0 0 35

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.