CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of Carnac(seed) - scored higher in this pairwise comparison

  4. Performance of Pknots - scored lower in this pairwise comparison

  5. Compile and download dataset for Carnac(seed) & Pknots [.zip] - may take several seconds...


Overview

Metric Carnac(seed) Pknots
MCC 0.498 > 0.484
Average MCC ± 95% Confidence Intervals 0.382 ± 0.083 < 0.466 ± 0.074
Sensitivity 0.276 < 0.470
Positive Predictive Value 0.907 > 0.509
Total TP 398 < 679
Total TN 155788 > 154892
Total FP 87 < 780
Total FP CONTRA 4 < 99
Total FP INCONS 37 < 557
Total FP COMP 46 < 124
Total FN 1046 > 765
P-value 3.76761760679e-06

^top




Performance plots


  1. Comparison of performance of Carnac(seed) and Pknots. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for Carnac(seed) and Pknots).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for Carnac(seed) and Pknots).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for Carnac(seed) and Pknots. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for Carnac(seed) and Pknots).

^top





Performance of Carnac(seed) - scored higher in this pairwise comparison

1. Total counts & total scores for Carnac(seed)

Total Base Pair Counts
Total TP 398
Total TN 155788
Total FP 87
Total FP CONTRA 4
Total FP INCONS 37
Total FP COMP 46
Total FN 1046
Total Scores
MCC 0.498
Average MCC ± 95% Confidence Intervals 0.382 ± 0.083
Sensitivity 0.276
Positive Predictive Value 0.907
Nr of predictions 90

^top



2. Individual counts for Carnac(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
PDB_00005 0.88 0.79 1.00 11 935 0 0 0 0 3
PDB_00012 1.00 1.00 1.00 7 399 3 0 0 3 0
PDB_00553 0.79 0.64 1.00 7 458 0 0 0 0 4
PDB_00716 0.00 0.00 0.00 0 2701 0 0 0 0 23
PDB_00810 0.64 0.41 1.00 7 1074 0 0 0 0 10
PDB_01001 0.00 0.00 0.00 0 2145 0 0 0 0 18
PDB_01050 0.62 0.38 1.00 5 625 1 0 0 1 8
PDB_01051 0.00 0.00 0.00 0 903 0 0 0 0 13
PDB_01152 0.88 0.79 1.00 11 550 0 0 0 0 3
RFA_00389 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00390 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00391 0.93 0.87 1.00 13 1418 2 0 0 2 2
RFA_00409 0.77 0.60 1.00 9 1422 1 0 0 1 6
RFA_00416 0.93 0.93 0.93 14 1470 4 0 1 3 1
RFA_00433 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00434 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00436 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00440 1.00 1.00 1.00 15 1470 2 0 0 2 0
RFA_00442 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00446 0.97 0.93 1.00 14 1471 2 0 0 2 1
RFA_00449 0.97 0.93 1.00 14 1471 2 0 0 2 1
RFA_00587 0.00 0.00 0.00 0 4851 0 0 0 0 15
RFA_00632 0.00 0.00 0.00 0 4095 0 0 0 0 28
RFA_00636 0.00 0.00 0.00 0 4005 0 0 0 0 28
RFA_00642 0.00 0.00 0.00 0 2926 0 0 0 0 18
RFA_00643 0.00 0.00 0.00 0 2211 0 0 0 0 18
RFA_00644 0.00 0.00 0.00 0 2701 0 0 0 0 18
RFA_00645 0.00 0.00 0.00 0 2415 0 0 0 0 18
RFA_00649 0.00 0.00 0.00 0 2145 0 0 0 0 18
RFA_00651 0.00 0.00 0.00 0 2080 0 0 0 0 18
RFA_00653 0.00 0.00 0.00 0 2145 0 0 0 0 18
RFA_00654 0.00 0.00 0.00 0 2415 0 0 0 0 18
RFA_00658 0.40 0.29 0.57 4 1121 5 0 3 2 10
RFA_00659 -0.01 0.00 0.00 0 1124 4 0 4 0 14
RFA_00664 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00667 -0.01 0.00 0.00 0 985 5 1 4 0 14
RFA_00668 0.50 0.43 0.60 6 980 4 0 4 0 8
RFA_00672 0.00 0.00 0.00 0 903 0 0 0 0 13
RFA_00673 0.47 0.36 0.63 5 1120 4 0 3 1 9
RFA_00674 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00675 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00677 0.53 0.43 0.67 6 981 4 0 3 1 8
RFA_00678 0.00 0.00 0.00 0 946 0 0 0 0 14
RFA_00680 0.84 0.71 1.00 10 1118 2 0 0 2 4
RFA_00684 0.84 0.71 1.00 10 980 1 0 0 1 4
RFA_00685 0.59 0.36 1.00 5 985 0 0 0 0 9
RFA_00703 0.00 0.00 0.00 0 4278 0 0 0 0 14
RFA_00704 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00705 0.84 0.71 1.00 10 1025 0 0 0 0 4
RFA_00706 0.53 0.29 1.00 4 1031 0 0 0 0 10
RFA_00707 0.53 0.29 1.00 4 1031 0 0 0 0 10
RFA_00708 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00709 -0.01 0.00 0.00 0 984 7 0 6 1 14
RFA_00710 -0.01 0.00 0.00 0 987 3 0 3 0 14
RFA_00711 -0.01 0.00 0.00 0 1031 4 0 4 0 14
RFA_00715 -0.01 0.00 0.00 0 943 3 1 2 0 14
RFA_00716 0.00 0.00 0.00 0 946 0 0 0 0 14
RFA_00717 0.00 0.00 0.00 0 903 0 0 0 0 14
RFA_00730 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00731 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00733 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00734 0.64 0.42 1.00 5 898 0 0 0 0 7
RFA_00736 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00737 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00745 0.64 0.42 1.00 5 941 1 0 0 1 7
RFA_00749 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00758 0.00 0.00 0.00 0 903 0 0 0 0 12
RFA_00762 0.64 0.42 1.00 5 898 0 0 0 0 7
RFA_00763 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00764 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00765 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00767 0.74 0.56 1.00 10 1881 0 0 0 0 8
RFA_00768 0.74 0.56 1.00 10 1881 0 0 0 0 8
RFA_00769 0.71 0.56 0.91 10 1942 1 1 0 0 8
RFA_00770 0.62 0.39 1.00 7 2009 0 0 0 0 11
RFA_00773 0.71 0.56 0.91 10 1942 1 1 0 0 8
RFA_00779 0.74 0.56 1.00 10 1943 0 0 0 0 8
RFA_00808 0.00 0.00 0.00 0 2016 0 0 0 0 16
RFA_00809 0.00 0.00 0.00 0 2145 0 0 0 0 16
SPR_00020 0.00 0.00 0.00 0 2701 0 0 0 0 20
SPR_00137 0.00 0.00 0.00 0 2926 0 0 0 0 21
SPR_00273 0.00 0.00 0.00 0 2850 0 0 0 0 21
SPR_00394 0.00 0.00 0.00 0 3655 0 0 0 0 21
SPR_00402 0.00 0.00 0.00 0 2556 0 0 0 0 21
SPR_00721 0.00 0.00 0.00 0 2926 0 0 0 0 21
SPR_00816 0.00 0.00 0.00 0 3486 0 0 0 0 20
SRP_00134 0.00 0.00 0.00 0 5995 0 0 0 0 29
SRP_00137 0.00 0.00 0.00 0 4186 0 0 0 0 25
SRP_00285 0.00 0.00 0.00 0 3741 0 0 0 0 30
SRP_00341 0.00 0.00 0.00 0 4753 0 0 0 0 31

^top



Performance of Pknots - scored lower in this pairwise comparison

1. Total counts & total scores for Pknots

Total Base Pair Counts
Total TP 679
Total TN 154892
Total FP 780
Total FP CONTRA 99
Total FP INCONS 557
Total FP COMP 124
Total FN 765
Total Scores
MCC 0.484
Average MCC ± 95% Confidence Intervals 0.466 ± 0.074
Sensitivity 0.470
Positive Predictive Value 0.509
Nr of predictions 90

^top



2. Individual counts for Pknots [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
PDB_00005 1.00 1.00 1.00 14 932 0 0 0 0 0
PDB_00012 1.00 1.00 1.00 7 399 3 0 0 3 0
PDB_00553 1.00 1.00 1.00 11 454 0 0 0 0 0
PDB_00716 0.69 0.70 0.70 16 2678 8 0 7 1 7
PDB_00810 0.46 0.41 0.54 7 1068 6 2 4 0 10
PDB_01001 0.16 0.17 0.17 3 2127 15 3 12 0 15
PDB_01050 0.92 0.85 1.00 11 619 2 0 0 2 2
PDB_01051 0.31 0.31 0.33 4 891 9 1 7 1 9
PDB_01152 0.88 0.79 1.00 11 550 0 0 0 0 3
RFA_00389 0.69 0.67 0.71 10 1417 6 0 4 2 5
RFA_00390 0.69 0.67 0.71 10 1417 6 0 4 2 5
RFA_00391 0.66 0.67 0.67 10 1416 7 0 5 2 5
RFA_00409 0.65 0.53 0.80 8 1421 4 0 2 2 7
RFA_00416 0.93 0.93 0.93 14 1470 5 0 1 4 1
RFA_00433 0.66 0.67 0.67 10 1416 7 0 5 2 5
RFA_00434 -0.01 0.00 0.00 0 1417 14 3 11 0 15
RFA_00436 0.97 0.93 1.00 14 1417 3 0 0 3 1
RFA_00440 0.97 0.93 1.00 14 1471 3 0 0 3 1
RFA_00442 0.65 0.53 0.80 8 1421 4 0 2 2 7
RFA_00446 0.97 0.93 1.00 14 1471 3 0 0 3 1
RFA_00449 0.77 0.60 1.00 9 1476 2 0 0 2 6
RFA_00587 1.00 1.00 1.00 15 4836 13 0 0 13 0
RFA_00632 0.40 0.43 0.39 12 4064 19 4 15 0 16
RFA_00636 0.80 0.86 0.75 24 3973 8 6 2 0 4
RFA_00642 -0.01 0.00 0.00 0 2911 15 2 13 0 18
RFA_00643 -0.01 0.00 0.00 0 2196 15 1 14 0 18
RFA_00644 -0.01 0.00 0.00 0 2682 19 4 15 0 18
RFA_00645 0.18 0.17 0.20 3 2400 13 1 11 1 15
RFA_00649 0.48 0.50 0.47 9 2126 11 0 10 1 9
RFA_00651 0.53 0.50 0.56 9 2064 8 0 7 1 9
RFA_00653 0.33 0.33 0.33 6 2127 12 3 9 0 12
RFA_00654 -0.01 0.00 0.00 0 2397 18 7 11 0 18
RFA_00658 0.33 0.29 0.40 4 1118 8 0 6 2 10
RFA_00659 -0.01 0.00 0.00 0 1119 11 0 9 2 14
RFA_00664 -0.01 0.00 0.00 0 980 11 0 10 1 14
RFA_00667 -0.01 0.00 0.00 0 978 13 1 11 1 14
RFA_00668 0.46 0.43 0.50 6 978 7 0 6 1 8
RFA_00672 -0.01 0.00 0.00 0 897 6 0 6 0 13
RFA_00673 -0.01 0.00 0.00 0 1120 9 0 8 1 14
RFA_00674 -0.01 0.00 0.00 0 1116 12 0 12 0 14
RFA_00675 0.29 0.29 0.31 4 977 9 1 8 0 10
RFA_00677 0.61 0.57 0.67 8 978 6 0 4 2 6
RFA_00678 0.33 0.29 0.40 4 936 7 0 6 1 10
RFA_00680 0.71 0.71 0.71 10 1114 7 0 4 3 4
RFA_00684 -0.01 0.00 0.00 0 978 12 0 12 0 14
RFA_00685 -0.01 0.00 0.00 0 979 11 0 11 0 14
RFA_00703 0.39 0.43 0.35 6 4261 24 3 8 13 8
RFA_00704 -0.01 0.00 0.00 0 980 10 0 10 0 14
RFA_00705 0.74 0.71 0.77 10 1022 4 0 3 1 4
RFA_00706 -0.01 0.00 0.00 0 1026 9 1 8 0 14
RFA_00707 -0.01 0.00 0.00 0 1026 9 1 8 0 14
RFA_00708 0.35 0.29 0.44 4 1026 7 0 5 2 10
RFA_00709 0.23 0.21 0.27 3 979 9 0 8 1 11
RFA_00710 -0.01 0.00 0.00 0 979 11 0 11 0 14
RFA_00711 -0.01 0.00 0.00 0 1026 9 0 9 0 14
RFA_00715 -0.01 0.00 0.00 0 933 13 0 13 0 14
RFA_00716 -0.01 0.00 0.00 0 936 10 2 8 0 14
RFA_00717 0.48 0.43 0.55 6 892 5 0 5 0 8
RFA_00730 0.28 0.25 0.33 3 894 6 1 5 0 9
RFA_00731 0.41 0.42 0.42 5 891 8 2 5 1 7
RFA_00733 0.43 0.42 0.45 5 892 7 2 4 1 7
RFA_00734 0.16 0.17 0.18 2 892 10 0 9 1 10
RFA_00736 0.41 0.42 0.42 5 891 8 2 5 1 7
RFA_00737 0.60 0.58 0.64 7 892 5 0 4 1 5
RFA_00745 0.43 0.42 0.45 5 935 7 2 4 1 7
RFA_00749 0.41 0.42 0.42 5 891 8 2 5 1 7
RFA_00758 -0.01 0.00 0.00 0 893 10 0 10 0 12
RFA_00762 0.39 0.42 0.38 5 890 8 2 6 0 7
RFA_00763 0.41 0.42 0.42 5 891 7 2 5 0 7
RFA_00764 0.43 0.42 0.45 5 892 7 2 4 1 7
RFA_00765 0.60 0.58 0.64 7 892 5 0 4 1 5
RFA_00767 1.00 1.00 1.00 18 1873 4 0 0 4 0
RFA_00768 1.00 1.00 1.00 18 1873 3 0 0 3 0
RFA_00769 0.54 0.56 0.53 10 1934 9 4 5 0 8
RFA_00770 0.97 0.94 1.00 17 1999 3 0 0 3 1
RFA_00773 0.97 1.00 0.95 18 1934 4 1 0 3 0
RFA_00779 0.83 0.83 0.83 15 1935 4 1 2 1 3
RFA_00808 1.00 1.00 1.00 16 2000 0 0 0 0 0
RFA_00809 0.79 0.81 0.76 13 2128 4 0 4 0 3
SPR_00020 0.76 0.80 0.73 16 2679 8 2 4 2 4
SPR_00137 0.84 0.86 0.82 18 2904 9 0 4 5 3
SPR_00273 0.31 0.33 0.29 7 2826 17 4 13 0 14
SPR_00394 0.51 0.52 0.50 11 3633 15 3 8 4 10
SPR_00402 0.73 0.76 0.70 16 2533 8 0 7 1 5
SPR_00721 1.00 1.00 1.00 21 2905 1 0 0 1 0
SPR_00816 1.00 1.00 1.00 20 3466 4 0 0 4 0
SRP_00134 0.37 0.41 0.34 12 5960 26 9 14 3 17
SRP_00137 0.37 0.40 0.36 10 4158 18 8 10 0 15
SRP_00285 0.44 0.43 0.45 13 3712 16 2 14 0 17
SRP_00341 0.10 0.10 0.11 3 4726 24 2 22 0 28

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.