CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of RNASampler(seed) - scored higher in this pairwise comparison

  4. Performance of CRWrnafold - scored lower in this pairwise comparison

  5. Compile and download dataset for RNASampler(seed) & CRWrnafold [.zip] - may take several seconds...


Overview

Metric RNASampler(seed) CRWrnafold
MCC 0.629 > 0.473
Average MCC ± 95% Confidence Intervals 0.470 ± 0.066 > 0.463 ± 0.061
Sensitivity 0.468 > 0.465
Positive Predictive Value 0.847 > 0.485
Total TP 1072 > 1065
Total TN 696315 > 695384
Total FP 320 < 1470
Total FP CONTRA 57 < 235
Total FP INCONS 137 < 897
Total FP COMP 126 < 338
Total FN 1217 < 1224
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of RNASampler(seed) and CRWrnafold. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for RNASampler(seed) and CRWrnafold).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for RNASampler(seed) and CRWrnafold).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for RNASampler(seed) and CRWrnafold. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for RNASampler(seed) and CRWrnafold).

^top





Performance of RNASampler(seed) - scored higher in this pairwise comparison

1. Total counts & total scores for RNASampler(seed)

Total Base Pair Counts
Total TP 1072
Total TN 696315
Total FP 320
Total FP CONTRA 57
Total FP INCONS 137
Total FP COMP 126
Total FN 1217
Total Scores
MCC 0.629
Average MCC ± 95% Confidence Intervals 0.470 ± 0.066
Sensitivity 0.468
Positive Predictive Value 0.847
Nr of predictions 98

^top



2. Individual counts for RNASampler(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
PDB_00005 0.88 0.79 1.00 11 935 0 0 0 0 3
PDB_00012 1.00 1.00 1.00 7 399 3 0 0 3 0
PDB_00213 0.76 0.59 1.00 24 5026 0 0 0 0 17
PDB_00553 1.00 1.00 1.00 11 454 0 0 0 0 0
PDB_00716 0.50 0.39 0.64 9 2687 6 0 5 1 14
PDB_00810 0.42 0.18 1.00 3 1078 0 0 0 0 14
PDB_01050 0.92 0.85 1.00 11 619 2 0 0 2 2
PDB_01051 0.45 0.31 0.67 4 897 6 0 2 4 9
PDB_01092 0.72 0.62 0.84 32 10115 8 0 6 2 20
PDB_01152 0.84 0.71 1.00 10 551 0 0 0 0 4
RFA_00390 0.00 0.00 0.00 0 1431 0 0 0 0 15
RFA_00416 0.63 0.40 1.00 6 1479 1 0 0 1 9
RFA_00433 0.63 0.40 1.00 6 1425 1 0 0 1 9
RFA_00587 1.00 1.00 1.00 15 4836 5 0 0 5 0
RFA_00603 0.63 0.51 0.78 18 13507 8 3 2 3 17
RFA_00604 0.73 0.66 0.82 23 13502 14 4 1 9 12
RFA_00605 0.41 0.31 0.55 11 15205 11 1 8 2 24
RFA_00606 0.44 0.36 0.54 14 21295 18 7 5 6 25
RFA_00607 0.71 0.56 0.91 20 17744 9 0 2 7 16
RFA_00609 0.82 0.68 1.00 25 17930 6 0 0 6 12
RFA_00610 0.81 0.70 0.93 26 17177 9 0 2 7 11
RFA_00611 0.68 0.57 0.80 20 13016 8 3 2 3 15
RFA_00613 0.58 0.46 0.73 16 12858 6 4 2 0 19
RFA_00615 0.63 0.51 0.78 18 13343 5 3 2 0 17
RFA_00620 0.52 0.44 0.63 17 21918 15 2 8 5 22
RFA_00626 0.80 0.64 1.00 56 56560 5 0 0 5 31
RFA_00627 0.77 0.63 0.93 55 56894 8 0 4 4 32
RFA_00628 0.78 0.64 0.95 55 57233 12 0 3 9 31
RFA_00630 0.77 0.66 0.90 57 56890 15 0 6 9 30
RFA_00632 0.78 0.68 0.90 19 4074 2 1 1 0 9
RFA_00636 0.85 0.75 0.95 21 3983 1 1 0 0 7
RFA_00642 0.58 0.39 0.88 7 2918 1 0 1 0 11
RFA_00643 0.21 0.17 0.27 3 2200 8 0 8 0 15
RFA_00644 0.44 0.33 0.60 6 2691 4 1 3 0 12
RFA_00645 -0.01 0.00 0.00 0 2403 12 3 9 0 18
RFA_00649 0.44 0.33 0.60 6 2135 4 1 3 0 12
RFA_00651 0.33 0.22 0.50 4 2072 4 1 3 0 14
RFA_00653 -0.01 0.00 0.00 0 2133 12 3 9 0 18
RFA_00654 0.39 0.28 0.56 5 2406 4 1 3 0 13
RFA_00658 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00659 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00664 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00667 0.53 0.29 1.00 4 986 0 0 0 0 10
RFA_00668 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00672 0.00 0.00 0.00 0 903 0 0 0 0 13
RFA_00673 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00674 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00675 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00677 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00678 0.00 0.00 0.00 0 946 0 0 0 0 14
RFA_00680 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00684 0.53 0.29 1.00 4 986 0 0 0 0 10
RFA_00685 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00695 0.00 0.00 0.00 0 7021 0 0 0 0 14
RFA_00703 0.46 0.21 1.00 3 4275 0 0 0 0 11
RFA_00704 -0.01 0.00 0.00 0 987 3 0 3 0 14
RFA_00705 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00706 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00707 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00708 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00709 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00710 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00711 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00715 0.00 0.00 0.00 0 946 0 0 0 0 14
RFA_00716 -0.01 0.00 0.00 0 943 3 0 3 0 14
RFA_00717 0.00 0.00 0.00 0 903 0 0 0 0 14
RFA_00730 0.50 0.25 1.00 3 900 0 0 0 0 9
RFA_00731 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00733 0.50 0.25 1.00 3 900 0 0 0 0 9
RFA_00734 0.64 0.42 1.00 5 898 0 0 0 0 7
RFA_00736 0.50 0.25 1.00 3 900 0 0 0 0 9
RFA_00737 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00745 0.50 0.25 1.00 3 943 0 0 0 0 9
RFA_00749 0.50 0.25 1.00 3 900 0 0 0 0 9
RFA_00758 0.50 0.25 1.00 3 900 0 0 0 0 9
RFA_00762 0.64 0.42 1.00 5 898 0 0 0 0 7
RFA_00763 0.50 0.25 1.00 3 900 0 0 0 0 9
RFA_00764 0.50 0.25 1.00 3 900 0 0 0 0 9
RFA_00765 0.50 0.25 1.00 3 900 0 0 0 0 9
RFA_00767 0.65 0.56 0.77 10 1878 3 3 0 0 8
RFA_00768 0.65 0.56 0.77 10 1878 3 3 0 0 8
RFA_00769 0.59 0.56 0.63 10 1937 6 1 5 0 8
RFA_00770 0.52 0.39 0.70 7 2006 3 3 0 0 11
RFA_00773 0.59 0.50 0.69 9 1940 4 4 0 0 9
RFA_00779 0.65 0.56 0.77 10 1940 3 3 0 0 8
RFA_00781 0.85 0.72 1.00 23 5027 0 0 0 0 9
RFA_00786 0.79 0.63 1.00 20 5030 0 0 0 0 12
RFA_00791 0.71 0.56 0.90 18 5131 2 0 2 0 14
RFA_00792 0.81 0.72 0.92 23 5025 2 0 2 0 9
RFA_00801 0.74 0.63 0.87 20 5027 4 0 3 1 12
RFA_00808 0.58 0.56 0.60 9 2001 6 0 6 0 7
RFA_00809 0.36 0.38 0.35 6 2128 11 1 10 0 10
RFA_00814 0.87 0.78 0.97 32 25167 6 0 1 5 9
RFA_00815 0.88 0.78 1.00 32 24499 8 0 0 8 9
RFA_00816 0.86 0.76 0.97 31 23188 6 0 1 5 10
RFA_00817 0.86 0.76 0.97 31 21913 1 0 1 0 10
RFA_00818 0.86 0.73 1.00 30 20271 6 0 0 6 11
RFA_00819 0.88 0.78 1.00 32 27934 5 0 0 5 9

^top



Performance of CRWrnafold - scored lower in this pairwise comparison

1. Total counts & total scores for CRWrnafold

Total Base Pair Counts
Total TP 1065
Total TN 695384
Total FP 1470
Total FP CONTRA 235
Total FP INCONS 897
Total FP COMP 338
Total FN 1224
Total Scores
MCC 0.473
Average MCC ± 95% Confidence Intervals 0.463 ± 0.061
Sensitivity 0.465
Positive Predictive Value 0.485
Nr of predictions 98

^top



2. Individual counts for CRWrnafold [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
PDB_00005 0.88 0.79 1.00 11 935 0 0 0 0 3
PDB_00012 1.00 1.00 1.00 7 399 3 0 0 3 0
PDB_00213 0.92 0.85 1.00 35 5015 0 0 0 0 6
PDB_00553 1.00 1.00 1.00 11 454 0 0 0 0 0
PDB_00716 -0.01 0.00 0.00 0 2680 22 0 21 1 23
PDB_00810 0.67 0.59 0.77 10 1068 3 0 3 0 7
PDB_01050 0.92 0.85 1.00 11 619 1 0 0 1 2
PDB_01051 0.79 0.69 0.90 9 893 5 0 1 4 4
PDB_01092 0.68 0.62 0.76 32 10111 12 1 9 2 20
PDB_01152 0.88 0.79 1.00 11 550 0 0 0 0 3
RFA_00390 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00416 1.00 1.00 1.00 15 1470 3 0 0 3 0
RFA_00433 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00587 1.00 1.00 1.00 15 4836 12 0 0 12 0
RFA_00603 0.27 0.29 0.26 10 13491 31 8 21 2 25
RFA_00604 0.61 0.63 0.59 22 13493 26 2 13 11 13
RFA_00605 0.34 0.34 0.34 12 15190 36 4 19 13 23
RFA_00606 0.41 0.46 0.37 18 21272 38 17 14 7 21
RFA_00607 0.29 0.31 0.28 11 17726 39 5 24 10 25
RFA_00609 0.52 0.57 0.48 21 17911 36 8 15 13 16
RFA_00610 0.52 0.54 0.50 20 17165 33 4 16 13 17
RFA_00611 0.45 0.49 0.43 17 13001 27 8 15 4 18
RFA_00613 0.43 0.43 0.44 15 12846 26 8 11 7 20
RFA_00615 0.27 0.29 0.26 10 13328 29 7 21 1 25
RFA_00620 0.55 0.59 0.51 23 21900 40 4 18 18 16
RFA_00626 0.64 0.63 0.65 55 56532 42 8 21 13 32
RFA_00627 0.65 0.63 0.67 55 56871 42 6 21 15 32
RFA_00628 0.89 0.85 0.94 73 57213 30 0 5 25 13
RFA_00630 0.57 0.57 0.57 50 56865 51 12 26 13 37
RFA_00632 0.37 0.36 0.38 10 4069 19 0 16 3 18
RFA_00636 0.51 0.50 0.52 14 3978 13 3 10 0 14
RFA_00642 0.17 0.17 0.18 3 2909 17 0 14 3 15
RFA_00643 0.24 0.22 0.27 4 2196 12 2 9 1 14
RFA_00644 0.21 0.22 0.21 4 2682 15 4 11 0 14
RFA_00645 -0.01 0.00 0.00 0 2401 14 2 12 0 18
RFA_00649 0.57 0.56 0.59 10 2128 7 1 6 0 8
RFA_00651 0.35 0.33 0.38 6 2064 10 1 9 0 12
RFA_00653 0.35 0.33 0.38 6 2129 10 2 8 0 12
RFA_00654 0.26 0.28 0.26 5 2396 14 2 12 0 13
RFA_00658 0.59 0.50 0.70 7 1118 4 0 3 1 7
RFA_00659 0.18 0.14 0.25 2 1120 8 0 6 2 12
RFA_00664 -0.01 0.00 0.00 0 980 11 0 10 1 14
RFA_00667 0.30 0.29 0.33 4 978 9 1 7 1 10
RFA_00668 0.46 0.43 0.50 6 978 6 0 6 0 8
RFA_00672 -0.01 0.00 0.00 0 897 6 0 6 0 13
RFA_00673 0.47 0.36 0.63 5 1120 4 0 3 1 9
RFA_00674 -0.01 0.00 0.00 0 1118 10 0 10 0 14
RFA_00675 0.29 0.29 0.31 4 977 9 1 8 0 10
RFA_00677 0.69 0.64 0.75 9 978 5 0 3 2 5
RFA_00678 -0.01 0.00 0.00 0 941 5 0 5 0 14
RFA_00680 0.74 0.71 0.77 10 1115 6 0 3 3 4
RFA_00684 0.72 0.64 0.82 9 979 3 0 2 1 5
RFA_00685 0.42 0.36 0.50 5 980 6 0 5 1 9
RFA_00695 0.67 0.64 0.69 9 7008 28 1 3 24 5
RFA_00703 0.67 0.64 0.69 9 4265 18 1 3 14 5
RFA_00704 -0.01 0.00 0.00 0 981 10 0 9 1 14
RFA_00705 0.74 0.71 0.77 10 1022 4 0 3 1 4
RFA_00706 0.33 0.29 0.40 4 1025 6 0 6 0 10
RFA_00707 -0.01 0.00 0.00 0 1024 11 3 8 0 14
RFA_00708 0.35 0.29 0.44 4 1026 6 0 5 1 10
RFA_00709 -0.01 0.00 0.00 0 982 9 0 8 1 14
RFA_00710 -0.01 0.00 0.00 0 980 10 0 10 0 14
RFA_00711 -0.01 0.00 0.00 0 1024 11 0 11 0 14
RFA_00715 -0.01 0.00 0.00 0 936 10 0 10 0 14
RFA_00716 -0.01 0.00 0.00 0 939 7 0 7 0 14
RFA_00717 0.31 0.29 0.36 4 892 8 0 7 1 10
RFA_00730 0.69 0.67 0.73 8 892 4 1 2 1 4
RFA_00731 0.41 0.42 0.42 5 891 8 3 4 1 7
RFA_00733 0.69 0.67 0.73 8 892 4 1 2 1 4
RFA_00734 0.64 0.67 0.62 8 890 5 1 4 0 4
RFA_00736 0.41 0.42 0.42 5 891 8 3 4 1 7
RFA_00737 0.81 0.67 1.00 8 895 1 0 0 1 4
RFA_00745 0.69 0.67 0.73 8 935 4 1 2 1 4
RFA_00749 0.41 0.42 0.42 5 891 8 3 4 1 7
RFA_00758 0.63 0.58 0.70 7 893 3 1 2 0 5
RFA_00762 0.64 0.67 0.62 8 890 5 1 4 0 4
RFA_00763 0.69 0.67 0.73 8 892 3 1 2 0 4
RFA_00764 0.69 0.67 0.73 8 892 4 1 2 1 4
RFA_00765 0.81 0.67 1.00 8 895 1 0 0 1 4
RFA_00767 0.59 0.56 0.63 10 1875 6 3 3 0 8
RFA_00768 0.47 0.44 0.50 8 1875 8 1 7 0 10
RFA_00769 0.54 0.56 0.53 10 1934 9 4 5 0 8
RFA_00770 -0.01 0.00 0.00 0 2002 17 1 13 3 18
RFA_00773 0.59 0.56 0.63 10 1937 6 1 5 0 8
RFA_00779 0.59 0.56 0.63 10 1937 6 3 3 0 8
RFA_00781 0.74 0.75 0.73 24 5017 9 2 7 0 8
RFA_00786 0.17 0.16 0.19 5 5024 21 1 20 0 27
RFA_00791 0.46 0.44 0.48 14 5122 15 0 15 0 18
RFA_00792 0.94 0.94 0.94 30 5018 4 0 2 2 2
RFA_00801 0.38 0.34 0.42 11 5024 16 1 14 1 21
RFA_00808 -0.01 0.00 0.00 0 2000 18 2 14 2 16
RFA_00809 0.40 0.38 0.43 6 2131 8 1 7 0 10
RFA_00814 0.24 0.27 0.21 11 25148 48 10 31 7 30
RFA_00815 0.36 0.41 0.31 17 24477 49 6 31 12 24
RFA_00816 0.21 0.24 0.19 10 23166 50 20 24 6 31
RFA_00817 0.10 0.12 0.08 5 21883 57 17 40 0 36
RFA_00818 0.00 0.00 0.00 0 20249 56 19 33 4 41
RFA_00819 0.44 0.44 0.44 18 27925 57 0 23 34 23

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.