CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of RNAfold - scored higher in this pairwise comparison

  4. Performance of CMfinder(seed) - scored lower in this pairwise comparison

  5. Compile and download dataset for RNAfold & CMfinder(seed) [.zip] - may take several seconds...


Overview

Metric RNAfold CMfinder(seed)
MCC 0.592 > 0.486
Average MCC ± 95% Confidence Intervals 0.545 ± 0.048 > 0.505 ± 0.044
Sensitivity 0.589 > 0.312
Positive Predictive Value 0.597 < 0.757
Total TP 3590 > 1905
Total TN 2877740 < 2881233
Total FP 2978 > 688
Total FP CONTRA 442 > 40
Total FP INCONS 1978 > 572
Total FP COMP 558 > 76
Total FN 2507 < 4192
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of RNAfold and CMfinder(seed). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for RNAfold and CMfinder(seed)).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for RNAfold and CMfinder(seed)).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for RNAfold and CMfinder(seed). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for RNAfold and CMfinder(seed)).

^top





Performance of RNAfold - scored higher in this pairwise comparison

1. Total counts & total scores for RNAfold

Total Base Pair Counts
Total TP 3590
Total TN 2877740
Total FP 2978
Total FP CONTRA 442
Total FP INCONS 1978
Total FP COMP 558
Total FN 2507
Total Scores
MCC 0.592
Average MCC ± 95% Confidence Intervals 0.545 ± 0.048
Sensitivity 0.589
Positive Predictive Value 0.597
Nr of predictions 129

^top



2. Individual counts for RNAfold [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_00013 0.21 0.22 0.21 25 103623 116 12 80 24 90
CRW_00016 0.64 0.62 0.67 74 77311 47 5 31 11 46
CRW_00610 0.21 0.21 0.22 17 36239 59 7 52 0 64
CRW_00613 0.65 0.60 0.70 47 34913 28 1 19 8 31
CRW_00614 0.16 0.19 0.14 11 121692 131 25 43 63 46
CRW_00618 0.12 0.13 0.12 8 56212 82 13 47 22 53
CRW_00633 0.44 0.43 0.45 46 63444 58 13 43 2 60
CRW_00634 0.50 0.52 0.49 49 64519 57 10 42 5 45
CRW_00670 0.69 0.68 0.71 82 70009 36 9 25 2 38
CRW_00671 0.60 0.59 0.61 69 62722 49 6 38 5 48
CRW_00672 0.48 0.50 0.47 55 72273 65 19 43 3 56
CRW_00674 0.53 0.53 0.52 66 83310 62 12 48 2 58
CRW_00676 0.54 0.56 0.52 64 93406 74 17 41 16 51
CRW_00692 0.40 0.42 0.38 38 68536 66 24 37 5 52
PDB_00005 0.88 0.79 1.00 11 935 0 0 0 0 3
PDB_00012 1.00 1.00 1.00 7 399 3 0 0 3 0
PDB_00213 0.92 0.85 1.00 35 5015 0 0 0 0 6
PDB_00553 1.00 1.00 1.00 11 454 0 0 0 0 0
PDB_00716 -0.01 0.00 0.00 0 2678 24 1 22 1 23
PDB_00810 0.71 0.65 0.79 11 1067 4 0 3 1 6
PDB_00827 0.50 0.46 0.55 37 26961 31 1 29 1 44
PDB_01050 0.96 0.92 1.00 12 618 2 0 0 2 1
PDB_01051 0.75 0.69 0.82 9 892 6 0 2 4 4
PDB_01092 0.71 0.63 0.80 33 10112 10 1 7 2 19
PDB_01152 0.96 0.93 1.00 13 548 0 0 0 0 1
RFA_00603 0.64 0.69 0.60 24 13490 19 4 12 3 11
RFA_00604 0.69 0.71 0.68 25 13493 24 2 10 12 10
RFA_00605 0.37 0.34 0.40 12 15195 27 3 15 9 23
RFA_00606 0.67 0.67 0.68 26 21283 25 7 5 13 13
RFA_00607 0.83 0.83 0.83 30 17730 18 4 2 12 6
RFA_00609 0.75 0.76 0.74 28 17917 28 2 8 18 9
RFA_00610 0.90 0.86 0.94 32 17171 18 0 2 16 5
RFA_00611 0.63 0.63 0.63 22 13006 15 3 10 2 13
RFA_00613 0.61 0.60 0.62 21 12846 15 4 9 2 14
RFA_00615 0.61 0.63 0.59 22 13329 19 5 10 4 13
RFA_00620 0.64 0.62 0.67 24 21909 25 3 9 13 15
RFA_00626 0.96 0.92 1.00 80 56536 25 0 0 25 7
RFA_00627 0.72 0.69 0.75 60 56873 35 4 16 15 27
RFA_00628 0.93 0.91 0.96 78 57210 29 0 3 26 8
RFA_00630 0.66 0.66 0.66 57 56867 40 6 23 11 30
RFA_00632 0.38 0.39 0.37 11 4065 19 2 17 0 17
RFA_00636 0.42 0.43 0.43 12 3977 16 2 14 0 16
RFA_00639 0.17 0.18 0.16 16 54515 86 19 65 2 71
RFA_00642 0.25 0.22 0.29 4 2912 10 2 8 0 14
RFA_00643 0.24 0.22 0.27 4 2196 12 2 9 1 14
RFA_00644 -0.01 0.00 0.00 0 2681 20 4 16 0 18
RFA_00645 -0.01 0.00 0.00 0 2401 14 3 11 0 18
RFA_00649 0.34 0.33 0.35 6 2128 13 2 9 2 12
RFA_00651 0.35 0.33 0.38 6 2064 10 1 9 0 12
RFA_00653 0.34 0.33 0.35 6 2128 11 3 8 0 12
RFA_00654 0.27 0.28 0.28 5 2397 13 2 11 0 13
RFA_00658 0.59 0.50 0.70 7 1118 5 0 3 2 7
RFA_00659 -0.01 0.00 0.00 0 1120 10 1 7 2 14
RFA_00664 -0.01 0.00 0.00 0 980 11 0 10 1 14
RFA_00667 0.29 0.29 0.31 4 977 10 1 8 1 10
RFA_00668 0.41 0.43 0.40 6 975 9 0 9 0 8
RFA_00672 -0.01 0.00 0.00 0 895 9 0 8 1 13
RFA_00673 0.42 0.36 0.50 5 1118 6 0 5 1 9
RFA_00674 0.71 0.71 0.71 10 1114 5 0 4 1 4
RFA_00675 0.29 0.29 0.31 4 977 9 1 8 0 10
RFA_00677 0.42 0.43 0.43 6 976 9 0 8 1 8
RFA_00678 0.43 0.29 0.67 4 940 2 0 2 0 10
RFA_00680 0.74 0.71 0.77 10 1115 6 0 3 3 4
RFA_00684 0.66 0.64 0.69 9 977 5 0 4 1 5
RFA_00685 0.42 0.36 0.50 5 980 6 0 5 1 9
RFA_00704 -0.01 0.00 0.00 0 981 9 0 9 0 14
RFA_00705 0.74 0.71 0.77 10 1022 4 0 3 1 4
RFA_00706 0.37 0.29 0.50 4 1027 4 0 4 0 10
RFA_00707 -0.01 0.00 0.00 0 1026 9 3 6 0 14
RFA_00708 0.40 0.29 0.57 4 1028 3 0 3 0 10
RFA_00709 -0.01 0.00 0.00 0 982 9 0 8 1 14
RFA_00710 -0.01 0.00 0.00 0 981 9 0 9 0 14
RFA_00711 0.33 0.29 0.40 4 1025 7 0 6 1 10
RFA_00715 -0.01 0.00 0.00 0 936 10 0 10 0 14
RFA_00716 -0.01 0.00 0.00 0 935 11 0 11 0 14
RFA_00717 0.66 0.64 0.69 9 890 4 0 4 0 5
RFA_00730 0.69 0.67 0.73 8 892 4 1 2 1 4
RFA_00731 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00733 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00734 0.61 0.67 0.57 8 889 7 1 5 1 4
RFA_00736 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00737 0.75 0.75 0.75 9 891 5 0 3 2 3
RFA_00745 0.66 0.67 0.67 8 934 5 1 3 1 4
RFA_00749 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00758 0.96 0.92 1.00 11 892 0 0 0 0 1
RFA_00762 0.64 0.67 0.62 8 890 5 1 4 0 4
RFA_00763 0.75 0.75 0.75 9 891 4 0 3 1 3
RFA_00764 0.66 0.67 0.67 8 891 5 1 3 1 4
RFA_00765 0.75 0.75 0.75 9 891 5 0 3 2 3
RFA_00769 0.52 0.56 0.50 10 1933 10 4 6 0 8
RFA_00770 0.68 0.56 0.83 10 2004 5 0 2 3 8
RFA_00773 0.55 0.56 0.56 10 1935 8 2 6 0 8
RFA_00779 0.55 0.56 0.56 10 1935 8 3 5 0 8
RFA_00781 0.58 0.59 0.58 19 5017 14 2 12 0 13
RFA_00786 0.71 0.72 0.70 23 5017 10 2 8 0 9
RFA_00791 0.50 0.50 0.50 16 5119 16 5 11 0 16
RFA_00792 0.94 0.94 0.94 30 5018 4 0 2 2 2
RFA_00801 0.85 0.81 0.90 26 5021 4 0 3 1 6
RFA_00808 -0.01 0.00 0.00 0 2001 18 2 13 3 16
RFA_00809 0.36 0.38 0.35 6 2128 11 1 10 0 10
RFA_00814 0.79 0.80 0.77 33 25157 26 4 6 16 8
RFA_00815 0.65 0.71 0.60 29 24483 34 7 12 15 12
RFA_00816 0.59 0.66 0.53 27 23169 32 13 11 8 14
RFA_00817 0.10 0.12 0.08 5 21885 55 16 39 0 36
RFA_00818 0.59 0.63 0.55 26 20254 31 10 11 10 15
RFA_00819 0.96 0.95 0.98 39 27926 31 0 1 30 2
SRP_00016 0.86 0.85 0.87 93 47788 16 2 12 2 16
SRP_00130 0.24 0.24 0.24 23 49676 71 10 61 0 74
SRP_00142 0.57 0.59 0.55 55 45956 48 7 38 3 38
SRP_00143 0.86 0.85 0.86 86 49670 20 1 13 6 15
SRP_00144 0.74 0.74 0.74 70 46571 31 1 23 7 24
SRP_00197 0.49 0.48 0.51 48 49361 49 4 42 3 53
SRP_00198 0.62 0.60 0.63 62 50942 40 2 34 4 41
SRP_00199 0.26 0.26 0.26 27 50938 77 4 71 2 78
SRP_00201 0.85 0.84 0.85 93 47786 18 5 11 2 18
SRP_00202 0.85 0.85 0.85 99 54169 21 2 15 4 18
SRP_00203 0.73 0.75 0.72 79 48718 37 3 28 6 26
SRP_00204 0.78 0.77 0.78 79 49040 24 4 18 2 23
SRP_00206 0.82 0.82 0.82 89 47786 21 5 15 1 19
SRP_00208 0.67 0.67 0.67 68 46564 35 7 26 2 33
SRP_00255 0.60 0.62 0.59 58 47487 47 3 38 6 36
SRP_00276 0.86 0.85 0.86 94 50612 16 5 10 1 16
SRP_00278 0.72 0.72 0.72 76 45950 32 4 26 2 29
SRP_00321 0.77 0.77 0.78 83 47479 27 3 21 3 25
SRP_00322 0.62 0.62 0.62 68 48406 43 5 37 1 41
SRP_00323 0.92 0.92 0.93 98 47173 11 0 7 4 9
SRP_00330 0.42 0.42 0.42 40 49991 58 2 53 3 55
SRP_00332 0.80 0.81 0.80 86 50295 25 6 16 3 20
SRP_00339 0.47 0.48 0.46 45 44453 57 6 47 4 49

^top



Performance of CMfinder(seed) - scored lower in this pairwise comparison

1. Total counts & total scores for CMfinder(seed)

Total Base Pair Counts
Total TP 1905
Total TN 2881233
Total FP 688
Total FP CONTRA 40
Total FP INCONS 572
Total FP COMP 76
Total FN 4192
Total Scores
MCC 0.486
Average MCC ± 95% Confidence Intervals 0.505 ± 0.044
Sensitivity 0.312
Positive Predictive Value 0.757
Nr of predictions 129

^top



2. Individual counts for CMfinder(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_00013 0.37 0.14 1.00 16 103724 0 0 0 0 99
CRW_00016 0.36 0.14 0.89 17 77402 2 0 2 0 103
CRW_00610 0.49 0.31 0.78 25 36283 7 1 6 0 56
CRW_00613 0.54 0.33 0.87 26 34950 4 0 4 0 52
CRW_00614 0.46 0.25 0.88 14 121755 9 0 2 7 43
CRW_00618 0.36 0.13 1.00 8 56272 0 0 0 0 53
CRW_00633 0.32 0.12 0.81 13 63530 3 1 2 0 93
CRW_00634 0.40 0.17 0.94 16 64603 1 0 1 0 78
CRW_00670 0.32 0.18 0.55 22 70085 18 0 18 0 98
CRW_00671 0.20 0.09 0.41 11 62808 16 0 16 0 106
CRW_00672 0.38 0.15 0.94 17 72372 1 0 1 0 94
CRW_00674 0.40 0.17 0.95 21 83414 1 0 1 0 103
CRW_00676 0.12 0.04 0.33 5 93513 10 1 9 0 110
CRW_00692 0.41 0.17 1.00 15 68620 0 0 0 0 75
PDB_00005 0.47 0.36 0.63 5 938 3 0 3 0 9
PDB_00012 1.00 1.00 1.00 7 399 2 0 0 2 0
PDB_00213 0.85 0.73 1.00 30 5020 0 0 0 0 11
PDB_00553 0.00 0.00 0.00 0 465 0 0 0 0 11
PDB_00716 -0.01 0.00 0.00 0 2688 13 0 13 0 23
PDB_00810 0.76 0.59 1.00 10 1071 0 0 0 0 7
PDB_00827 0.25 0.10 0.62 8 27015 5 0 5 0 73
PDB_01050 0.68 0.46 1.00 6 624 1 0 0 1 7
PDB_01051 0.83 0.69 1.00 9 894 4 0 0 4 4
PDB_01092 0.38 0.21 0.69 11 10137 6 0 5 1 41
PDB_01152 0.92 0.86 1.00 12 549 0 0 0 0 2
RFA_00603 0.70 0.60 0.81 21 13504 7 0 5 2 14
RFA_00604 0.63 0.51 0.78 18 13507 8 0 5 3 17
RFA_00605 0.14 0.06 0.33 2 15219 5 0 4 1 33
RFA_00606 0.34 0.21 0.57 8 21307 8 2 4 2 31
RFA_00607 0.08 0.06 0.13 2 17750 15 3 11 1 34
RFA_00609 0.32 0.22 0.47 8 17938 12 2 7 3 29
RFA_00610 0.39 0.30 0.52 11 17184 13 2 8 3 26
RFA_00611 0.65 0.51 0.82 18 13019 5 0 4 1 17
RFA_00613 0.70 0.57 0.87 20 12857 4 0 3 1 15
RFA_00615 0.57 0.49 0.68 17 13341 10 0 8 2 18
RFA_00620 0.00 0.00 0.00 0 21939 6 0 6 0 39
RFA_00626 0.44 0.32 0.60 28 56569 22 2 17 3 59
RFA_00627 0.53 0.39 0.72 34 56906 15 3 10 2 53
RFA_00628 0.58 0.43 0.77 37 57243 14 3 8 3 49
RFA_00630 0.45 0.32 0.64 28 56909 17 3 13 1 59
RFA_00632 0.27 0.25 0.29 7 4071 17 0 17 0 21
RFA_00636 0.71 0.64 0.78 18 3982 5 2 3 0 10
RFA_00639 0.00 0.00 0.00 0 54575 44 6 34 4 87
RFA_00642 0.44 0.28 0.71 5 2919 2 0 2 0 13
RFA_00643 -0.01 0.00 0.00 0 2203 8 0 8 0 18
RFA_00644 0.00 0.00 0.00 0 2693 8 0 8 0 18
RFA_00645 0.29 0.17 0.50 3 2409 3 0 3 0 15
RFA_00649 0.35 0.22 0.57 4 2138 3 0 3 0 14
RFA_00651 0.35 0.22 0.57 4 2073 3 0 3 0 14
RFA_00653 -0.01 0.00 0.00 0 2137 8 0 8 0 18
RFA_00654 0.38 0.22 0.67 4 2409 2 0 2 0 14
RFA_00658 0.40 0.29 0.57 4 1121 4 0 3 1 10
RFA_00659 0.23 0.14 0.40 2 1123 4 0 3 1 12
RFA_00664 0.40 0.29 0.57 4 983 4 0 3 1 10
RFA_00667 0.40 0.29 0.57 4 983 4 0 3 1 10
RFA_00668 0.35 0.21 0.60 3 985 3 0 2 1 11
RFA_00672 0.41 0.31 0.57 4 896 3 0 3 0 9
RFA_00673 0.23 0.14 0.40 2 1123 4 0 3 1 12
RFA_00674 0.40 0.29 0.57 4 1121 4 0 3 1 10
RFA_00675 0.35 0.21 0.60 3 985 3 0 2 1 11
RFA_00677 0.40 0.29 0.57 4 983 4 0 3 1 10
RFA_00678 0.43 0.29 0.67 4 940 3 0 2 1 10
RFA_00680 0.40 0.29 0.57 4 1121 4 0 3 1 10
RFA_00684 0.43 0.29 0.67 4 984 3 0 2 1 10
RFA_00685 0.32 0.21 0.50 3 984 4 0 3 1 11
RFA_00704 0.32 0.21 0.50 3 984 4 0 3 1 11
RFA_00705 0.40 0.29 0.57 4 1028 4 0 3 1 10
RFA_00706 0.43 0.29 0.67 4 1029 3 0 2 1 10
RFA_00707 0.43 0.29 0.67 4 1029 3 0 2 1 10
RFA_00708 0.40 0.29 0.57 4 1028 4 0 3 1 10
RFA_00709 0.32 0.21 0.50 3 984 4 0 3 1 11
RFA_00710 0.32 0.21 0.50 3 984 3 0 3 0 11
RFA_00711 0.43 0.29 0.67 4 1029 3 0 2 1 10
RFA_00715 0.30 0.14 0.67 2 943 2 0 1 1 12
RFA_00716 0.43 0.29 0.67 4 940 3 0 2 1 10
RFA_00717 0.30 0.21 0.43 3 896 4 0 4 0 11
RFA_00730 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00731 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00733 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00734 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00736 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00737 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00745 0.91 0.83 1.00 10 936 0 0 0 0 2
RFA_00749 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00758 0.86 0.75 1.00 9 894 0 0 0 0 3
RFA_00762 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00763 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00764 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00765 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00769 0.71 0.50 1.00 9 1944 0 0 0 0 9
RFA_00770 0.71 0.50 1.00 9 2007 0 0 0 0 9
RFA_00773 0.00 0.00 0.00 0 1949 4 0 4 0 18
RFA_00779 0.71 0.50 1.00 9 1944 0 0 0 0 9
RFA_00781 0.97 0.94 1.00 30 5020 0 0 0 0 2
RFA_00786 0.95 0.91 1.00 29 5021 0 0 0 0 3
RFA_00791 0.92 0.88 0.97 28 5122 1 0 1 0 4
RFA_00792 0.97 0.94 1.00 30 5020 0 0 0 0 2
RFA_00801 0.95 0.91 1.00 29 5021 0 0 0 0 3
RFA_00808 0.68 0.56 0.82 9 2005 2 0 2 0 7
RFA_00809 0.41 0.38 0.46 6 2132 7 1 6 0 10
RFA_00814 0.68 0.51 0.91 21 25177 2 0 2 0 20
RFA_00815 0.68 0.51 0.91 21 24508 2 0 2 0 20
RFA_00816 0.68 0.51 0.91 21 23197 2 0 2 0 20
RFA_00817 0.68 0.56 0.82 23 21917 5 0 5 0 18
RFA_00818 0.60 0.49 0.74 20 20274 8 1 6 1 21
RFA_00819 0.62 0.51 0.75 21 27938 7 0 7 0 20
SRP_00016 0.54 0.35 0.83 38 47849 8 0 8 0 71
SRP_00130 0.50 0.32 0.78 31 49730 10 1 8 1 66
SRP_00142 0.46 0.37 0.58 34 45997 26 1 24 1 59
SRP_00143 0.50 0.30 0.86 30 49735 5 0 5 0 71
SRP_00144 0.54 0.37 0.80 35 46621 11 0 9 2 59
SRP_00197 0.46 0.26 0.84 26 49424 5 0 5 0 75
SRP_00198 0.56 0.37 0.84 38 50995 7 0 7 0 65
SRP_00199 0.54 0.35 0.82 37 50995 8 0 8 0 68
SRP_00201 0.54 0.35 0.83 39 47848 8 0 8 0 72
SRP_00202 0.43 0.21 0.86 25 54256 4 0 4 0 92
SRP_00203 0.51 0.34 0.77 36 48781 12 0 11 1 69
SRP_00204 0.52 0.34 0.80 35 49097 9 1 8 0 67
SRP_00206 0.55 0.36 0.83 39 47848 8 0 8 0 69
SRP_00208 0.52 0.33 0.83 33 46625 7 1 6 0 68
SRP_00255 0.49 0.27 0.89 25 47558 3 0 3 0 69
SRP_00276 0.51 0.33 0.80 36 50676 9 0 9 0 74
SRP_00278 0.52 0.33 0.81 35 46013 8 0 8 0 70
SRP_00321 0.42 0.21 0.82 23 47558 5 1 4 0 85
SRP_00322 0.52 0.33 0.82 36 48472 8 0 8 0 73
SRP_00323 0.54 0.34 0.88 36 47237 5 0 5 0 71
SRP_00330 0.44 0.23 0.85 22 50060 5 0 4 1 73
SRP_00332 0.44 0.25 0.75 27 50367 9 0 9 0 79
SRP_00339 0.56 0.39 0.79 37 44504 10 2 8 0 57

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.