CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of CMfinder(seed) - scored higher in this pairwise comparison

  4. Performance of Vsfold5 - scored lower in this pairwise comparison

  5. Compile and download dataset for CMfinder(seed) & Vsfold5 [.zip] - may take several seconds...


Overview

Metric CMfinder(seed) Vsfold5
MCC 0.488 > 0.285
Average MCC ± 95% Confidence Intervals 0.506 ± 0.044 > 0.319 ± 0.045
Sensitivity 0.316 > 0.272
Positive Predictive Value 0.755 > 0.301
Total TP 1875 > 1611
Total TN 2655754 > 2652889
Total FP 679 < 4005
Total FP CONTRA 40 < 537
Total FP INCONS 570 < 3202
Total FP COMP 69 < 266
Total FN 4050 < 4314
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of CMfinder(seed) and Vsfold5. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for CMfinder(seed) and Vsfold5).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for CMfinder(seed) and Vsfold5).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for CMfinder(seed) and Vsfold5. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for CMfinder(seed) and Vsfold5).

^top





Performance of CMfinder(seed) - scored higher in this pairwise comparison

1. Total counts & total scores for CMfinder(seed)

Total Base Pair Counts
Total TP 1875
Total TN 2655754
Total FP 679
Total FP CONTRA 40
Total FP INCONS 570
Total FP COMP 69
Total FN 4050
Total Scores
MCC 0.488
Average MCC ± 95% Confidence Intervals 0.506 ± 0.044
Sensitivity 0.316
Positive Predictive Value 0.755
Nr of predictions 127

^top



2. Individual counts for CMfinder(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_00016 0.36 0.14 0.89 17 77402 2 0 2 0 103
CRW_00610 0.49 0.31 0.78 25 36283 7 1 6 0 56
CRW_00613 0.54 0.33 0.87 26 34950 4 0 4 0 52
CRW_00618 0.36 0.13 1.00 8 56272 0 0 0 0 53
CRW_00633 0.32 0.12 0.81 13 63530 3 1 2 0 93
CRW_00634 0.40 0.17 0.94 16 64603 1 0 1 0 78
CRW_00670 0.32 0.18 0.55 22 70085 18 0 18 0 98
CRW_00671 0.20 0.09 0.41 11 62808 16 0 16 0 106
CRW_00672 0.38 0.15 0.94 17 72372 1 0 1 0 94
CRW_00674 0.40 0.17 0.95 21 83414 1 0 1 0 103
CRW_00676 0.12 0.04 0.33 5 93513 10 1 9 0 110
CRW_00692 0.41 0.17 1.00 15 68620 0 0 0 0 75
PDB_00005 0.47 0.36 0.63 5 938 3 0 3 0 9
PDB_00012 1.00 1.00 1.00 7 399 2 0 0 2 0
PDB_00213 0.85 0.73 1.00 30 5020 0 0 0 0 11
PDB_00553 0.00 0.00 0.00 0 465 0 0 0 0 11
PDB_00716 -0.01 0.00 0.00 0 2688 13 0 13 0 23
PDB_00810 0.76 0.59 1.00 10 1071 0 0 0 0 7
PDB_00827 0.25 0.10 0.62 8 27015 5 0 5 0 73
PDB_01050 0.68 0.46 1.00 6 624 1 0 0 1 7
PDB_01051 0.83 0.69 1.00 9 894 4 0 0 4 4
PDB_01092 0.38 0.21 0.69 11 10137 6 0 5 1 41
PDB_01152 0.92 0.86 1.00 12 549 0 0 0 0 2
RFA_00603 0.70 0.60 0.81 21 13504 7 0 5 2 14
RFA_00604 0.63 0.51 0.78 18 13507 8 0 5 3 17
RFA_00605 0.14 0.06 0.33 2 15219 5 0 4 1 33
RFA_00606 0.34 0.21 0.57 8 21307 8 2 4 2 31
RFA_00607 0.08 0.06 0.13 2 17750 15 3 11 1 34
RFA_00609 0.32 0.22 0.47 8 17938 12 2 7 3 29
RFA_00610 0.39 0.30 0.52 11 17184 13 2 8 3 26
RFA_00611 0.65 0.51 0.82 18 13019 5 0 4 1 17
RFA_00613 0.70 0.57 0.87 20 12857 4 0 3 1 15
RFA_00615 0.57 0.49 0.68 17 13341 10 0 8 2 18
RFA_00620 0.00 0.00 0.00 0 21939 6 0 6 0 39
RFA_00626 0.44 0.32 0.60 28 56569 22 2 17 3 59
RFA_00627 0.53 0.39 0.72 34 56906 15 3 10 2 53
RFA_00628 0.58 0.43 0.77 37 57243 14 3 8 3 49
RFA_00630 0.45 0.32 0.64 28 56909 17 3 13 1 59
RFA_00632 0.27 0.25 0.29 7 4071 17 0 17 0 21
RFA_00636 0.71 0.64 0.78 18 3982 5 2 3 0 10
RFA_00639 0.00 0.00 0.00 0 54575 44 6 34 4 87
RFA_00642 0.44 0.28 0.71 5 2919 2 0 2 0 13
RFA_00643 -0.01 0.00 0.00 0 2203 8 0 8 0 18
RFA_00644 0.00 0.00 0.00 0 2693 8 0 8 0 18
RFA_00645 0.29 0.17 0.50 3 2409 3 0 3 0 15
RFA_00649 0.35 0.22 0.57 4 2138 3 0 3 0 14
RFA_00651 0.35 0.22 0.57 4 2073 3 0 3 0 14
RFA_00653 -0.01 0.00 0.00 0 2137 8 0 8 0 18
RFA_00654 0.38 0.22 0.67 4 2409 2 0 2 0 14
RFA_00658 0.40 0.29 0.57 4 1121 4 0 3 1 10
RFA_00659 0.23 0.14 0.40 2 1123 4 0 3 1 12
RFA_00664 0.40 0.29 0.57 4 983 4 0 3 1 10
RFA_00667 0.40 0.29 0.57 4 983 4 0 3 1 10
RFA_00668 0.35 0.21 0.60 3 985 3 0 2 1 11
RFA_00672 0.41 0.31 0.57 4 896 3 0 3 0 9
RFA_00673 0.23 0.14 0.40 2 1123 4 0 3 1 12
RFA_00674 0.40 0.29 0.57 4 1121 4 0 3 1 10
RFA_00675 0.35 0.21 0.60 3 985 3 0 2 1 11
RFA_00677 0.40 0.29 0.57 4 983 4 0 3 1 10
RFA_00678 0.43 0.29 0.67 4 940 3 0 2 1 10
RFA_00680 0.40 0.29 0.57 4 1121 4 0 3 1 10
RFA_00684 0.43 0.29 0.67 4 984 3 0 2 1 10
RFA_00685 0.32 0.21 0.50 3 984 4 0 3 1 11
RFA_00704 0.32 0.21 0.50 3 984 4 0 3 1 11
RFA_00705 0.40 0.29 0.57 4 1028 4 0 3 1 10
RFA_00706 0.43 0.29 0.67 4 1029 3 0 2 1 10
RFA_00707 0.43 0.29 0.67 4 1029 3 0 2 1 10
RFA_00708 0.40 0.29 0.57 4 1028 4 0 3 1 10
RFA_00709 0.32 0.21 0.50 3 984 4 0 3 1 11
RFA_00710 0.32 0.21 0.50 3 984 3 0 3 0 11
RFA_00711 0.43 0.29 0.67 4 1029 3 0 2 1 10
RFA_00715 0.30 0.14 0.67 2 943 2 0 1 1 12
RFA_00716 0.43 0.29 0.67 4 940 3 0 2 1 10
RFA_00717 0.30 0.21 0.43 3 896 4 0 4 0 11
RFA_00730 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00731 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00733 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00734 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00736 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00737 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00745 0.91 0.83 1.00 10 936 0 0 0 0 2
RFA_00749 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00758 0.86 0.75 1.00 9 894 0 0 0 0 3
RFA_00762 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00763 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00764 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00765 0.91 0.83 1.00 10 893 0 0 0 0 2
RFA_00769 0.71 0.50 1.00 9 1944 0 0 0 0 9
RFA_00770 0.71 0.50 1.00 9 2007 0 0 0 0 9
RFA_00773 0.00 0.00 0.00 0 1949 4 0 4 0 18
RFA_00779 0.71 0.50 1.00 9 1944 0 0 0 0 9
RFA_00781 0.97 0.94 1.00 30 5020 0 0 0 0 2
RFA_00786 0.95 0.91 1.00 29 5021 0 0 0 0 3
RFA_00791 0.92 0.88 0.97 28 5122 1 0 1 0 4
RFA_00792 0.97 0.94 1.00 30 5020 0 0 0 0 2
RFA_00801 0.95 0.91 1.00 29 5021 0 0 0 0 3
RFA_00808 0.68 0.56 0.82 9 2005 2 0 2 0 7
RFA_00809 0.41 0.38 0.46 6 2132 7 1 6 0 10
RFA_00814 0.68 0.51 0.91 21 25177 2 0 2 0 20
RFA_00815 0.68 0.51 0.91 21 24508 2 0 2 0 20
RFA_00816 0.68 0.51 0.91 21 23197 2 0 2 0 20
RFA_00817 0.68 0.56 0.82 23 21917 5 0 5 0 18
RFA_00818 0.60 0.49 0.74 20 20274 8 1 6 1 21
RFA_00819 0.62 0.51 0.75 21 27938 7 0 7 0 20
SRP_00016 0.54 0.35 0.83 38 47849 8 0 8 0 71
SRP_00130 0.50 0.32 0.78 31 49730 10 1 8 1 66
SRP_00142 0.46 0.37 0.58 34 45997 26 1 24 1 59
SRP_00143 0.50 0.30 0.86 30 49735 5 0 5 0 71
SRP_00144 0.54 0.37 0.80 35 46621 11 0 9 2 59
SRP_00197 0.46 0.26 0.84 26 49424 5 0 5 0 75
SRP_00198 0.56 0.37 0.84 38 50995 7 0 7 0 65
SRP_00199 0.54 0.35 0.82 37 50995 8 0 8 0 68
SRP_00201 0.54 0.35 0.83 39 47848 8 0 8 0 72
SRP_00202 0.43 0.21 0.86 25 54256 4 0 4 0 92
SRP_00203 0.51 0.34 0.77 36 48781 12 0 11 1 69
SRP_00204 0.52 0.34 0.80 35 49097 9 1 8 0 67
SRP_00206 0.55 0.36 0.83 39 47848 8 0 8 0 69
SRP_00208 0.52 0.33 0.83 33 46625 7 1 6 0 68
SRP_00255 0.49 0.27 0.89 25 47558 3 0 3 0 69
SRP_00276 0.51 0.33 0.80 36 50676 9 0 9 0 74
SRP_00278 0.52 0.33 0.81 35 46013 8 0 8 0 70
SRP_00321 0.42 0.21 0.82 23 47558 5 1 4 0 85
SRP_00322 0.52 0.33 0.82 36 48472 8 0 8 0 73
SRP_00323 0.54 0.34 0.88 36 47237 5 0 5 0 71
SRP_00330 0.44 0.23 0.85 22 50060 5 0 4 1 73
SRP_00332 0.44 0.25 0.75 27 50367 9 0 9 0 79
SRP_00339 0.56 0.39 0.79 37 44504 10 2 8 0 57

^top



Performance of Vsfold5 - scored lower in this pairwise comparison

1. Total counts & total scores for Vsfold5

Total Base Pair Counts
Total TP 1611
Total TN 2652889
Total FP 4005
Total FP CONTRA 537
Total FP INCONS 3202
Total FP COMP 266
Total FN 4314
Total Scores
MCC 0.285
Average MCC ± 95% Confidence Intervals 0.319 ± 0.045
Sensitivity 0.272
Positive Predictive Value 0.301
Nr of predictions 127

^top



2. Individual counts for Vsfold5 [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_00016 0.40 0.38 0.42 45 77315 67 4 57 6 75
CRW_00610 0.29 0.28 0.30 23 36238 57 6 48 3 58
CRW_00613 0.48 0.46 0.51 36 34909 41 2 33 6 42
CRW_00618 0.24 0.26 0.23 16 56210 76 9 45 22 45
CRW_00633 0.00 0.00 0.00 0 63546 0 0 0 0 106
CRW_00634 0.32 0.33 0.32 31 64523 75 10 56 9 63
CRW_00670 0.40 0.38 0.43 45 70020 61 3 57 1 75
CRW_00671 0.29 0.26 0.33 31 62740 64 2 62 0 86
CRW_00672 0.39 0.40 0.39 44 72278 70 14 54 2 67
CRW_00674 0.33 0.31 0.36 38 83330 74 5 63 6 86
CRW_00676 0.34 0.36 0.33 41 93405 88 19 63 6 74
CRW_00692 0.36 0.38 0.34 34 68534 67 21 46 0 56
PDB_00005 0.78 0.79 0.79 11 932 3 1 2 0 3
PDB_00012 1.00 1.00 1.00 7 399 2 0 0 2 0
PDB_00213 0.40 0.37 0.45 15 5017 18 0 18 0 26
PDB_00553 0.95 0.91 1.00 10 455 0 0 0 0 1
PDB_00716 0.17 0.17 0.18 4 2679 19 0 18 1 19
PDB_00810 0.45 0.35 0.60 6 1071 4 2 2 0 11
PDB_00827 0.51 0.42 0.62 34 26973 22 0 21 1 47
PDB_01050 0.92 0.85 1.00 11 619 2 0 0 2 2
PDB_01051 0.36 0.38 0.36 5 889 9 2 7 0 8
PDB_01092 0.73 0.69 0.77 36 10106 12 2 9 1 16
PDB_01152 0.88 0.79 1.00 11 550 0 0 0 0 3
RFA_00603 0.39 0.37 0.41 13 13498 26 5 14 7 22
RFA_00604 0.48 0.49 0.49 17 13495 24 8 10 6 18
RFA_00605 0.09 0.09 0.09 3 15192 40 2 28 10 32
RFA_00606 0.41 0.49 0.35 19 21267 43 15 20 8 20
RFA_00607 0.45 0.47 0.43 17 17726 30 8 15 7 19
RFA_00609 0.54 0.57 0.53 21 17915 28 4 15 9 16
RFA_00610 0.64 0.65 0.63 24 17167 23 2 12 9 13
RFA_00611 0.57 0.54 0.59 19 13009 16 1 12 3 16
RFA_00613 0.39 0.37 0.41 13 12848 22 6 13 3 22
RFA_00615 0.50 0.49 0.52 17 13333 19 4 12 3 18
RFA_00620 0.23 0.23 0.24 9 21907 41 6 23 12 30
RFA_00626 0.31 0.31 0.32 27 56531 62 15 43 4 60
RFA_00627 0.26 0.26 0.26 23 56863 69 16 51 2 64
RFA_00628 0.34 0.35 0.34 30 57202 66 13 46 7 56
RFA_00630 0.27 0.29 0.26 25 56857 78 13 58 7 62
RFA_00632 -0.01 0.00 0.00 0 4073 22 0 22 0 28
RFA_00636 0.58 0.57 0.59 16 3978 11 1 10 0 12
RFA_00639 0.06 0.06 0.06 5 54526 88 15 69 4 82
RFA_00642 -0.01 0.00 0.00 0 2914 12 1 11 0 18
RFA_00643 0.38 0.39 0.39 7 2193 11 5 6 0 11
RFA_00644 0.22 0.22 0.22 4 2683 14 1 13 0 14
RFA_00645 -0.01 0.00 0.00 0 2400 15 0 15 0 18
RFA_00649 -0.01 0.00 0.00 0 2130 15 1 14 0 18
RFA_00651 0.47 0.44 0.50 8 2064 8 1 7 0 10
RFA_00653 -0.01 0.00 0.00 0 2131 14 3 11 0 18
RFA_00654 0.54 0.44 0.67 8 2403 5 1 3 1 10
RFA_00658 0.40 0.29 0.57 4 1121 5 0 3 2 10
RFA_00659 -0.01 0.00 0.00 0 1125 5 0 3 2 14
RFA_00664 -0.01 0.00 0.00 0 980 11 0 10 1 14
RFA_00667 0.21 0.21 0.23 3 977 10 3 7 0 11
RFA_00668 0.40 0.36 0.45 5 979 6 0 6 0 9
RFA_00672 0.00 0.00 0.00 0 903 0 0 0 0 13
RFA_00673 0.40 0.29 0.57 4 1121 4 0 3 1 10
RFA_00674 -0.01 0.00 0.00 0 1117 11 0 11 0 14
RFA_00675 -0.01 0.00 0.00 0 981 9 1 8 0 14
RFA_00677 0.64 0.57 0.73 8 979 5 0 3 2 6
RFA_00678 0.32 0.21 0.50 3 940 3 0 3 0 11
RFA_00680 0.69 0.64 0.75 9 1116 6 0 3 3 5
RFA_00684 0.75 0.57 1.00 8 982 1 0 0 1 6
RFA_00685 0.35 0.29 0.44 4 981 6 0 5 1 10
RFA_00704 -0.01 0.00 0.00 0 983 8 0 7 1 14
RFA_00705 0.69 0.64 0.75 9 1023 4 0 3 1 5
RFA_00706 -0.01 0.00 0.00 0 1022 13 2 11 0 14
RFA_00707 0.33 0.29 0.40 4 1025 7 0 6 1 10
RFA_00708 -0.01 0.00 0.00 0 1027 8 0 8 0 14
RFA_00709 -0.01 0.00 0.00 0 980 11 0 10 1 14
RFA_00710 -0.01 0.00 0.00 0 980 10 0 10 0 14
RFA_00711 -0.01 0.00 0.00 0 1026 9 0 9 0 14
RFA_00715 -0.01 0.00 0.00 0 937 9 0 9 0 14
RFA_00716 -0.01 0.00 0.00 0 936 10 0 10 0 14
RFA_00717 -0.01 0.00 0.00 0 894 9 0 9 0 14
RFA_00730 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00731 -0.01 0.00 0.00 0 896 7 1 6 0 12
RFA_00733 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00734 0.16 0.17 0.18 2 892 10 2 7 1 10
RFA_00736 -0.01 0.00 0.00 0 896 7 1 6 0 12
RFA_00737 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00745 -0.01 0.00 0.00 0 941 5 1 4 0 12
RFA_00749 0.51 0.42 0.63 5 895 4 1 2 1 7
RFA_00758 -0.01 0.00 0.00 0 898 5 0 5 0 12
RFA_00762 0.39 0.42 0.38 5 890 8 2 6 0 7
RFA_00763 0.64 0.42 1.00 5 898 0 0 0 0 7
RFA_00764 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00765 0.64 0.42 1.00 5 898 1 0 0 1 7
RFA_00769 0.50 0.50 0.50 9 1935 9 4 5 0 9
RFA_00770 0.65 0.56 0.77 10 2003 3 3 0 0 8
RFA_00773 -0.01 0.00 0.00 0 1936 19 1 16 2 18
RFA_00779 0.69 0.61 0.79 11 1939 3 3 0 0 7
RFA_00781 0.55 0.53 0.57 17 5020 13 1 12 0 15
RFA_00786 -0.01 0.00 0.00 0 5015 35 5 30 0 32
RFA_00791 0.38 0.34 0.42 11 5125 15 3 12 0 21
RFA_00792 0.57 0.53 0.61 17 5022 11 0 11 0 15
RFA_00801 0.52 0.50 0.55 16 5021 14 3 10 1 16
RFA_00808 1.00 1.00 1.00 16 2000 0 0 0 0 0
RFA_00809 0.45 0.44 0.47 7 2130 8 1 7 0 9
RFA_00814 0.31 0.34 0.29 14 25151 46 10 25 11 27
RFA_00815 0.22 0.24 0.20 10 24480 46 18 23 5 31
RFA_00816 0.31 0.32 0.30 13 23177 38 12 18 8 28
RFA_00817 0.32 0.34 0.30 14 21898 35 10 23 2 27
RFA_00818 0.19 0.20 0.19 8 20258 41 8 27 6 33
RFA_00819 0.27 0.27 0.28 11 27926 53 6 23 24 30
SRP_00016 0.14 0.13 0.15 14 47800 82 3 78 1 95
SRP_00130 0.11 0.10 0.12 10 49686 75 10 64 1 87
SRP_00142 0.25 0.26 0.25 24 45959 75 10 63 2 69
SRP_00143 0.20 0.20 0.20 20 49668 82 4 78 0 81
SRP_00144 0.15 0.14 0.16 13 46582 71 11 59 1 81
SRP_00197 0.23 0.23 0.24 23 49359 73 9 64 0 78
SRP_00198 0.11 0.11 0.13 11 50953 76 11 65 0 92
SRP_00199 0.24 0.24 0.25 25 50938 77 10 67 0 80
SRP_00201 0.27 0.26 0.28 29 47792 74 12 62 0 82
SRP_00202 0.51 0.50 0.53 59 54173 53 8 45 0 58
SRP_00203 0.19 0.19 0.19 20 48725 84 4 79 1 85
SRP_00204 0.06 0.06 0.07 6 49055 80 11 69 0 96
SRP_00206 0.22 0.20 0.24 22 47803 74 5 65 4 86
SRP_00208 0.09 0.09 0.10 9 46574 83 9 73 1 92
SRP_00255 0.22 0.22 0.22 21 47490 75 8 67 0 73
SRP_00276 0.25 0.25 0.27 27 50620 75 8 66 1 83
SRP_00278 0.17 0.16 0.17 17 45958 81 7 74 0 88
SRP_00321 0.15 0.14 0.16 15 47491 80 6 74 0 93
SRP_00322 0.16 0.16 0.17 17 48417 82 4 78 0 92
SRP_00323 0.23 0.21 0.25 23 47187 69 2 66 1 84
SRP_00330 0.12 0.13 0.12 12 49989 85 13 72 0 83
SRP_00332 0.29 0.29 0.29 31 50297 75 8 67 0 75
SRP_00339 0.19 0.18 0.19 17 44463 71 13 58 0 77

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.