CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of MaxExpect - scored higher in this pairwise comparison

  4. Performance of Murlet(seed) - scored lower in this pairwise comparison

  5. Compile and download dataset for MaxExpect & Murlet(seed) [.zip] - may take several seconds...


Overview

Metric MaxExpect Murlet(seed)
MCC 0.598 > 0.540
Average MCC ± 95% Confidence Intervals 0.588 ± 0.057 > 0.396 ± 0.065
Sensitivity 0.585 > 0.360
Positive Predictive Value 0.613 < 0.811
Total TP 2825 > 1738
Total TN 2787032 < 2789495
Total FP 2218 > 489
Total FP CONTRA 272 > 21
Total FP INCONS 1510 > 385
Total FP COMP 436 > 83
Total FN 2003 < 3090
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of MaxExpect and Murlet(seed). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for MaxExpect and Murlet(seed)).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for MaxExpect and Murlet(seed)).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for MaxExpect and Murlet(seed). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for MaxExpect and Murlet(seed)).

^top





Performance of MaxExpect - scored higher in this pairwise comparison

1. Total counts & total scores for MaxExpect

Total Base Pair Counts
Total TP 2825
Total TN 2787032
Total FP 2218
Total FP CONTRA 272
Total FP INCONS 1510
Total FP COMP 436
Total FN 2003
Total Scores
MCC 0.598
Average MCC ± 95% Confidence Intervals 0.588 ± 0.057
Sensitivity 0.585
Positive Predictive Value 0.613
Nr of predictions 123

^top



2. Individual counts for MaxExpect [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_01343 0.37 0.35 0.38 154 977498 261 31 218 12 280
PDB_00005 0.88 0.79 1.00 11 935 0 0 0 0 3
PDB_00012 1.00 1.00 1.00 7 399 3 0 0 3 0
PDB_00213 0.92 0.85 1.00 35 5015 0 0 0 0 6
PDB_00553 1.00 1.00 1.00 11 454 0 0 0 0 0
PDB_00716 -0.01 0.00 0.00 0 2678 24 1 22 1 23
PDB_00810 0.87 0.76 1.00 13 1068 0 0 0 0 4
PDB_01050 0.92 0.85 1.00 11 619 2 0 0 2 2
PDB_01051 0.88 0.85 0.92 11 891 5 0 1 4 2
PDB_01092 0.75 0.63 0.89 33 10116 6 0 4 2 19
PDB_01152 0.88 0.79 1.00 11 550 0 0 0 0 3
RFA_00390 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00416 0.97 0.93 1.00 14 1471 3 0 0 3 1
RFA_00433 1.00 1.00 1.00 15 1416 2 0 0 2 0
RFA_00587 1.00 1.00 1.00 15 4836 8 0 0 8 0
RFA_00603 0.69 0.74 0.63 26 13489 18 4 11 3 9
RFA_00604 0.72 0.74 0.70 26 13493 21 2 9 10 9
RFA_00605 0.35 0.34 0.36 12 15192 32 3 18 11 23
RFA_00606 0.68 0.67 0.70 26 21284 24 7 4 13 13
RFA_00607 0.83 0.83 0.83 30 17730 17 4 2 11 6
RFA_00609 0.55 0.59 0.51 22 17912 28 4 17 7 15
RFA_00610 0.90 0.86 0.94 32 17171 11 0 2 9 5
RFA_00611 0.87 0.86 0.88 30 13007 11 0 4 7 5
RFA_00613 0.73 0.69 0.77 24 12849 12 4 3 5 11
RFA_00615 0.67 0.66 0.68 23 13332 11 4 7 0 12
RFA_00620 0.44 0.49 0.40 19 21897 42 7 22 13 20
RFA_00626 0.88 0.84 0.92 73 56537 23 2 4 17 14
RFA_00627 0.71 0.68 0.74 59 56873 37 5 16 16 28
RFA_00628 0.91 0.88 0.94 76 57210 25 0 5 20 10
RFA_00630 0.66 0.66 0.66 57 56867 44 7 22 15 30
RFA_00632 0.34 0.36 0.33 10 4065 20 2 18 0 18
RFA_00636 0.41 0.43 0.40 12 3975 18 2 16 0 16
RFA_00639 0.49 0.51 0.48 44 54523 57 12 36 9 43
RFA_00642 0.22 0.17 0.30 3 2916 7 1 6 0 15
RFA_00643 0.22 0.22 0.24 4 2194 14 2 11 1 14
RFA_00644 0.00 0.00 0.00 0 2693 8 0 8 0 18
RFA_00645 -0.01 0.00 0.00 0 2404 11 2 9 0 18
RFA_00649 0.79 0.72 0.87 13 2130 4 1 1 2 5
RFA_00651 0.26 0.22 0.31 4 2067 10 0 9 1 14
RFA_00653 0.34 0.33 0.35 6 2128 11 2 9 0 12
RFA_00654 -0.01 0.00 0.00 0 2401 14 2 12 0 18
RFA_00658 0.59 0.50 0.70 7 1118 5 0 3 2 7
RFA_00659 -0.01 0.00 0.00 0 1120 10 1 7 2 14
RFA_00664 -0.01 0.00 0.00 0 980 11 0 10 1 14
RFA_00667 0.23 0.21 0.27 3 979 8 1 7 0 11
RFA_00668 0.36 0.36 0.38 5 977 8 0 8 0 9
RFA_00672 -0.01 0.00 0.00 0 897 6 0 6 0 13
RFA_00673 0.35 0.29 0.44 4 1119 6 0 5 1 10
RFA_00674 0.23 0.21 0.27 3 1117 8 0 8 0 11
RFA_00675 -0.01 0.00 0.00 0 980 11 0 10 1 14
RFA_00677 0.36 0.36 0.38 5 977 9 0 8 1 9
RFA_00678 0.43 0.29 0.67 4 940 2 0 2 0 10
RFA_00680 0.69 0.64 0.75 9 1116 6 0 3 3 5
RFA_00684 0.61 0.57 0.67 8 978 5 0 4 1 6
RFA_00685 0.35 0.29 0.44 4 981 6 0 5 1 10
RFA_00695 0.56 0.50 0.64 7 7010 28 1 3 24 7
RFA_00703 0.67 0.64 0.69 9 4265 18 1 3 14 5
RFA_00704 -0.01 0.00 0.00 0 988 3 0 2 1 14
RFA_00705 0.64 0.64 0.64 9 1021 5 0 5 0 5
RFA_00706 0.37 0.29 0.50 4 1027 4 0 4 0 10
RFA_00707 -0.01 0.00 0.00 0 1028 7 1 6 0 14
RFA_00708 0.40 0.29 0.57 4 1028 3 0 3 0 10
RFA_00709 -0.01 0.00 0.00 0 981 9 0 9 0 14
RFA_00710 -0.01 0.00 0.00 0 981 9 0 9 0 14
RFA_00711 -0.01 0.00 0.00 0 1026 9 0 9 0 14
RFA_00715 -0.01 0.00 0.00 0 933 13 0 13 0 14
RFA_00716 -0.01 0.00 0.00 0 935 11 0 11 0 14
RFA_00717 0.59 0.57 0.62 8 890 5 0 5 0 6
RFA_00730 0.69 0.67 0.73 8 892 4 1 2 1 4
RFA_00731 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00733 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00734 0.78 0.75 0.82 9 892 3 0 2 1 3
RFA_00736 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00737 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00745 1.00 1.00 1.00 12 934 1 0 0 1 0
RFA_00749 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00758 0.96 0.92 1.00 11 892 0 0 0 0 1
RFA_00762 0.66 0.67 0.67 8 891 4 0 4 0 4
RFA_00763 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00764 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00765 1.00 1.00 1.00 12 891 1 0 0 1 0
RFA_00767 0.63 0.56 0.71 10 1877 4 0 4 0 8
RFA_00768 0.47 0.44 0.50 8 1875 8 1 7 0 10
RFA_00769 0.54 0.56 0.53 10 1934 9 4 5 0 8
RFA_00770 0.63 0.56 0.71 10 2002 7 0 4 3 8
RFA_00773 0.59 0.56 0.63 10 1937 6 1 5 0 8
RFA_00779 0.61 0.56 0.67 10 1938 5 2 3 0 8
RFA_00781 0.73 0.75 0.71 24 5016 10 2 8 0 8
RFA_00786 0.64 0.66 0.64 21 5017 12 2 10 0 11
RFA_00791 0.95 0.91 1.00 29 5122 1 0 0 1 3
RFA_00792 0.95 0.94 0.97 30 5019 2 0 1 1 2
RFA_00801 0.81 0.78 0.83 25 5020 6 0 5 1 7
RFA_00808 0.62 0.56 0.69 9 2003 4 0 4 0 7
RFA_00809 0.37 0.38 0.38 6 2129 10 1 9 0 10
RFA_00814 0.24 0.27 0.22 11 25149 45 9 31 5 30
RFA_00815 0.57 0.61 0.54 25 24485 29 4 17 8 16
RFA_00816 0.63 0.66 0.60 27 23175 24 11 7 6 14
RFA_00817 0.10 0.12 0.08 5 21883 57 17 40 0 36
RFA_00818 0.48 0.54 0.43 22 20250 30 12 17 1 19
RFA_00819 0.96 0.95 0.98 39 27926 33 0 1 32 2
RFA_00820 0.87 0.84 0.91 134 151378 25 1 12 12 26
RFA_00830 0.48 0.45 0.51 69 152494 108 5 60 43 85
SRP_00079 0.75 0.73 0.77 64 41822 28 3 16 9 24
SRP_00124 0.64 0.62 0.65 53 37047 33 3 25 5 32
SRP_00137 0.74 0.84 0.66 21 4154 11 4 7 0 4
SRP_00141 0.94 0.93 0.95 38 6288 3 1 1 1 3
SRP_00146 0.93 0.94 0.92 34 5216 3 1 2 0 2
SRP_00182 0.73 0.71 0.74 72 45959 28 3 22 3 29
SRP_00197 0.21 0.20 0.24 20 49370 65 3 62 0 81
SRP_00198 0.22 0.21 0.23 22 50943 75 7 68 0 81
SRP_00199 0.65 0.65 0.65 68 50935 39 3 34 2 37
SRP_00202 0.87 0.86 0.87 101 54169 18 2 13 3 16
SRP_00241 0.44 0.48 0.41 39 45961 58 14 42 2 43
SRP_00255 0.60 0.62 0.59 58 47487 45 2 39 4 36
SRP_00260 0.62 0.63 0.61 63 47791 44 5 36 3 37
SRP_00278 0.67 0.68 0.66 71 45949 39 6 30 3 34
SRP_00285 0.98 0.97 1.00 29 3712 0 0 0 0 1
SRP_00322 0.67 0.65 0.68 71 48412 34 1 32 1 38
SRP_00328 0.62 0.64 0.61 54 39532 40 8 27 5 31
SRP_00329 0.57 0.58 0.57 49 39817 41 6 31 4 36
SRP_00338 0.97 0.97 0.97 35 5424 1 0 1 0 1
SRP_00339 0.66 0.66 0.66 62 44457 36 2 30 4 32
SRP_00340 0.49 0.52 0.47 43 41236 52 9 40 3 39

^top



Performance of Murlet(seed) - scored lower in this pairwise comparison

1. Total counts & total scores for Murlet(seed)

Total Base Pair Counts
Total TP 1738
Total TN 2789495
Total FP 489
Total FP CONTRA 21
Total FP INCONS 385
Total FP COMP 83
Total FN 3090
Total Scores
MCC 0.540
Average MCC ± 95% Confidence Intervals 0.396 ± 0.065
Sensitivity 0.360
Positive Predictive Value 0.811
Nr of predictions 123

^top



2. Individual counts for Murlet(seed) [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
CRW_01343 0.64 0.61 0.68 263 977515 145 6 117 22 171
PDB_00005 0.53 0.29 1.00 4 942 2 0 0 2 10
PDB_00012 0.65 0.43 1.00 3 403 0 0 0 0 4
PDB_00213 0.73 0.59 0.92 24 5024 2 0 2 0 17
PDB_00553 0.45 0.27 0.75 3 461 1 0 1 0 8
PDB_00716 0.23 0.13 0.43 3 2694 4 0 4 0 20
PDB_00810 0.63 0.53 0.75 9 1069 3 0 3 0 8
PDB_01050 0.55 0.31 1.00 4 626 1 0 0 1 9
PDB_01051 0.00 0.00 0.00 0 903 0 0 0 0 13
PDB_01092 0.69 0.48 1.00 25 10128 0 0 0 0 27
PDB_01152 0.76 0.64 0.90 9 551 1 0 1 0 5
RFA_00390 0.00 0.00 0.00 0 1431 0 0 0 0 15
RFA_00416 0.00 0.00 0.00 0 1485 0 0 0 0 15
RFA_00433 0.00 0.00 0.00 0 1431 0 0 0 0 15
RFA_00587 0.00 0.00 0.00 0 4851 0 0 0 0 15
RFA_00603 0.00 0.00 0.00 0 13530 0 0 0 0 35
RFA_00604 0.00 0.00 0.00 0 13530 0 0 0 0 35
RFA_00605 0.00 0.00 0.00 0 15225 0 0 0 0 35
RFA_00606 0.00 0.00 0.00 0 21321 0 0 0 0 39
RFA_00607 0.00 0.00 0.00 0 17766 0 0 0 0 36
RFA_00609 0.00 0.00 0.00 0 17955 0 0 0 0 37
RFA_00610 0.00 0.00 0.00 0 17205 0 0 0 0 37
RFA_00611 0.00 0.00 0.00 0 13041 0 0 0 0 35
RFA_00613 0.00 0.00 0.00 0 12880 0 0 0 0 35
RFA_00615 0.00 0.00 0.00 0 13366 0 0 0 0 35
RFA_00620 0.00 0.00 0.00 0 21945 0 0 0 0 39
RFA_00626 0.81 0.66 1.00 57 56559 4 0 0 4 30
RFA_00627 0.84 0.70 1.00 61 56892 5 0 0 5 26
RFA_00628 0.83 0.69 1.00 59 57232 8 0 0 8 27
RFA_00630 0.82 0.67 1.00 58 56895 4 0 0 4 29
RFA_00632 0.21 0.11 0.43 3 4088 4 0 4 0 25
RFA_00636 0.21 0.11 0.43 3 3998 4 0 4 0 25
RFA_00639 0.21 0.11 0.40 10 54590 15 4 11 0 77
RFA_00642 0.57 0.44 0.73 8 2915 3 0 3 0 10
RFA_00643 -0.01 0.00 0.00 0 2203 8 0 8 0 18
RFA_00644 0.42 0.33 0.55 6 2690 5 0 5 0 12
RFA_00645 0.00 0.00 0.00 0 2407 8 0 8 0 18
RFA_00649 -0.01 0.00 0.00 0 2137 8 0 8 0 18
RFA_00651 0.42 0.33 0.55 6 2069 5 0 5 0 12
RFA_00653 -0.01 0.00 0.00 0 2137 8 0 8 0 18
RFA_00654 0.57 0.44 0.73 8 2404 3 0 3 0 10
RFA_00658 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00659 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00664 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00667 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00668 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00672 0.00 0.00 0.00 0 903 0 0 0 0 13
RFA_00673 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00674 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00675 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00677 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00678 0.00 0.00 0.00 0 946 0 0 0 0 14
RFA_00680 0.00 0.00 0.00 0 1128 0 0 0 0 14
RFA_00684 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00685 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00695 0.00 0.00 0.00 0 7021 0 0 0 0 14
RFA_00703 0.00 0.00 0.00 0 4278 0 0 0 0 14
RFA_00704 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00705 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00706 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00707 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00708 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00709 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00710 0.00 0.00 0.00 0 990 0 0 0 0 14
RFA_00711 0.00 0.00 0.00 0 1035 0 0 0 0 14
RFA_00715 0.00 0.00 0.00 0 946 0 0 0 0 14
RFA_00716 0.00 0.00 0.00 0 946 0 0 0 0 14
RFA_00717 0.00 0.00 0.00 0 903 0 0 0 0 14
RFA_00730 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00731 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00733 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00734 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00736 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00737 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00745 1.00 1.00 1.00 12 934 0 0 0 0 0
RFA_00749 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00758 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00762 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00763 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00764 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00765 1.00 1.00 1.00 12 891 0 0 0 0 0
RFA_00767 0.47 0.33 0.67 6 1882 3 0 3 0 12
RFA_00768 0.41 0.28 0.63 5 1883 3 0 3 0 13
RFA_00769 0.57 0.44 0.73 8 1942 3 0 3 0 10
RFA_00770 0.57 0.44 0.73 8 2005 3 0 3 0 10
RFA_00773 0.41 0.28 0.63 5 1945 3 0 3 0 13
RFA_00779 0.47 0.33 0.67 6 1944 3 0 3 0 12
RFA_00781 0.83 0.75 0.92 24 5024 2 0 2 0 8
RFA_00786 0.83 0.75 0.92 24 5024 2 0 2 0 8
RFA_00791 0.83 0.75 0.92 24 5125 2 0 2 0 8
RFA_00792 0.83 0.75 0.92 24 5024 2 0 2 0 8
RFA_00801 0.80 0.72 0.88 23 5024 3 0 3 0 9
RFA_00808 0.75 0.56 1.00 9 2007 0 0 0 0 7
RFA_00809 0.50 0.38 0.67 6 2136 3 0 3 0 10
RFA_00814 0.75 0.63 0.90 26 25171 4 0 3 1 15
RFA_00815 0.83 0.68 1.00 28 24503 1 0 0 1 13
RFA_00816 0.75 0.63 0.90 26 23191 4 0 3 1 15
RFA_00817 0.83 0.68 1.00 28 21917 1 0 0 1 13
RFA_00818 0.83 0.68 1.00 28 20273 1 0 0 1 13
RFA_00819 0.83 0.68 1.00 28 27938 0 0 0 0 13
RFA_00820 0.71 0.56 0.89 90 151424 13 3 8 2 70
RFA_00830 0.59 0.45 0.78 70 152538 23 3 17 3 84
SRP_00079 0.54 0.33 0.88 29 41872 6 0 4 2 59
SRP_00124 0.15 0.09 0.24 8 37094 27 3 23 1 77
SRP_00137 0.49 0.24 1.00 6 4180 0 0 0 0 19
SRP_00141 0.35 0.12 1.00 5 6323 1 0 0 1 36
SRP_00146 0.41 0.17 1.00 6 5247 0 0 0 0 30
SRP_00182 0.59 0.35 1.00 35 46021 1 0 0 1 66
SRP_00197 0.47 0.28 0.80 28 49420 8 0 7 1 73
SRP_00198 0.47 0.27 0.82 28 51006 7 0 6 1 75
SRP_00199 0.47 0.24 0.93 25 51013 3 0 2 1 80
SRP_00202 0.48 0.25 0.94 29 54254 2 0 2 0 88
SRP_00241 0.51 0.33 0.79 27 46022 9 0 7 2 55
SRP_00255 0.65 0.43 1.00 40 47546 2 0 0 2 54
SRP_00260 0.21 0.14 0.33 14 47852 29 1 28 0 86
SRP_00278 0.53 0.30 0.94 31 46023 3 0 2 1 74
SRP_00285 0.45 0.20 1.00 6 3735 0 0 0 0 24
SRP_00322 0.44 0.25 0.79 27 48482 8 0 7 1 82
SRP_00328 0.49 0.34 0.71 29 39580 16 1 11 4 56
SRP_00329 0.50 0.34 0.73 29 39863 16 0 11 5 56
SRP_00338 0.41 0.17 1.00 6 5454 0 0 0 0 30
SRP_00339 0.49 0.32 0.77 30 44512 9 0 9 0 64
SRP_00340 0.44 0.27 0.73 22 41298 12 0 8 4 60

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.