CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of HotKnots - scored higher in this pairwise comparison

  4. Performance of MaxExpect - scored lower in this pairwise comparison

  5. Compile and download dataset for HotKnots & MaxExpect [.zip] - may take several seconds...


Overview

Metric HotKnots MaxExpect
MCC 0.728 > 0.717
Average MCC ± 95% Confidence Intervals 0.780 ± 0.054 > 0.746 ± 0.059
Sensitivity 0.755 > 0.733
Positive Predictive Value 0.707 < 0.707
Total TP 1725 > 1676
Total TN 221635 < 221705
Total FP 1032 > 1004
Total FP CONTRA 246 > 224
Total FP INCONS 470 < 471
Total FP COMP 316 > 309
Total FN 560 < 609
P-value 3.26115971074e-07

^top




Performance plots


  1. Comparison of performance of HotKnots and MaxExpect. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for HotKnots and MaxExpect).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for HotKnots and MaxExpect).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for HotKnots and MaxExpect. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for HotKnots and MaxExpect).

^top





Performance of HotKnots - scored higher in this pairwise comparison

1. Total counts & total scores for HotKnots

Total Base Pair Counts
Total TP 1725
Total TN 221635
Total FP 1032
Total FP CONTRA 246
Total FP INCONS 470
Total FP COMP 316
Total FN 560
Total Scores
MCC 0.728
Average MCC ± 95% Confidence Intervals 0.780 ± 0.054
Sensitivity 0.755
Positive Predictive Value 0.707
Nr of predictions 115

^top



2. Individual counts for HotKnots [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 99 0 0 0 0 0
2KE6_A 1.00 1.00 1.00 18 449 1 0 0 1 0
2KPV_A - 0.96 0.92 1.00 12 201 0 0 0 0 1
2KRL_A - 0.91 0.87 0.95 20 2003 9 1 0 8 3
2KU0_A - 0.96 0.92 1.00 11 281 0 0 0 0 1
2KUR_A 1.00 1.00 1.00 19 448 0 0 0 0 0
2KUU_A 1.00 1.00 1.00 18 429 1 0 0 1 0
2KUV_A 1.00 1.00 1.00 19 420 0 0 0 0 0
2KUW_A 1.00 1.00 1.00 18 452 1 0 0 1 0
2KX8_A 1.00 1.00 1.00 16 355 0 0 0 0 0
2KXM_A - 1.00 1.00 1.00 8 141 0 0 0 0 0
2KZL_A - 1.00 1.00 1.00 13 507 1 0 0 1 0
2L1F_A 1.00 1.00 1.00 23 740 0 0 0 0 0
2L1F_B 1.00 1.00 1.00 24 767 0 0 0 0 0
2L2K_A - 1.00 1.00 1.00 17 332 0 0 0 0 0
2L3C_B - 1.00 1.00 1.00 14 225 0 0 0 0 0
2L3E_A - 1.00 1.00 1.00 12 228 1 0 0 1 0
2L3J_B 1.00 1.00 1.00 30 955 0 0 0 0 0
2L5Z_A - 1.00 1.00 1.00 9 102 0 0 0 0 0
2L94_A 1.00 1.00 1.00 18 339 1 0 0 1 0
2LA5_A - 0.77 0.71 0.83 5 269 3 0 1 2 2
2LBS_A - 1.00 1.00 1.00 14 193 0 0 0 0 0
2LC8_A 0.64 0.61 0.69 11 512 5 0 5 0 7
2LDL_A - 1.00 1.00 1.00 9 131 1 0 0 1 0
2LDT_A - 1.00 1.00 1.00 11 151 0 0 0 0 0
2LHP_A - 1.00 1.00 1.00 15 246 0 0 0 0 0
2LI4_A - 1.00 1.00 1.00 14 175 0 0 0 0 0
2LJJ_A - 1.00 1.00 1.00 7 123 3 0 0 3 0
2LK3_A - 1.00 1.00 1.00 9 91 0 0 0 0 0
2LKR_A - 1.00 1.00 1.00 29 2411 8 0 0 8 0
2LQZ_A - 1.00 1.00 1.00 8 124 2 0 0 2 0
2LWK_A - 0.95 0.91 1.00 10 197 1 0 0 1 1
2RRC_A - 1.00 1.00 1.00 7 68 0 0 0 0 0
2WRQ_Y 1.00 1.00 1.00 9 1143 14 0 0 14 0
2WW9_D - 0.08 0.10 0.08 1 729 15 2 9 4 9
2WW9_E - 0.89 0.80 1.00 4 168 0 0 0 0 1
2WW9_F - 0.68 0.75 0.67 6 103 4 1 2 1 2
2WWQ_V 1.00 1.00 1.00 19 1185 5 0 0 5 0
2XKV_B 0.64 0.73 0.57 8 1821 25 0 6 19 3
2XQD_Y 0.98 0.95 1.00 20 1109 1 0 0 1 1
2XXA_G 0.35 0.34 0.38 12 2013 21 1 19 1 23
2Y8W_B - 1.00 1.00 1.00 6 82 1 0 0 1 0
2Y9C_V - 0.79 0.83 0.75 15 886 8 0 5 3 3
2YIE_X - 0.53 0.57 0.50 4 536 6 1 3 2 3
2YIE_Z - -0.02 0.00 0.00 0 587 15 6 9 0 8
3A2K_C 0.47 0.50 0.46 11 1084 13 3 10 0 11
3ADB_C - 0.86 0.85 0.88 28 1787 5 0 4 1 5
3AKZ_H 0.48 0.50 0.48 10 1106 13 4 7 2 10
3AM1_B - 0.86 0.86 0.86 25 1437 4 0 4 0 4
3AMU_B 0.81 0.79 0.83 15 1139 6 0 3 3 4
3GX2_A 0.81 0.79 0.85 22 1423 5 2 2 1 6
3IAB_R - 1.00 1.00 1.00 12 373 0 0 0 0 0
3IVN_B 0.91 0.83 1.00 19 884 0 0 0 0 4
3IWN_A 0.69 0.68 0.70 19 1445 9 1 7 1 9
3IYQ_A 0.24 0.33 0.17 17 22341 96 47 35 14 34
3IZ4_A 0.56 0.60 0.52 57 25426 57 26 27 4 38
3IZF_C 0.89 0.91 0.86 32 2603 9 0 5 4 3
3J0L_g - -0.01 0.00 0.00 0 173 5 1 2 2 2
3J0L_7 - 0.38 0.50 0.31 5 503 12 4 7 1 5
3J0L_h - 0.87 0.81 0.93 26 2112 4 1 1 2 6
3J0L_1 - 0.83 0.77 0.91 10 473 4 0 1 3 3
3J0L_a - 0.55 0.64 0.50 7 397 8 5 2 1 4
3J0L_8 - 1.00 1.00 1.00 7 69 0 0 0 0 0
3J0L_2 - 0.34 0.35 0.35 9 2224 23 2 15 6 17
3J16_L 0.31 0.33 0.30 7 1136 16 3 13 0 14
3J20_0 0.54 0.57 0.52 12 1196 12 3 8 1 9
3J20_1 0.75 0.75 0.75 15 1092 8 0 5 3 5
3J2C_O - 0.71 0.68 0.74 28 3949 15 3 7 5 13
3J2C_M - 0.83 0.85 0.81 82 39824 58 8 11 39 14
3J2L_3 0.80 0.82 0.78 28 2984 12 1 7 4 6
3JYV_7 -0.02 0.00 0.00 0 1089 22 4 18 0 20
3JYX_4 0.61 0.83 0.45 10 4734 31 10 2 19 2
3JYX_3 0.77 0.80 0.75 12 2362 27 0 4 23 3
3KTW_C - 0.96 0.96 0.96 24 1755 9 0 1 8 1
3LA5_A 0.91 0.84 1.00 21 933 0 0 0 0 4
3NDB_M - 1.00 1.00 1.00 44 3670 4 0 0 4 0
3NKB_B - 0.69 0.74 0.67 14 714 7 0 7 0 5
3NMU_E - 1.00 1.00 1.00 3 213 5 0 0 5 0
3NPB_A 0.85 0.78 0.94 29 2247 6 0 2 4 8
3O58_2 0.87 0.87 0.87 27 2723 12 0 4 8 4
3O58_3 0.26 0.36 0.19 8 4722 35 16 18 1 14
3OVB_D - 1.00 1.00 1.00 11 215 1 0 0 1 0
3OVS_D - 1.00 1.00 1.00 12 204 0 0 0 0 0
3P22_G - 1.00 1.00 1.00 11 301 0 0 0 0 0
3PDR_A 0.81 0.80 0.82 40 4791 11 3 6 2 10
3R4F_A - 1.00 1.00 1.00 22 884 1 0 0 1 0
3R9X_C - 1.00 1.00 1.00 8 221 2 0 0 2 0
3RKF_A 0.91 0.83 1.00 20 846 0 0 0 0 4
3SD1_A 0.77 0.76 0.79 22 1505 6 4 2 0 7
3SIU_F - 1.00 1.00 1.00 8 137 0 0 0 0 0
3SN2_B 0.95 0.92 1.00 11 143 0 0 0 0 1
3TRZ_Z - 1.00 1.00 1.00 5 87 1 0 0 1 0
3TS0_U - 1.00 1.00 1.00 6 112 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 103 0 0 0 0 0
3U4M_B - 0.49 0.55 0.46 12 1250 14 3 11 0 10
3UZL_B 0.49 0.50 0.50 8 1277 15 0 8 7 8
3VJR_D - 1.00 1.00 1.00 12 239 0 0 0 0 0
3W3S_B 0.60 0.61 0.61 20 1956 14 4 9 1 13
3ZEX_F - -0.01 0.00 0.00 0 908 11 2 4 5 4
3ZEX_C -0.01 0.00 0.00 0 5328 46 11 35 0 29
3ZEX_G - 0.86 0.84 0.88 38 6464 22 0 5 17 7
3ZEX_D 0.90 0.86 0.94 30 2764 6 0 2 4 5
3ZEX_E - 0.00 0.00 0.00 0 8255 54 17 32 5 34
3ZEX_H - 0.26 0.37 0.19 7 3588 30 19 11 0 12
4A1C_2 0.19 0.25 0.15 5 4483 42 12 16 14 15
4A1C_3 0.88 0.86 0.89 32 2727 7 0 4 3 5
4A4U_A - 1.00 1.00 1.00 9 96 0 0 0 0 0
4AOB_A 0.60 0.59 0.63 17 1410 11 4 6 1 12
4ATO_G - 0.72 1.00 0.54 7 207 6 6 0 0 0
4ENB_A 1.00 1.00 1.00 15 457 0 0 0 0 0
4ENC_A 0.97 1.00 0.94 15 480 1 1 0 0 0
4FNJ_A - 0.91 0.83 1.00 10 240 0 0 0 0 2
4FRG_B 0.36 0.38 0.38 9 1178 15 2 13 0 15
4FRN_A 0.59 0.57 0.62 16 1822 10 2 8 0 12
4HXH_A - 1.00 1.00 1.00 6 89 0 0 0 0 0

^top



Performance of MaxExpect - scored lower in this pairwise comparison

1. Total counts & total scores for MaxExpect

Total Base Pair Counts
Total TP 1676
Total TN 221705
Total FP 1004
Total FP CONTRA 224
Total FP INCONS 471
Total FP COMP 309
Total FN 609
Total Scores
MCC 0.717
Average MCC ± 95% Confidence Intervals 0.746 ± 0.059
Sensitivity 0.733
Positive Predictive Value 0.707
Nr of predictions 115

^top



2. Individual counts for MaxExpect [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 99 0 0 0 0 0
2KE6_A 0.94 0.94 0.94 17 449 2 0 1 1 1
2KPV_A - 0.96 0.92 1.00 12 201 0 0 0 0 1
2KRL_A - 0.91 0.87 0.95 20 2003 9 1 0 8 3
2KU0_A - -0.04 0.00 0.00 0 283 9 1 8 0 12
2KUR_A 0.95 0.95 0.95 18 448 1 0 1 0 1
2KUU_A 0.94 0.94 0.94 17 429 2 0 1 1 1
2KUV_A 0.94 0.95 0.95 18 420 1 0 1 0 1
2KUW_A 0.94 0.94 0.94 17 452 2 0 1 1 1
2KX8_A 1.00 1.00 1.00 16 355 0 0 0 0 0
2KXM_A - 0.93 0.88 1.00 7 142 0 0 0 0 1
2KZL_A - 1.00 1.00 1.00 13 507 1 0 0 1 0
2L1F_A 1.00 1.00 1.00 23 740 0 0 0 0 0
2L1F_B 1.00 1.00 1.00 24 767 0 0 0 0 0
2L2K_A - 1.00 1.00 1.00 17 332 0 0 0 0 0
2L3C_B - 1.00 1.00 1.00 14 225 0 0 0 0 0
2L3E_A - 1.00 1.00 1.00 12 228 1 0 0 1 0
2L3J_B 1.00 1.00 1.00 30 955 0 0 0 0 0
2L5Z_A - 1.00 1.00 1.00 9 102 0 0 0 0 0
2L94_A 1.00 1.00 1.00 18 339 1 0 0 1 0
2LA5_A - 0.77 0.71 0.83 5 269 3 0 1 2 2
2LBS_A - 1.00 1.00 1.00 14 193 0 0 0 0 0
2LC8_A -0.03 0.00 0.00 0 513 15 3 12 0 18
2LDL_A - 1.00 1.00 1.00 9 131 1 0 0 1 0
2LDT_A - 1.00 1.00 1.00 11 151 0 0 0 0 0
2LHP_A - 1.00 1.00 1.00 15 246 0 0 0 0 0
2LI4_A - 1.00 1.00 1.00 14 175 0 0 0 0 0
2LJJ_A - 1.00 1.00 1.00 7 123 3 0 0 3 0
2LK3_A - 1.00 1.00 1.00 9 91 0 0 0 0 0
2LKR_A - 0.84 0.83 0.86 24 2412 11 0 4 7 5
2LQZ_A - 1.00 1.00 1.00 8 124 2 0 0 2 0
2LWK_A - 1.00 1.00 1.00 11 196 1 0 0 1 0
2RRC_A - 0.83 0.71 1.00 5 70 0 0 0 0 2
2WRQ_Y 0.94 0.89 1.00 8 1144 14 0 0 14 1
2WW9_D - 0.08 0.10 0.08 1 729 15 2 9 4 9
2WW9_E - -0.03 0.00 0.00 0 168 5 0 4 1 5
2WW9_F - 0.64 0.75 0.60 6 102 4 1 3 0 2
2WWQ_V 0.97 0.95 1.00 18 1186 3 0 0 3 1
2XKV_B 0.64 0.73 0.57 8 1821 26 0 6 20 3
2XQD_Y 0.98 0.95 1.00 20 1109 1 0 0 1 1
2XXA_G 0.24 0.26 0.25 9 2009 28 1 26 1 26
2Y8W_B - 1.00 1.00 1.00 6 82 1 0 0 1 0
2Y9C_V - 0.79 0.83 0.75 15 886 8 0 5 3 3
2YIE_X - 0.53 0.57 0.50 4 536 6 1 3 2 3
2YIE_Z - -0.02 0.00 0.00 0 587 15 6 9 0 8
3A2K_C 0.47 0.50 0.46 11 1084 13 3 10 0 11
3ADB_C - 0.86 0.85 0.88 28 1787 4 0 4 0 5
3AKZ_H 1.00 1.00 1.00 20 1107 2 0 0 2 0
3AM1_B - 0.86 0.86 0.86 25 1437 4 0 4 0 4
3AMU_B 0.73 0.79 0.68 15 1135 10 0 7 3 4
3GX2_A 0.93 0.89 0.96 25 1423 2 1 0 1 3
3IAB_R - 1.00 1.00 1.00 12 373 0 0 0 0 0
3IVN_B 0.91 0.83 1.00 19 884 0 0 0 0 4
3IWN_A 0.80 0.79 0.81 22 1445 6 1 4 1 6
3IYQ_A 0.24 0.33 0.17 17 22342 96 41 40 15 34
3IZ4_A 0.61 0.61 0.62 58 25442 42 14 22 6 37
3IZF_C 0.89 0.91 0.86 32 2603 8 0 5 3 3
3J0L_g - -0.01 0.00 0.00 0 174 4 0 2 2 2
3J0L_7 - -0.02 0.00 0.00 0 507 12 3 9 0 10
3J0L_h - 0.93 0.88 1.00 28 2112 2 0 0 2 4
3J0L_1 - 0.83 0.77 0.91 10 473 4 0 1 3 3
3J0L_a - 0.39 0.36 0.44 4 402 6 4 1 1 7
3J0L_8 - 1.00 1.00 1.00 7 69 0 0 0 0 0
3J0L_2 - 0.30 0.31 0.31 8 2224 25 2 16 7 18
3J16_L 0.34 0.33 0.37 7 1140 12 0 12 0 14
3J20_0 0.53 0.57 0.50 12 1195 13 2 10 1 9
3J20_1 0.75 0.75 0.75 15 1092 6 0 5 1 5
3J2C_O - 0.74 0.71 0.78 29 3950 13 0 8 5 12
3J2C_M - 0.48 0.54 0.43 52 39804 88 29 40 19 44
3J2L_3 0.80 0.82 0.78 28 2984 12 1 7 4 6
3JYV_7 0.48 0.45 0.53 9 1094 9 3 5 1 11
3JYX_4 0.55 0.83 0.37 10 4729 34 14 3 17 2
3JYX_3 0.77 0.80 0.75 12 2362 24 0 4 20 3
3KTW_C - 0.72 0.72 0.72 18 1755 14 2 5 7 7
3LA5_A 0.91 0.84 1.00 21 933 0 0 0 0 4
3NDB_M - 0.94 0.93 0.95 41 3671 7 0 2 5 3
3NKB_B - 0.69 0.74 0.67 14 714 7 0 7 0 5
3NMU_E - -0.02 0.00 0.00 0 208 10 5 3 2 3
3NPB_A 0.77 0.73 0.82 27 2245 10 0 6 4 10
3O58_2 0.91 0.94 0.88 29 2721 11 0 4 7 2
3O58_3 0.40 0.50 0.33 11 4731 36 8 14 14 11
3OVB_D - 1.00 1.00 1.00 11 215 1 0 0 1 0
3OVS_D - 1.00 1.00 1.00 12 204 0 0 0 0 0
3P22_G - 1.00 1.00 1.00 11 301 0 0 0 0 0
3PDR_A 0.92 0.90 0.94 45 4792 5 1 2 2 5
3R4F_A - 1.00 1.00 1.00 22 884 1 0 0 1 0
3R9X_C - 1.00 1.00 1.00 8 221 2 0 0 2 0
3RKF_A 0.91 0.83 1.00 20 846 0 0 0 0 4
3SD1_A 0.70 0.69 0.71 20 1505 8 3 5 0 9
3SIU_F - 1.00 1.00 1.00 8 137 0 0 0 0 0
3SN2_B 1.00 1.00 1.00 12 142 0 0 0 0 0
3TRZ_Z - 1.00 1.00 1.00 5 87 1 0 0 1 0
3TS0_U - 1.00 1.00 1.00 6 112 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 103 0 0 0 0 0
3U4M_B - 0.58 0.59 0.59 13 1254 9 2 7 0 9
3UZL_B 1.00 1.00 1.00 16 1277 8 0 0 8 0
3VJR_D - 1.00 1.00 1.00 12 239 0 0 0 0 0
3W3S_B 0.97 0.94 1.00 31 1958 1 0 0 1 2
3ZEX_F - -0.01 0.00 0.00 0 908 11 2 4 5 4
3ZEX_C 0.54 0.45 0.65 13 5354 17 2 5 10 16
3ZEX_G - 0.98 0.96 1.00 43 6464 17 0 0 17 2
3ZEX_D 0.82 0.77 0.87 27 2765 11 0 4 7 8
3ZEX_E - -0.01 0.00 0.00 0 8252 54 21 31 2 34
3ZEX_H - 0.26 0.37 0.19 7 3589 29 18 11 0 12
4A1C_2 0.19 0.25 0.16 5 4484 41 10 17 14 15
4A1C_3 0.83 0.81 0.86 30 2728 7 0 5 2 7
4A4U_A - 1.00 1.00 1.00 9 96 0 0 0 0 0
4AOB_A 0.60 0.59 0.63 17 1410 11 4 6 1 12
4ATO_G - 0.45 0.43 0.50 3 214 3 3 0 0 4
4ENB_A 0.85 0.73 1.00 11 461 0 0 0 0 4
4ENC_A 0.36 0.33 0.42 5 484 8 0 7 1 10
4FNJ_A - 0.91 0.83 1.00 10 240 0 0 0 0 2
4FRG_B 0.29 0.25 0.35 6 1185 11 4 7 0 18
4FRN_A 0.51 0.46 0.57 13 1825 10 5 5 0 15
4HXH_A - 1.00 1.00 1.00 6 89 0 0 0 0 0

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.