CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of MaxExpect - scored higher in this pairwise comparison

  4. Performance of RNAshapes - scored lower in this pairwise comparison

  5. Compile and download dataset for MaxExpect & RNAshapes [.zip] - may take several seconds...


Overview

Metric MaxExpect RNAshapes
MCC 0.717 > 0.709
Average MCC ± 95% Confidence Intervals 0.746 ± 0.059 < 0.760 ± 0.055
Sensitivity 0.733 > 0.728
Positive Predictive Value 0.707 > 0.697
Total TP 1676 > 1663
Total TN 221705 > 221691
Total FP 1004 < 1047
Total FP CONTRA 224 > 215
Total FP INCONS 471 < 507
Total FP COMP 309 < 325
Total FN 609 < 622
P-value 3.03703286746e-07

^top




Performance plots


  1. Comparison of performance of MaxExpect and RNAshapes. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for MaxExpect and RNAshapes).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for MaxExpect and RNAshapes).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for MaxExpect and RNAshapes. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for MaxExpect and RNAshapes).

^top





Performance of MaxExpect - scored higher in this pairwise comparison

1. Total counts & total scores for MaxExpect

Total Base Pair Counts
Total TP 1676
Total TN 221705
Total FP 1004
Total FP CONTRA 224
Total FP INCONS 471
Total FP COMP 309
Total FN 609
Total Scores
MCC 0.717
Average MCC ± 95% Confidence Intervals 0.746 ± 0.059
Sensitivity 0.733
Positive Predictive Value 0.707
Nr of predictions 115

^top



2. Individual counts for MaxExpect [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 99 0 0 0 0 0
2KE6_A 0.94 0.94 0.94 17 449 2 0 1 1 1
2KPV_A - 0.96 0.92 1.00 12 201 0 0 0 0 1
2KRL_A - 0.91 0.87 0.95 20 2003 9 1 0 8 3
2KU0_A - -0.04 0.00 0.00 0 283 9 1 8 0 12
2KUR_A 0.95 0.95 0.95 18 448 1 0 1 0 1
2KUU_A 0.94 0.94 0.94 17 429 2 0 1 1 1
2KUV_A 0.94 0.95 0.95 18 420 1 0 1 0 1
2KUW_A 0.94 0.94 0.94 17 452 2 0 1 1 1
2KX8_A 1.00 1.00 1.00 16 355 0 0 0 0 0
2KXM_A - 0.93 0.88 1.00 7 142 0 0 0 0 1
2KZL_A - 1.00 1.00 1.00 13 507 1 0 0 1 0
2L1F_A 1.00 1.00 1.00 23 740 0 0 0 0 0
2L1F_B 1.00 1.00 1.00 24 767 0 0 0 0 0
2L2K_A - 1.00 1.00 1.00 17 332 0 0 0 0 0
2L3C_B - 1.00 1.00 1.00 14 225 0 0 0 0 0
2L3E_A - 1.00 1.00 1.00 12 228 1 0 0 1 0
2L3J_B 1.00 1.00 1.00 30 955 0 0 0 0 0
2L5Z_A - 1.00 1.00 1.00 9 102 0 0 0 0 0
2L94_A 1.00 1.00 1.00 18 339 1 0 0 1 0
2LA5_A - 0.77 0.71 0.83 5 269 3 0 1 2 2
2LBS_A - 1.00 1.00 1.00 14 193 0 0 0 0 0
2LC8_A -0.03 0.00 0.00 0 513 15 3 12 0 18
2LDL_A - 1.00 1.00 1.00 9 131 1 0 0 1 0
2LDT_A - 1.00 1.00 1.00 11 151 0 0 0 0 0
2LHP_A - 1.00 1.00 1.00 15 246 0 0 0 0 0
2LI4_A - 1.00 1.00 1.00 14 175 0 0 0 0 0
2LJJ_A - 1.00 1.00 1.00 7 123 3 0 0 3 0
2LK3_A - 1.00 1.00 1.00 9 91 0 0 0 0 0
2LKR_A - 0.84 0.83 0.86 24 2412 11 0 4 7 5
2LQZ_A - 1.00 1.00 1.00 8 124 2 0 0 2 0
2LWK_A - 1.00 1.00 1.00 11 196 1 0 0 1 0
2RRC_A - 0.83 0.71 1.00 5 70 0 0 0 0 2
2WRQ_Y 0.94 0.89 1.00 8 1144 14 0 0 14 1
2WW9_D - 0.08 0.10 0.08 1 729 15 2 9 4 9
2WW9_E - -0.03 0.00 0.00 0 168 5 0 4 1 5
2WW9_F - 0.64 0.75 0.60 6 102 4 1 3 0 2
2WWQ_V 0.97 0.95 1.00 18 1186 3 0 0 3 1
2XKV_B 0.64 0.73 0.57 8 1821 26 0 6 20 3
2XQD_Y 0.98 0.95 1.00 20 1109 1 0 0 1 1
2XXA_G 0.24 0.26 0.25 9 2009 28 1 26 1 26
2Y8W_B - 1.00 1.00 1.00 6 82 1 0 0 1 0
2Y9C_V - 0.79 0.83 0.75 15 886 8 0 5 3 3
2YIE_X - 0.53 0.57 0.50 4 536 6 1 3 2 3
2YIE_Z - -0.02 0.00 0.00 0 587 15 6 9 0 8
3A2K_C 0.47 0.50 0.46 11 1084 13 3 10 0 11
3ADB_C - 0.86 0.85 0.88 28 1787 4 0 4 0 5
3AKZ_H 1.00 1.00 1.00 20 1107 2 0 0 2 0
3AM1_B - 0.86 0.86 0.86 25 1437 4 0 4 0 4
3AMU_B 0.73 0.79 0.68 15 1135 10 0 7 3 4
3GX2_A 0.93 0.89 0.96 25 1423 2 1 0 1 3
3IAB_R - 1.00 1.00 1.00 12 373 0 0 0 0 0
3IVN_B 0.91 0.83 1.00 19 884 0 0 0 0 4
3IWN_A 0.80 0.79 0.81 22 1445 6 1 4 1 6
3IYQ_A 0.24 0.33 0.17 17 22342 96 41 40 15 34
3IZ4_A 0.61 0.61 0.62 58 25442 42 14 22 6 37
3IZF_C 0.89 0.91 0.86 32 2603 8 0 5 3 3
3J0L_g - -0.01 0.00 0.00 0 174 4 0 2 2 2
3J0L_7 - -0.02 0.00 0.00 0 507 12 3 9 0 10
3J0L_h - 0.93 0.88 1.00 28 2112 2 0 0 2 4
3J0L_1 - 0.83 0.77 0.91 10 473 4 0 1 3 3
3J0L_a - 0.39 0.36 0.44 4 402 6 4 1 1 7
3J0L_8 - 1.00 1.00 1.00 7 69 0 0 0 0 0
3J0L_2 - 0.30 0.31 0.31 8 2224 25 2 16 7 18
3J16_L 0.34 0.33 0.37 7 1140 12 0 12 0 14
3J20_0 0.53 0.57 0.50 12 1195 13 2 10 1 9
3J20_1 0.75 0.75 0.75 15 1092 6 0 5 1 5
3J2C_O - 0.74 0.71 0.78 29 3950 13 0 8 5 12
3J2C_M - 0.48 0.54 0.43 52 39804 88 29 40 19 44
3J2L_3 0.80 0.82 0.78 28 2984 12 1 7 4 6
3JYV_7 0.48 0.45 0.53 9 1094 9 3 5 1 11
3JYX_4 0.55 0.83 0.37 10 4729 34 14 3 17 2
3JYX_3 0.77 0.80 0.75 12 2362 24 0 4 20 3
3KTW_C - 0.72 0.72 0.72 18 1755 14 2 5 7 7
3LA5_A 0.91 0.84 1.00 21 933 0 0 0 0 4
3NDB_M - 0.94 0.93 0.95 41 3671 7 0 2 5 3
3NKB_B - 0.69 0.74 0.67 14 714 7 0 7 0 5
3NMU_E - -0.02 0.00 0.00 0 208 10 5 3 2 3
3NPB_A 0.77 0.73 0.82 27 2245 10 0 6 4 10
3O58_2 0.91 0.94 0.88 29 2721 11 0 4 7 2
3O58_3 0.40 0.50 0.33 11 4731 36 8 14 14 11
3OVB_D - 1.00 1.00 1.00 11 215 1 0 0 1 0
3OVS_D - 1.00 1.00 1.00 12 204 0 0 0 0 0
3P22_G - 1.00 1.00 1.00 11 301 0 0 0 0 0
3PDR_A 0.92 0.90 0.94 45 4792 5 1 2 2 5
3R4F_A - 1.00 1.00 1.00 22 884 1 0 0 1 0
3R9X_C - 1.00 1.00 1.00 8 221 2 0 0 2 0
3RKF_A 0.91 0.83 1.00 20 846 0 0 0 0 4
3SD1_A 0.70 0.69 0.71 20 1505 8 3 5 0 9
3SIU_F - 1.00 1.00 1.00 8 137 0 0 0 0 0
3SN2_B 1.00 1.00 1.00 12 142 0 0 0 0 0
3TRZ_Z - 1.00 1.00 1.00 5 87 1 0 0 1 0
3TS0_U - 1.00 1.00 1.00 6 112 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 103 0 0 0 0 0
3U4M_B - 0.58 0.59 0.59 13 1254 9 2 7 0 9
3UZL_B 1.00 1.00 1.00 16 1277 8 0 0 8 0
3VJR_D - 1.00 1.00 1.00 12 239 0 0 0 0 0
3W3S_B 0.97 0.94 1.00 31 1958 1 0 0 1 2
3ZEX_F - -0.01 0.00 0.00 0 908 11 2 4 5 4
3ZEX_C 0.54 0.45 0.65 13 5354 17 2 5 10 16
3ZEX_G - 0.98 0.96 1.00 43 6464 17 0 0 17 2
3ZEX_D 0.82 0.77 0.87 27 2765 11 0 4 7 8
3ZEX_E - -0.01 0.00 0.00 0 8252 54 21 31 2 34
3ZEX_H - 0.26 0.37 0.19 7 3589 29 18 11 0 12
4A1C_2 0.19 0.25 0.16 5 4484 41 10 17 14 15
4A1C_3 0.83 0.81 0.86 30 2728 7 0 5 2 7
4A4U_A - 1.00 1.00 1.00 9 96 0 0 0 0 0
4AOB_A 0.60 0.59 0.63 17 1410 11 4 6 1 12
4ATO_G - 0.45 0.43 0.50 3 214 3 3 0 0 4
4ENB_A 0.85 0.73 1.00 11 461 0 0 0 0 4
4ENC_A 0.36 0.33 0.42 5 484 8 0 7 1 10
4FNJ_A - 0.91 0.83 1.00 10 240 0 0 0 0 2
4FRG_B 0.29 0.25 0.35 6 1185 11 4 7 0 18
4FRN_A 0.51 0.46 0.57 13 1825 10 5 5 0 15
4HXH_A - 1.00 1.00 1.00 6 89 0 0 0 0 0

^top



Performance of RNAshapes - scored lower in this pairwise comparison

1. Total counts & total scores for RNAshapes

Total Base Pair Counts
Total TP 1663
Total TN 221691
Total FP 1047
Total FP CONTRA 215
Total FP INCONS 507
Total FP COMP 325
Total FN 622
Total Scores
MCC 0.709
Average MCC ± 95% Confidence Intervals 0.760 ± 0.055
Sensitivity 0.728
Positive Predictive Value 0.697
Nr of predictions 115

^top



2. Individual counts for RNAshapes [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 99 0 0 0 0 0
2KE6_A 1.00 1.00 1.00 18 449 1 0 0 1 0
2KPV_A - 0.96 0.92 1.00 12 201 0 0 0 0 1
2KRL_A - 0.91 0.87 0.95 20 2003 9 1 0 8 3
2KU0_A - 0.96 0.92 1.00 11 281 0 0 0 0 1
2KUR_A 1.00 1.00 1.00 19 448 0 0 0 0 0
2KUU_A 1.00 1.00 1.00 18 429 1 0 0 1 0
2KUV_A 1.00 1.00 1.00 19 420 0 0 0 0 0
2KUW_A 1.00 1.00 1.00 18 452 1 0 0 1 0
2KX8_A 0.97 0.94 1.00 15 356 0 0 0 0 1
2KXM_A - 0.93 0.88 1.00 7 142 0 0 0 0 1
2KZL_A - 1.00 1.00 1.00 13 507 1 0 0 1 0
2L1F_A 0.89 0.87 0.91 20 741 2 0 2 0 3
2L1F_B 0.98 0.96 1.00 23 768 0 0 0 0 1
2L2K_A - 0.97 0.94 1.00 16 333 0 0 0 0 1
2L3C_B - 1.00 1.00 1.00 14 225 0 0 0 0 0
2L3E_A - 1.00 1.00 1.00 12 228 1 0 0 1 0
2L3J_B 0.98 0.97 1.00 29 956 0 0 0 0 1
2L5Z_A - 1.00 1.00 1.00 9 102 0 0 0 0 0
2L94_A 1.00 1.00 1.00 18 339 1 0 0 1 0
2LA5_A - 0.77 0.71 0.83 5 269 3 0 1 2 2
2LBS_A - 1.00 1.00 1.00 14 193 0 0 0 0 0
2LC8_A 0.64 0.61 0.69 11 512 5 0 5 0 7
2LDL_A - 1.00 1.00 1.00 9 131 1 0 0 1 0
2LDT_A - 1.00 1.00 1.00 11 151 0 0 0 0 0
2LHP_A - 1.00 1.00 1.00 15 246 0 0 0 0 0
2LI4_A - 1.00 1.00 1.00 14 175 0 0 0 0 0
2LJJ_A - 1.00 1.00 1.00 7 123 3 0 0 3 0
2LK3_A - 1.00 1.00 1.00 9 91 0 0 0 0 0
2LKR_A - 1.00 1.00 1.00 29 2411 6 0 0 6 0
2LQZ_A - 1.00 1.00 1.00 8 124 2 0 0 2 0
2LWK_A - 0.95 0.91 1.00 10 197 1 0 0 1 1
2RRC_A - 0.83 0.71 1.00 5 70 0 0 0 0 2
2WRQ_Y 1.00 1.00 1.00 9 1143 13 0 0 13 0
2WW9_D - 0.11 0.10 0.14 1 734 11 2 4 5 9
2WW9_E - 0.89 0.80 1.00 4 168 0 0 0 0 1
2WW9_F - 0.68 0.75 0.67 6 103 4 1 2 1 2
2WWQ_V 1.00 1.00 1.00 19 1185 5 0 0 5 0
2XKV_B 0.64 0.73 0.57 8 1821 24 0 6 18 3
2XQD_Y 0.70 0.67 0.74 14 1110 6 0 5 1 7
2XXA_G 1.00 1.00 1.00 35 2010 1 0 0 1 0
2Y8W_B - 1.00 1.00 1.00 6 82 1 0 0 1 0
2Y9C_V - 0.79 0.83 0.75 15 886 8 0 5 3 3
2YIE_X - 0.33 0.43 0.27 3 533 11 3 5 3 4
2YIE_Z - -0.02 0.00 0.00 0 587 15 6 9 0 8
3A2K_C 0.47 0.50 0.46 11 1084 13 3 10 0 11
3ADB_C - 0.71 0.70 0.74 23 1788 8 0 8 0 10
3AKZ_H 0.78 0.80 0.76 16 1106 7 2 3 2 4
3AM1_B - 0.71 0.72 0.70 21 1436 9 3 6 0 8
3AMU_B 0.81 0.79 0.83 15 1139 6 0 3 3 4
3GX2_A 0.91 0.89 0.93 25 1422 3 1 1 1 3
3IAB_R - 1.00 1.00 1.00 12 373 0 0 0 0 0
3IVN_B 0.91 0.83 1.00 19 884 0 0 0 0 4
3IWN_A 0.83 0.79 0.88 22 1447 3 1 2 0 6
3IYQ_A 0.19 0.27 0.14 14 22340 95 43 43 9 37
3IZ4_A 0.57 0.58 0.56 55 25437 50 16 28 6 40
3IZF_C 0.76 0.77 0.75 27 2604 14 0 9 5 8
3J0L_g - -0.01 0.00 0.00 0 174 4 0 2 2 2
3J0L_7 - -0.02 0.00 0.00 0 507 12 2 10 0 10
3J0L_h - 0.87 0.81 0.93 26 2112 4 1 1 2 6
3J0L_1 - 0.83 0.77 0.91 10 473 4 0 1 3 3
3J0L_a - 0.18 0.18 0.22 2 402 8 3 4 1 9
3J0L_8 - 1.00 1.00 1.00 7 69 0 0 0 0 0
3J0L_2 - 0.46 0.46 0.48 12 2225 20 1 12 7 14
3J16_L 0.59 0.57 0.63 12 1140 7 0 7 0 9
3J20_0 0.51 0.57 0.48 12 1194 14 3 10 1 9
3J20_1 1.00 1.00 1.00 20 1092 1 0 0 1 0
3J2C_O - 0.43 0.44 0.44 18 3946 27 3 20 4 23
3J2C_M - 0.61 0.65 0.58 62 39818 68 17 28 23 34
3J2L_3 0.77 0.79 0.75 27 2984 14 1 8 5 7
3JYV_7 -0.02 0.00 0.00 0 1092 20 1 18 1 20
3JYX_4 0.61 0.83 0.45 10 4734 35 10 2 23 2
3JYX_3 0.77 0.80 0.75 12 2362 26 0 4 22 3
3KTW_C - 0.57 0.60 0.56 15 1753 19 2 10 7 10
3LA5_A 0.89 0.80 1.00 20 934 0 0 0 0 5
3NDB_M - 0.98 0.98 0.98 43 3670 5 0 1 4 1
3NKB_B - 0.75 0.74 0.78 14 717 6 0 4 2 5
3NMU_E - 1.00 1.00 1.00 3 213 5 0 0 5 0
3NPB_A 0.84 0.76 0.93 28 2248 5 1 1 3 9
3O58_2 0.73 0.74 0.72 23 2722 18 0 9 9 8
3O58_3 0.41 0.50 0.34 11 4732 38 6 15 17 11
3OVB_D - 1.00 1.00 1.00 11 215 1 0 0 1 0
3OVS_D - 1.00 1.00 1.00 12 204 0 0 0 0 0
3P22_G - 1.00 1.00 1.00 11 301 0 0 0 0 0
3PDR_A 0.80 0.80 0.80 40 4790 12 3 7 2 10
3R4F_A - 1.00 1.00 1.00 22 884 1 0 0 1 0
3R9X_C - 1.00 1.00 1.00 8 221 2 0 0 2 0
3RKF_A 0.91 0.83 1.00 20 846 0 0 0 0 4
3SD1_A 0.77 0.76 0.79 22 1505 6 4 2 0 7
3SIU_F - 1.00 1.00 1.00 8 137 0 0 0 0 0
3SN2_B 0.95 0.92 1.00 11 143 0 0 0 0 1
3TRZ_Z - 1.00 1.00 1.00 5 87 1 0 0 1 0
3TS0_U - 1.00 1.00 1.00 6 112 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 103 0 0 0 0 0
3U4M_B - 0.49 0.55 0.46 12 1250 14 3 11 0 10
3UZL_B 0.43 0.50 0.38 8 1272 17 4 9 4 8
3VJR_D - 1.00 1.00 1.00 12 239 0 0 0 0 0
3W3S_B 0.60 0.58 0.63 19 1959 12 2 9 1 14
3ZEX_F - -0.01 0.00 0.00 0 908 11 2 4 5 4
3ZEX_C 0.29 0.34 0.26 10 5335 42 5 24 13 19
3ZEX_G - 0.66 0.67 0.67 30 6462 27 4 11 12 15
3ZEX_D 0.86 0.80 0.93 28 2766 6 0 2 4 7
3ZEX_E - -0.01 0.00 0.00 0 8247 59 20 37 2 34
3ZEX_H - 0.28 0.37 0.22 7 3593 32 11 14 7 12
4A1C_2 0.19 0.25 0.16 5 4484 41 11 16 14 15
4A1C_3 0.86 0.84 0.89 31 2728 7 0 4 3 6
4A4U_A - 1.00 1.00 1.00 9 96 0 0 0 0 0
4AOB_A 0.60 0.59 0.63 17 1410 11 4 6 1 12
4ATO_G - 0.38 0.43 0.38 3 212 5 5 0 0 4
4ENB_A 0.85 0.73 1.00 11 461 2 0 0 2 4
4ENC_A 0.37 0.33 0.45 5 485 7 0 6 1 10
4FNJ_A - 0.91 0.83 1.00 10 240 0 0 0 0 2
4FRG_B 0.36 0.38 0.38 9 1178 15 2 13 0 15
4FRN_A 0.59 0.57 0.62 16 1822 10 2 8 0 12
4HXH_A - 1.00 1.00 1.00 6 89 0 0 0 0 0

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.