CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of MaxExpect - scored higher in this pairwise comparison

  4. Performance of RNAfold - scored lower in this pairwise comparison

  5. Compile and download dataset for MaxExpect & RNAfold [.zip] - may take several seconds...


Overview

Metric MaxExpect RNAfold
MCC 0.552 > 0.519
Average MCC ± 95% Confidence Intervals 0.661 ± 0.055 < 0.669 ± 0.053
Sensitivity 0.480 > 0.459
Positive Predictive Value 0.635 > 0.587
Total TP 2742 > 2623
Total TN 6831037 > 6830882
Total FP 1738 < 2014
Total FP CONTRA 156 < 180
Total FP INCONS 1417 < 1667
Total FP COMP 165 < 167
Total FN 2968 < 3087
P-value 3.56938820447e-08

^top




Performance plots


  1. Comparison of performance of MaxExpect and RNAfold. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for MaxExpect and RNAfold).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for MaxExpect and RNAfold).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for MaxExpect and RNAfold. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for MaxExpect and RNAfold).

^top





Performance of MaxExpect - scored higher in this pairwise comparison

1. Total counts & total scores for MaxExpect

Total Base Pair Counts
Total TP 2742
Total TN 6831037
Total FP 1738
Total FP CONTRA 156
Total FP INCONS 1417
Total FP COMP 165
Total FN 2968
Total Scores
MCC 0.552
Average MCC ± 95% Confidence Intervals 0.661 ± 0.055
Sensitivity 0.480
Positive Predictive Value 0.635
Nr of predictions 118

^top



2. Individual counts for MaxExpect [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
2KE6_A 0.92 0.89 0.94 17 1110 2 0 1 1 2
2KPV_A - 0.96 0.92 1.00 12 549 0 0 0 0 1
2KRL_A - 0.76 0.60 0.96 24 5126 5 0 1 4 16
2KU0_A - -0.02 0.00 0.00 0 694 9 0 9 0 16
2KUR_A 0.90 0.86 0.95 18 1109 1 0 1 0 3
2KUU_A 0.87 0.81 0.94 17 1110 2 0 1 1 4
2KUV_A 0.88 0.82 0.95 18 1109 1 0 1 0 4
2KUW_A 0.90 0.86 0.95 18 1109 1 0 1 0 3
2KX8_A 0.94 0.89 1.00 16 845 0 0 0 0 2
2KXM_A - 0.83 0.70 1.00 7 344 0 0 0 0 3
2KZL_A - 0.86 0.74 1.00 14 1471 0 0 0 0 5
2L1F_B 0.98 0.96 1.00 24 2121 0 0 0 0 1
2L1F_A 0.98 0.96 1.00 23 2057 0 0 0 0 1
2L2K_A - 0.94 0.89 1.00 17 844 0 0 0 0 2
2L3C_B - 0.91 0.82 1.00 14 547 0 0 0 0 3
2L3E_A - 0.96 0.92 1.00 12 583 1 0 0 1 1
2L3J_B 0.94 0.88 1.00 30 2455 0 0 0 0 4
2L5Z_A - 0.86 0.75 1.00 9 316 0 0 0 0 3
2L94_A 0.97 0.95 1.00 19 971 0 0 0 0 1
2LA5_A - 0.39 0.26 0.63 5 622 3 1 2 0 14
2LBS_A - 1.00 1.00 1.00 14 482 0 0 0 0 0
2LC8_A -0.01 0.00 0.00 0 1525 15 2 13 0 20
2LDL_A - 0.90 0.82 1.00 9 342 1 0 0 1 2
2LDT_A - 0.85 0.73 1.00 11 454 0 0 0 0 4
2LHP_A - 0.97 0.94 1.00 15 651 0 0 0 0 1
2LI4_A - 0.93 0.88 1.00 14 482 0 0 0 0 2
2LJJ_A - 1.00 1.00 1.00 8 343 2 0 0 2 0
2LK3_A - 0.95 0.90 1.00 9 267 0 0 0 0 1
2LKR_A - 0.81 0.74 0.88 29 6072 6 0 4 2 10
2LQZ_A - 0.85 0.82 0.90 9 341 1 1 0 0 2
2LWK_A - 0.88 0.85 0.92 11 484 1 0 1 0 2
2RRC_A - 0.70 0.50 1.00 5 226 0 0 0 0 5
2WRQ_Y 0.53 0.53 0.53 9 2833 13 5 3 5 8
2WW9_F - 0.69 0.70 0.70 7 290 3 0 3 0 3
2WW9_E - -0.01 0.00 0.00 0 556 5 0 5 0 14
2WW9_D - 0.14 0.12 0.19 3 1937 13 0 13 0 23
2WWQ_V 0.76 0.64 0.90 18 2906 3 0 2 1 10
2XKV_B 0.51 0.50 0.53 10 4541 24 0 9 15 10
2XQD_Y 0.88 0.78 1.00 21 2829 0 0 0 0 6
2XXA_G 0.25 0.24 0.27 10 5114 27 1 26 0 32
2Y8W_B - 1.00 1.00 1.00 6 184 1 0 0 1 0
2Y9C_V - 0.65 0.63 0.68 15 2256 8 1 6 1 9
2YIE_X - 0.36 0.33 0.40 4 1368 6 2 4 0 8
2YIE_Z - -0.01 0.00 0.00 0 1525 15 3 12 0 12
3A2K_C 0.42 0.39 0.46 11 2902 13 2 11 0 17
3ADB_C - 0.80 0.74 0.88 28 4154 4 0 4 0 10
3AKZ_H 0.84 0.75 0.95 21 2679 1 1 0 0 7
3AM1_B - 0.78 0.71 0.86 25 3211 4 0 4 0 10
3AMU_B 0.64 0.59 0.70 16 2980 9 0 7 2 11
3GX2_A 0.77 0.63 0.96 25 4345 2 0 1 1 15
3IAB_R - 0.92 0.86 1.00 12 1023 0 0 0 0 2
3IVN_B 0.78 0.61 1.00 19 2327 0 0 0 0 12
3IWN_A 0.74 0.67 0.81 22 4251 6 0 5 1 11
3IYQ_A 0.29 0.31 0.27 29 60617 84 19 61 4 65
3IZ4_A 0.54 0.46 0.64 61 70781 39 3 31 5 71
3IZF_C 0.71 0.61 0.83 33 6863 7 1 6 0 21
3J0L_1 - 0.76 0.63 0.92 12 1212 2 0 1 1 7
3J0L_8 - 0.93 0.88 1.00 7 183 0 0 0 0 1
3J0L_a - 0.39 0.31 0.50 5 1118 5 2 3 0 11
3J0L_h - 0.81 0.65 1.00 28 6077 2 0 0 2 15
3J0L_7 - -0.01 0.00 0.00 0 1213 12 0 12 0 17
3J0L_g - -0.01 0.00 0.00 0 461 4 1 3 0 4
3J0L_2 - 0.25 0.24 0.26 8 6185 25 2 21 2 25
3J16_L 0.29 0.23 0.37 7 2756 12 0 12 0 23
3J20_2 0.49 0.44 0.56 276 1116272 221 10 207 4 357
3J20_1 0.73 0.70 0.76 16 2905 5 0 5 0 7
3J20_0 0.44 0.40 0.50 12 2826 13 0 12 1 18
3J2C_M - 0.38 0.31 0.47 64 106354 76 10 63 3 143
3J2C_O - 0.63 0.51 0.78 32 10255 10 0 9 1 31
3J2L_3 0.62 0.53 0.74 28 7837 12 0 10 2 25
3JYV_7 0.41 0.31 0.56 10 2832 8 0 8 0 22
3JYX_3 0.64 0.63 0.65 17 6302 19 1 8 10 10
3JYX_4 0.31 0.30 0.31 10 12214 34 4 18 12 23
3KIY_A - 0.54 0.45 0.65 553 4053271 314 23 281 10 688
3KTW_C - 0.54 0.47 0.65 20 4529 12 2 9 1 23
3LA5_A 0.78 0.62 1.00 21 2464 0 0 0 0 13
3NDB_M - 0.78 0.69 0.89 42 9133 6 0 5 1 19
3NKB_B - 0.59 0.54 0.67 14 1995 7 0 7 0 12
3NMU_E - -0.01 0.00 0.00 0 553 10 1 7 2 6
3NPB_A 0.71 0.61 0.82 28 6987 9 0 6 3 18
3O58_2 0.75 0.76 0.74 29 7221 11 3 7 1 9
3O58_3 0.35 0.34 0.35 12 12369 35 2 20 13 23
3OVB_D - 1.00 1.00 1.00 11 584 1 0 0 1 0
3OVS_D - 0.96 0.92 1.00 12 549 0 0 0 0 1
3P22_G - 0.96 0.92 1.00 11 730 0 0 0 0 1
3PDR_A 0.76 0.63 0.94 45 12832 5 0 3 2 27
3R4F_A - 0.96 0.92 1.00 23 2122 0 0 0 0 2
3R9X_C - 0.84 0.80 0.89 8 586 2 0 1 1 2
3RKF_A 0.76 0.59 1.00 20 2191 0 0 0 0 14
3SD1_A 0.58 0.48 0.71 20 3888 8 1 7 0 22
3SIU_F - 0.85 0.73 1.00 8 370 0 0 0 0 3
3SN2_B 1.00 1.00 1.00 12 394 0 0 0 0 0
3TRZ_Z - 0.91 0.83 1.00 5 205 1 0 0 1 1
3TS0_U - 1.00 1.00 1.00 6 247 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 271 0 0 0 0 0
3U4M_B - 0.45 0.35 0.59 13 3138 9 1 8 0 24
3UZL_B 0.77 0.62 0.96 23 3546 1 0 1 0 14
3VJR_D - 0.96 0.92 1.00 12 618 0 0 0 0 1
3W3S_B 0.88 0.78 1.00 31 4722 1 0 0 1 9
3ZEX_G - 0.77 0.66 0.91 49 16417 11 0 5 6 25
3ZEX_D 0.69 0.61 0.79 30 6983 8 1 7 0 19
3ZEX_B - 0.30 0.26 0.35 147 1071957 289 29 247 13 411
3ZEX_E - 0.00 0.00 0.00 0 21893 54 2 50 2 77
3ZEX_C 0.41 0.27 0.64 14 14174 16 2 6 8 38
3ZEX_F - 0.00 0.00 0.00 0 2617 11 2 9 0 12
3ZEX_H - 0.19 0.18 0.19 7 9009 29 5 24 0 31
4A1C_2 0.14 0.15 0.14 5 11746 41 4 26 11 28
4A1C_3 0.69 0.57 0.84 31 7103 6 0 6 0 23
4A4U_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
4AOB_A 0.50 0.40 0.63 17 4344 11 2 8 1 25
4ATO_G - 0.38 0.30 0.50 3 522 3 0 3 0 7
4ENB_A 0.76 0.58 1.00 11 1264 0 0 0 0 8
4ENC_A 0.31 0.26 0.38 5 1313 8 1 7 0 14
4FNJ_A - 0.79 0.63 1.00 10 585 0 0 0 0 6
4FRG_B 0.30 0.22 0.41 7 3469 10 1 9 0 25
4FRN_A 0.45 0.36 0.57 13 5128 10 2 8 0 23
4HXH_A - 1.00 1.00 1.00 6 319 0 0 0 0 0

^top



Performance of RNAfold - scored lower in this pairwise comparison

1. Total counts & total scores for RNAfold

Total Base Pair Counts
Total TP 2623
Total TN 6830882
Total FP 2014
Total FP CONTRA 180
Total FP INCONS 1667
Total FP COMP 167
Total FN 3087
Total Scores
MCC 0.519
Average MCC ± 95% Confidence Intervals 0.669 ± 0.053
Sensitivity 0.459
Positive Predictive Value 0.587
Nr of predictions 118

^top



2. Individual counts for RNAfold [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
2KE6_A 0.97 0.95 1.00 18 1110 1 0 0 1 1
2KPV_A - 0.96 0.92 1.00 12 549 0 0 0 0 1
2KRL_A - 0.76 0.60 0.96 24 5126 5 0 1 4 16
2KU0_A - 0.86 0.75 1.00 12 691 0 0 0 0 4
2KUR_A 0.95 0.90 1.00 19 1109 0 0 0 0 2
2KUU_A 0.92 0.86 1.00 18 1110 1 0 0 1 3
2KUV_A 0.93 0.86 1.00 19 1109 0 0 0 0 3
2KUW_A 0.95 0.90 1.00 19 1109 0 0 0 0 2
2KX8_A 0.94 0.89 1.00 16 845 0 0 0 0 2
2KXM_A - 0.89 0.80 1.00 8 343 0 0 0 0 2
2KZL_A - 0.86 0.74 1.00 14 1471 0 0 0 0 5
2L1F_B 0.98 0.96 1.00 24 2121 0 0 0 0 1
2L1F_A 0.98 0.96 1.00 23 2057 0 0 0 0 1
2L2K_A - 0.94 0.89 1.00 17 844 0 0 0 0 2
2L3C_B - 0.91 0.82 1.00 14 547 0 0 0 0 3
2L3E_A - 0.96 0.92 1.00 12 583 1 0 0 1 1
2L3J_B 0.94 0.88 1.00 30 2455 0 0 0 0 4
2L5Z_A - 0.86 0.75 1.00 9 316 0 0 0 0 3
2L94_A 0.97 0.95 1.00 19 971 0 0 0 0 1
2LA5_A - 0.39 0.26 0.63 5 622 3 1 2 0 14
2LBS_A - 1.00 1.00 1.00 14 482 0 0 0 0 0
2LC8_A 0.61 0.55 0.69 11 1524 5 0 5 0 9
2LDL_A - 0.90 0.82 1.00 9 342 1 0 0 1 2
2LDT_A - 0.85 0.73 1.00 11 454 0 0 0 0 4
2LHP_A - 0.97 0.94 1.00 15 651 0 0 0 0 1
2LI4_A - 0.93 0.88 1.00 14 482 0 0 0 0 2
2LJJ_A - 1.00 1.00 1.00 8 343 2 0 0 2 0
2LK3_A - 0.95 0.90 1.00 9 267 0 0 0 0 1
2LKR_A - 0.93 0.87 1.00 34 6071 3 0 0 3 5
2LQZ_A - 0.85 0.82 0.90 9 341 1 1 0 0 2
2LWK_A - 0.83 0.77 0.91 10 485 1 0 1 0 3
2RRC_A - 0.83 0.70 1.00 7 224 0 0 0 0 3
2WRQ_Y 0.57 0.59 0.56 10 2832 13 5 3 5 7
2WW9_F - 0.59 0.60 0.60 6 290 4 0 4 0 4
2WW9_E - 0.53 0.29 1.00 4 557 0 0 0 0 10
2WW9_D - 0.14 0.12 0.19 3 1937 13 0 13 0 23
2WWQ_V 0.76 0.68 0.86 19 2904 5 0 3 2 9
2XKV_B 0.51 0.50 0.53 10 4541 23 0 9 14 10
2XQD_Y 0.88 0.78 1.00 21 2829 0 0 0 0 6
2XXA_G 0.34 0.31 0.39 13 5118 20 1 19 0 29
2Y8W_B - 1.00 1.00 1.00 6 184 1 0 0 1 0
2Y9C_V - 0.65 0.63 0.68 15 2256 8 1 6 1 9
2YIE_X - 0.25 0.25 0.27 3 1367 11 1 7 3 9
2YIE_Z - -0.01 0.00 0.00 0 1525 15 3 12 0 12
3A2K_C 0.42 0.39 0.46 11 2902 13 2 11 0 17
3ADB_C - 0.33 0.32 0.35 12 4152 22 0 22 0 26
3AKZ_H 0.43 0.39 0.48 11 2678 12 2 10 0 17
3AM1_B - 0.68 0.63 0.73 22 3210 8 1 7 0 13
3AMU_B 0.70 0.59 0.84 16 2984 5 0 3 2 11
3GX2_A 0.68 0.55 0.85 22 4345 5 0 4 1 18
3IAB_R - 0.92 0.86 1.00 12 1023 0 0 0 0 2
3IVN_B 0.78 0.61 1.00 19 2327 0 0 0 0 12
3IWN_A 0.63 0.58 0.70 19 4251 9 0 8 1 14
3IYQ_A 0.31 0.34 0.29 32 60616 80 24 54 2 62
3IZ4_A 0.51 0.47 0.55 62 70763 52 10 41 1 70
3IZF_C 0.59 0.52 0.67 28 6861 14 1 13 0 26
3J0L_1 - 0.76 0.63 0.92 12 1212 2 0 1 1 7
3J0L_8 - 0.93 0.88 1.00 7 183 0 0 0 0 1
3J0L_a - 0.23 0.19 0.30 3 1118 7 1 6 0 13
3J0L_h - 0.75 0.60 0.93 26 6077 4 0 2 2 17
3J0L_7 - -0.01 0.00 0.00 0 1213 12 0 12 0 17
3J0L_g - -0.01 0.00 0.00 0 460 5 1 4 0 4
3J0L_2 - 0.25 0.24 0.26 8 6185 25 2 21 2 25
3J16_L 0.75 0.57 1.00 17 2758 0 0 0 0 13
3J20_2 0.46 0.41 0.53 258 1116279 231 14 214 3 375
3J20_1 0.96 0.91 1.00 21 2905 2 0 0 2 2
3J20_0 0.43 0.40 0.48 12 2825 14 0 13 1 18
3J2C_M - 0.51 0.41 0.63 85 106355 54 8 43 3 122
3J2C_O - 0.64 0.54 0.76 34 10251 12 0 11 1 29
3J2L_3 0.62 0.53 0.74 28 7837 12 0 10 2 25
3JYV_7 -0.01 0.00 0.00 0 2828 22 1 21 0 32
3JYX_3 0.62 0.63 0.61 17 6300 22 1 10 11 10
3JYX_4 0.31 0.30 0.32 10 12215 33 5 16 12 23
3KIY_A - 0.41 0.35 0.48 431 4053222 484 22 453 9 810
3KTW_C - 0.73 0.63 0.84 27 4528 6 2 3 1 16
3LA5_A 0.78 0.62 1.00 21 2464 0 0 0 0 13
3NDB_M - 0.77 0.69 0.86 42 9131 8 0 7 1 19
3NKB_B - 0.59 0.54 0.67 14 1995 7 0 7 0 12
3NMU_E - 0.54 0.50 0.60 3 556 5 0 2 3 3
3NPB_A 0.83 0.72 0.97 33 6987 4 0 1 3 13
3O58_2 0.59 0.61 0.58 23 7220 19 4 13 2 15
3O58_3 0.34 0.34 0.34 12 12368 34 2 21 11 23
3OVB_D - 1.00 1.00 1.00 11 584 1 0 0 1 0
3OVS_D - 0.96 0.92 1.00 12 549 0 0 0 0 1
3P22_G - 0.96 0.92 1.00 11 730 0 0 0 0 1
3PDR_A 0.75 0.63 0.90 45 12830 7 1 4 2 27
3R4F_A - 0.96 0.92 1.00 23 2122 0 0 0 0 2
3R9X_C - 0.84 0.80 0.89 8 586 2 0 1 1 2
3RKF_A 0.76 0.59 1.00 20 2191 0 0 0 0 14
3SD1_A 0.64 0.52 0.79 22 3888 6 1 5 0 20
3SIU_F - 0.85 0.73 1.00 8 370 0 0 0 0 3
3SN2_B 0.96 0.92 1.00 11 395 0 0 0 0 1
3TRZ_Z - 0.91 0.83 1.00 5 205 1 0 0 1 1
3TS0_U - 1.00 1.00 1.00 6 247 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 271 0 0 0 0 0
3U4M_B - 0.38 0.32 0.46 12 3134 14 0 14 0 25
3UZL_B 0.48 0.38 0.61 14 3547 9 0 9 0 23
3VJR_D - 0.96 0.92 1.00 12 618 0 0 0 0 1
3W3S_B 0.55 0.50 0.61 20 4720 14 1 12 1 20
3ZEX_G - 0.66 0.57 0.76 42 16416 18 0 13 5 32
3ZEX_D 0.77 0.65 0.91 32 6986 3 0 3 0 17
3ZEX_B - 0.30 0.27 0.33 151 1071922 318 35 272 11 407
3ZEX_E - 0.00 0.00 0.00 0 21893 54 4 48 2 77
3ZEX_C 0.24 0.21 0.28 11 14157 42 1 27 14 41
3ZEX_F - 0.00 0.00 0.00 0 2617 11 2 9 0 12
3ZEX_H - 0.18 0.18 0.18 7 9006 32 6 26 0 31
4A1C_2 0.14 0.15 0.14 5 11744 43 5 27 11 28
4A1C_3 0.70 0.59 0.82 32 7101 7 1 6 0 22
4A4U_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
4AOB_A 0.50 0.40 0.63 17 4344 11 2 8 1 25
4ATO_G - 0.32 0.30 0.38 3 520 5 1 4 0 7
4ENB_A 0.70 0.58 0.85 11 1262 2 1 1 0 8
4ENC_A 0.32 0.26 0.42 5 1314 7 1 6 0 14
4FNJ_A - 0.79 0.63 1.00 10 585 0 0 0 0 6
4FRG_B 0.32 0.28 0.38 9 3462 15 0 15 0 23
4FRN_A 0.52 0.44 0.62 16 5125 10 2 8 0 20
4HXH_A - 1.00 1.00 1.00 6 319 0 0 0 0 0

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.