CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of MaxExpect - scored higher in this pairwise comparison

  4. Performance of UNAFold - scored lower in this pairwise comparison

  5. Compile and download dataset for MaxExpect & UNAFold [.zip] - may take several seconds...


Overview

Metric MaxExpect UNAFold
MCC 0.552 > 0.527
Average MCC ± 95% Confidence Intervals 0.661 ± 0.055 < 0.672 ± 0.051
Sensitivity 0.480 > 0.463
Positive Predictive Value 0.635 > 0.600
Total TP 2742 > 2645
Total TN 6831037 > 6830945
Total FP 1738 < 1927
Total FP CONTRA 156 < 174
Total FP INCONS 1417 < 1588
Total FP COMP 165 = 165
Total FN 2968 < 3065
P-value 1.02208072244e-07

^top




Performance plots


  1. Comparison of performance of MaxExpect and UNAFold. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for MaxExpect and UNAFold).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for MaxExpect and UNAFold).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for MaxExpect and UNAFold. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for MaxExpect and UNAFold).

^top





Performance of MaxExpect - scored higher in this pairwise comparison

1. Total counts & total scores for MaxExpect

Total Base Pair Counts
Total TP 2742
Total TN 6831037
Total FP 1738
Total FP CONTRA 156
Total FP INCONS 1417
Total FP COMP 165
Total FN 2968
Total Scores
MCC 0.552
Average MCC ± 95% Confidence Intervals 0.661 ± 0.055
Sensitivity 0.480
Positive Predictive Value 0.635
Nr of predictions 118

^top



2. Individual counts for MaxExpect [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
2KE6_A 0.92 0.89 0.94 17 1110 2 0 1 1 2
2KPV_A - 0.96 0.92 1.00 12 549 0 0 0 0 1
2KRL_A - 0.76 0.60 0.96 24 5126 5 0 1 4 16
2KU0_A - -0.02 0.00 0.00 0 694 9 0 9 0 16
2KUR_A 0.90 0.86 0.95 18 1109 1 0 1 0 3
2KUU_A 0.87 0.81 0.94 17 1110 2 0 1 1 4
2KUV_A 0.88 0.82 0.95 18 1109 1 0 1 0 4
2KUW_A 0.90 0.86 0.95 18 1109 1 0 1 0 3
2KX8_A 0.94 0.89 1.00 16 845 0 0 0 0 2
2KXM_A - 0.83 0.70 1.00 7 344 0 0 0 0 3
2KZL_A - 0.86 0.74 1.00 14 1471 0 0 0 0 5
2L1F_B 0.98 0.96 1.00 24 2121 0 0 0 0 1
2L1F_A 0.98 0.96 1.00 23 2057 0 0 0 0 1
2L2K_A - 0.94 0.89 1.00 17 844 0 0 0 0 2
2L3C_B - 0.91 0.82 1.00 14 547 0 0 0 0 3
2L3E_A - 0.96 0.92 1.00 12 583 1 0 0 1 1
2L3J_B 0.94 0.88 1.00 30 2455 0 0 0 0 4
2L5Z_A - 0.86 0.75 1.00 9 316 0 0 0 0 3
2L94_A 0.97 0.95 1.00 19 971 0 0 0 0 1
2LA5_A - 0.39 0.26 0.63 5 622 3 1 2 0 14
2LBS_A - 1.00 1.00 1.00 14 482 0 0 0 0 0
2LC8_A -0.01 0.00 0.00 0 1525 15 2 13 0 20
2LDL_A - 0.90 0.82 1.00 9 342 1 0 0 1 2
2LDT_A - 0.85 0.73 1.00 11 454 0 0 0 0 4
2LHP_A - 0.97 0.94 1.00 15 651 0 0 0 0 1
2LI4_A - 0.93 0.88 1.00 14 482 0 0 0 0 2
2LJJ_A - 1.00 1.00 1.00 8 343 2 0 0 2 0
2LK3_A - 0.95 0.90 1.00 9 267 0 0 0 0 1
2LKR_A - 0.81 0.74 0.88 29 6072 6 0 4 2 10
2LQZ_A - 0.85 0.82 0.90 9 341 1 1 0 0 2
2LWK_A - 0.88 0.85 0.92 11 484 1 0 1 0 2
2RRC_A - 0.70 0.50 1.00 5 226 0 0 0 0 5
2WRQ_Y 0.53 0.53 0.53 9 2833 13 5 3 5 8
2WW9_F - 0.69 0.70 0.70 7 290 3 0 3 0 3
2WW9_E - -0.01 0.00 0.00 0 556 5 0 5 0 14
2WW9_D - 0.14 0.12 0.19 3 1937 13 0 13 0 23
2WWQ_V 0.76 0.64 0.90 18 2906 3 0 2 1 10
2XKV_B 0.51 0.50 0.53 10 4541 24 0 9 15 10
2XQD_Y 0.88 0.78 1.00 21 2829 0 0 0 0 6
2XXA_G 0.25 0.24 0.27 10 5114 27 1 26 0 32
2Y8W_B - 1.00 1.00 1.00 6 184 1 0 0 1 0
2Y9C_V - 0.65 0.63 0.68 15 2256 8 1 6 1 9
2YIE_X - 0.36 0.33 0.40 4 1368 6 2 4 0 8
2YIE_Z - -0.01 0.00 0.00 0 1525 15 3 12 0 12
3A2K_C 0.42 0.39 0.46 11 2902 13 2 11 0 17
3ADB_C - 0.80 0.74 0.88 28 4154 4 0 4 0 10
3AKZ_H 0.84 0.75 0.95 21 2679 1 1 0 0 7
3AM1_B - 0.78 0.71 0.86 25 3211 4 0 4 0 10
3AMU_B 0.64 0.59 0.70 16 2980 9 0 7 2 11
3GX2_A 0.77 0.63 0.96 25 4345 2 0 1 1 15
3IAB_R - 0.92 0.86 1.00 12 1023 0 0 0 0 2
3IVN_B 0.78 0.61 1.00 19 2327 0 0 0 0 12
3IWN_A 0.74 0.67 0.81 22 4251 6 0 5 1 11
3IYQ_A 0.29 0.31 0.27 29 60617 84 19 61 4 65
3IZ4_A 0.54 0.46 0.64 61 70781 39 3 31 5 71
3IZF_C 0.71 0.61 0.83 33 6863 7 1 6 0 21
3J0L_1 - 0.76 0.63 0.92 12 1212 2 0 1 1 7
3J0L_8 - 0.93 0.88 1.00 7 183 0 0 0 0 1
3J0L_a - 0.39 0.31 0.50 5 1118 5 2 3 0 11
3J0L_h - 0.81 0.65 1.00 28 6077 2 0 0 2 15
3J0L_7 - -0.01 0.00 0.00 0 1213 12 0 12 0 17
3J0L_g - -0.01 0.00 0.00 0 461 4 1 3 0 4
3J0L_2 - 0.25 0.24 0.26 8 6185 25 2 21 2 25
3J16_L 0.29 0.23 0.37 7 2756 12 0 12 0 23
3J20_2 0.49 0.44 0.56 276 1116272 221 10 207 4 357
3J20_1 0.73 0.70 0.76 16 2905 5 0 5 0 7
3J20_0 0.44 0.40 0.50 12 2826 13 0 12 1 18
3J2C_M - 0.38 0.31 0.47 64 106354 76 10 63 3 143
3J2C_O - 0.63 0.51 0.78 32 10255 10 0 9 1 31
3J2L_3 0.62 0.53 0.74 28 7837 12 0 10 2 25
3JYV_7 0.41 0.31 0.56 10 2832 8 0 8 0 22
3JYX_3 0.64 0.63 0.65 17 6302 19 1 8 10 10
3JYX_4 0.31 0.30 0.31 10 12214 34 4 18 12 23
3KIY_A - 0.54 0.45 0.65 553 4053271 314 23 281 10 688
3KTW_C - 0.54 0.47 0.65 20 4529 12 2 9 1 23
3LA5_A 0.78 0.62 1.00 21 2464 0 0 0 0 13
3NDB_M - 0.78 0.69 0.89 42 9133 6 0 5 1 19
3NKB_B - 0.59 0.54 0.67 14 1995 7 0 7 0 12
3NMU_E - -0.01 0.00 0.00 0 553 10 1 7 2 6
3NPB_A 0.71 0.61 0.82 28 6987 9 0 6 3 18
3O58_2 0.75 0.76 0.74 29 7221 11 3 7 1 9
3O58_3 0.35 0.34 0.35 12 12369 35 2 20 13 23
3OVB_D - 1.00 1.00 1.00 11 584 1 0 0 1 0
3OVS_D - 0.96 0.92 1.00 12 549 0 0 0 0 1
3P22_G - 0.96 0.92 1.00 11 730 0 0 0 0 1
3PDR_A 0.76 0.63 0.94 45 12832 5 0 3 2 27
3R4F_A - 0.96 0.92 1.00 23 2122 0 0 0 0 2
3R9X_C - 0.84 0.80 0.89 8 586 2 0 1 1 2
3RKF_A 0.76 0.59 1.00 20 2191 0 0 0 0 14
3SD1_A 0.58 0.48 0.71 20 3888 8 1 7 0 22
3SIU_F - 0.85 0.73 1.00 8 370 0 0 0 0 3
3SN2_B 1.00 1.00 1.00 12 394 0 0 0 0 0
3TRZ_Z - 0.91 0.83 1.00 5 205 1 0 0 1 1
3TS0_U - 1.00 1.00 1.00 6 247 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 271 0 0 0 0 0
3U4M_B - 0.45 0.35 0.59 13 3138 9 1 8 0 24
3UZL_B 0.77 0.62 0.96 23 3546 1 0 1 0 14
3VJR_D - 0.96 0.92 1.00 12 618 0 0 0 0 1
3W3S_B 0.88 0.78 1.00 31 4722 1 0 0 1 9
3ZEX_G - 0.77 0.66 0.91 49 16417 11 0 5 6 25
3ZEX_D 0.69 0.61 0.79 30 6983 8 1 7 0 19
3ZEX_B - 0.30 0.26 0.35 147 1071957 289 29 247 13 411
3ZEX_E - 0.00 0.00 0.00 0 21893 54 2 50 2 77
3ZEX_C 0.41 0.27 0.64 14 14174 16 2 6 8 38
3ZEX_F - 0.00 0.00 0.00 0 2617 11 2 9 0 12
3ZEX_H - 0.19 0.18 0.19 7 9009 29 5 24 0 31
4A1C_2 0.14 0.15 0.14 5 11746 41 4 26 11 28
4A1C_3 0.69 0.57 0.84 31 7103 6 0 6 0 23
4A4U_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
4AOB_A 0.50 0.40 0.63 17 4344 11 2 8 1 25
4ATO_G - 0.38 0.30 0.50 3 522 3 0 3 0 7
4ENB_A 0.76 0.58 1.00 11 1264 0 0 0 0 8
4ENC_A 0.31 0.26 0.38 5 1313 8 1 7 0 14
4FNJ_A - 0.79 0.63 1.00 10 585 0 0 0 0 6
4FRG_B 0.30 0.22 0.41 7 3469 10 1 9 0 25
4FRN_A 0.45 0.36 0.57 13 5128 10 2 8 0 23
4HXH_A - 1.00 1.00 1.00 6 319 0 0 0 0 0

^top



Performance of UNAFold - scored lower in this pairwise comparison

1. Total counts & total scores for UNAFold

Total Base Pair Counts
Total TP 2645
Total TN 6830945
Total FP 1927
Total FP CONTRA 174
Total FP INCONS 1588
Total FP COMP 165
Total FN 3065
Total Scores
MCC 0.527
Average MCC ± 95% Confidence Intervals 0.672 ± 0.051
Sensitivity 0.463
Positive Predictive Value 0.600
Nr of predictions 118

^top



2. Individual counts for UNAFold [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
2KE6_A 0.97 0.95 1.00 18 1110 1 0 0 1 1
2KPV_A - 0.96 0.92 1.00 12 549 0 0 0 0 1
2KRL_A - 0.76 0.60 0.96 24 5126 5 0 1 4 16
2KU0_A - 0.83 0.69 1.00 11 692 0 0 0 0 5
2KUR_A 0.95 0.90 1.00 19 1109 0 0 0 0 2
2KUU_A 0.92 0.86 1.00 18 1110 1 0 0 1 3
2KUV_A 0.93 0.86 1.00 19 1109 0 0 0 0 3
2KUW_A 0.95 0.90 1.00 19 1109 0 0 0 0 2
2KX8_A 0.94 0.89 1.00 16 845 0 0 0 0 2
2KXM_A - 0.83 0.70 1.00 7 344 0 0 0 0 3
2KZL_A - 0.86 0.74 1.00 14 1471 0 0 0 0 5
2L1F_B 0.98 0.96 1.00 24 2121 0 0 0 0 1
2L1F_A 0.98 0.96 1.00 23 2057 0 0 0 0 1
2L2K_A - 0.94 0.89 1.00 17 844 0 0 0 0 2
2L3C_B - 0.91 0.82 1.00 14 547 0 0 0 0 3
2L3E_A - 0.96 0.92 1.00 12 583 1 0 0 1 1
2L3J_B 0.94 0.88 1.00 30 2455 0 0 0 0 4
2L5Z_A - 0.86 0.75 1.00 9 316 0 0 0 0 3
2L94_A 0.97 0.95 1.00 19 971 0 0 0 0 1
2LA5_A - 0.39 0.26 0.63 5 622 3 1 2 0 14
2LBS_A - 1.00 1.00 1.00 14 482 0 0 0 0 0
2LC8_A -0.01 0.00 0.00 0 1525 15 2 13 0 20
2LDL_A - 0.90 0.82 1.00 9 342 1 0 0 1 2
2LDT_A - 0.81 0.67 1.00 10 455 0 0 0 0 5
2LHP_A - 0.97 0.94 1.00 15 651 0 0 0 0 1
2LI4_A - 0.93 0.88 1.00 14 482 0 0 0 0 2
2LJJ_A - 1.00 1.00 1.00 8 343 2 0 0 2 0
2LK3_A - 0.95 0.90 1.00 9 267 0 0 0 0 1
2LKR_A - 0.93 0.87 1.00 34 6071 1 0 0 1 5
2LQZ_A - 0.85 0.82 0.90 9 341 1 1 0 0 2
2LWK_A - 0.83 0.77 0.91 10 485 1 0 1 0 3
2RRC_A - 0.70 0.50 1.00 5 226 0 0 0 0 5
2WRQ_Y 0.57 0.59 0.56 10 2832 13 5 3 5 7
2WW9_F - 0.59 0.60 0.60 6 290 4 0 4 0 4
2WW9_E - 0.53 0.29 1.00 4 557 0 0 0 0 10
2WW9_D - 0.18 0.12 0.30 3 1943 7 3 4 0 23
2WWQ_V 0.78 0.68 0.90 19 2905 3 0 2 1 9
2XKV_B 0.47 0.45 0.50 9 4542 21 0 9 12 11
2XQD_Y 0.88 0.78 1.00 21 2829 0 0 0 0 6
2XXA_G 0.34 0.31 0.39 13 5118 20 1 19 0 29
2Y8W_B - 1.00 1.00 1.00 6 184 1 0 0 1 0
2Y9C_V - 0.62 0.58 0.67 14 2257 8 1 6 1 10
2YIE_X - 0.36 0.33 0.40 4 1368 6 2 4 0 8
2YIE_Z - -0.01 0.00 0.00 0 1525 15 3 12 0 12
3A2K_C 0.42 0.39 0.46 11 2902 13 2 11 0 17
3ADB_C - 0.80 0.74 0.88 28 4154 4 0 4 0 10
3AKZ_H 0.43 0.39 0.48 11 2678 12 2 10 0 17
3AM1_B - 0.68 0.63 0.73 22 3210 8 1 7 0 13
3AMU_B 0.70 0.59 0.84 16 2984 5 0 3 2 11
3GX2_A 0.68 0.55 0.85 22 4345 5 0 4 1 18
3IAB_R - 0.92 0.86 1.00 12 1023 0 0 0 0 2
3IVN_B 0.78 0.61 1.00 19 2327 0 0 0 0 12
3IWN_A 0.74 0.67 0.81 22 4251 6 0 5 1 11
3IYQ_A 0.31 0.33 0.28 31 60617 82 22 56 4 63
3IZ4_A 0.51 0.45 0.58 60 70772 50 4 40 6 72
3IZF_C 0.70 0.61 0.80 33 6862 8 1 7 0 21
3J0L_1 - 0.70 0.58 0.85 11 1212 2 0 2 0 8
3J0L_8 - 0.93 0.88 1.00 7 183 0 0 0 0 1
3J0L_a - 0.51 0.50 0.53 8 1113 7 3 4 0 8
3J0L_h - 0.75 0.60 0.93 26 6077 4 0 2 2 17
3J0L_7 - 0.48 0.41 0.58 7 1213 5 0 5 0 10
3J0L_g - -0.01 0.00 0.00 0 461 4 1 3 0 4
3J0L_2 - 0.25 0.24 0.26 8 6185 25 2 21 2 25
3J16_L 0.26 0.23 0.30 7 2752 16 1 15 0 23
3J20_2 0.47 0.41 0.53 260 1116278 230 15 212 3 373
3J20_1 0.73 0.70 0.76 16 2905 5 0 5 0 7
3J20_0 0.43 0.40 0.48 12 2825 14 0 13 1 18
3J2C_M - 0.56 0.46 0.69 96 106351 49 7 37 5 111
3J2C_O - 0.52 0.44 0.61 28 10250 19 0 18 1 35
3J2L_3 0.62 0.53 0.74 28 7837 12 0 10 2 25
3JYV_7 0.22 0.19 0.29 6 2829 15 0 15 0 26
3JYX_3 0.62 0.63 0.61 17 6300 22 1 10 11 10
3JYX_4 0.32 0.30 0.34 10 12217 31 3 16 12 23
3KIY_A - 0.41 0.35 0.48 429 4053241 467 22 436 9 812
3KTW_C - 0.77 0.65 0.90 28 4529 4 2 1 1 15
3LA5_A 0.78 0.62 1.00 21 2464 0 0 0 0 13
3NDB_M - 0.84 0.74 0.96 45 9133 3 0 2 1 16
3NKB_B - 0.59 0.54 0.67 14 1995 7 0 7 0 12
3NMU_E - 0.54 0.50 0.60 3 556 5 0 2 3 3
3NPB_A 0.78 0.65 0.94 30 6989 5 0 2 3 16
3O58_2 0.71 0.71 0.71 27 7222 12 4 7 1 11
3O58_3 0.35 0.34 0.35 12 12369 33 1 21 11 23
3OVB_D - 1.00 1.00 1.00 11 584 1 0 0 1 0
3OVS_D - 0.96 0.92 1.00 12 549 0 0 0 0 1
3P22_G - 0.96 0.92 1.00 11 730 0 0 0 0 1
3PDR_A 0.77 0.64 0.94 46 12831 5 0 3 2 26
3R4F_A - 0.96 0.92 1.00 23 2122 0 0 0 0 2
3R9X_C - 0.84 0.80 0.89 8 586 2 0 1 1 2
3RKF_A 0.76 0.59 1.00 20 2191 0 0 0 0 14
3SD1_A 0.64 0.52 0.79 22 3888 6 1 5 0 20
3SIU_F - 0.85 0.73 1.00 8 370 0 0 0 0 3
3SN2_B 0.96 0.92 1.00 11 395 0 0 0 0 1
3TRZ_Z - 0.91 0.83 1.00 5 205 1 0 0 1 1
3TS0_U - 1.00 1.00 1.00 6 247 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 271 0 0 0 0 0
3U4M_B - 0.38 0.32 0.46 12 3134 14 0 14 0 25
3UZL_B 0.48 0.38 0.61 14 3547 9 0 9 0 23
3VJR_D - 0.96 0.92 1.00 12 618 0 0 0 0 1
3W3S_B 0.56 0.50 0.63 20 4721 13 1 11 1 20
3ZEX_G - 0.75 0.64 0.89 47 16418 12 0 6 6 27
3ZEX_D 0.77 0.65 0.91 32 6986 3 0 3 0 17
3ZEX_B - 0.31 0.28 0.35 154 1071936 301 33 257 11 404
3ZEX_E - 0.00 0.00 0.00 0 21893 54 4 48 2 77
3ZEX_C 0.24 0.21 0.28 11 14157 42 1 27 14 41
3ZEX_F - 0.00 0.00 0.00 0 2617 11 2 9 0 12
3ZEX_H - 0.18 0.18 0.18 7 9006 32 6 26 0 31
4A1C_2 0.14 0.15 0.14 5 11745 42 5 26 11 28
4A1C_3 0.70 0.59 0.82 32 7101 7 1 6 0 22
4A4U_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
4AOB_A 0.50 0.40 0.63 17 4344 11 2 8 1 25
4ATO_G - 0.32 0.30 0.38 3 520 5 1 4 0 7
4ENB_A 0.70 0.58 0.85 11 1262 2 1 1 0 8
4ENC_A 0.32 0.26 0.42 5 1314 7 1 6 0 14
4FNJ_A - 0.79 0.63 1.00 10 585 0 0 0 0 6
4FRG_B 0.32 0.28 0.38 9 3462 15 0 15 0 23
4FRN_A 0.43 0.36 0.52 13 5126 12 2 10 0 23
4HXH_A - 1.00 1.00 1.00 6 319 0 0 0 0 0

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.