CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of MaxExpect - scored higher in this pairwise comparison

  4. Performance of Sfold - scored lower in this pairwise comparison

  5. Compile and download dataset for MaxExpect & Sfold [.zip] - may take several seconds...


Overview

Metric MaxExpect Sfold
MCC 0.552 > 0.544
Average MCC ± 95% Confidence Intervals 0.661 ± 0.055 < 0.683 ± 0.049
Sensitivity 0.480 > 0.460
Positive Predictive Value 0.635 < 0.645
Total TP 2742 > 2625
Total TN 6831037 < 6831284
Total FP 1738 > 1596
Total FP CONTRA 156 > 142
Total FP INCONS 1417 > 1301
Total FP COMP 165 > 153
Total FN 2968 < 3085
P-value 0.000150377120024

^top




Performance plots


  1. Comparison of performance of MaxExpect and Sfold. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for MaxExpect and Sfold).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for MaxExpect and Sfold).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for MaxExpect and Sfold. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for MaxExpect and Sfold).

^top





Performance of MaxExpect - scored higher in this pairwise comparison

1. Total counts & total scores for MaxExpect

Total Base Pair Counts
Total TP 2742
Total TN 6831037
Total FP 1738
Total FP CONTRA 156
Total FP INCONS 1417
Total FP COMP 165
Total FN 2968
Total Scores
MCC 0.552
Average MCC ± 95% Confidence Intervals 0.661 ± 0.055
Sensitivity 0.480
Positive Predictive Value 0.635
Nr of predictions 118

^top



2. Individual counts for MaxExpect [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
2KE6_A 0.92 0.89 0.94 17 1110 2 0 1 1 2
2KPV_A - 0.96 0.92 1.00 12 549 0 0 0 0 1
2KRL_A - 0.76 0.60 0.96 24 5126 5 0 1 4 16
2KU0_A - -0.02 0.00 0.00 0 694 9 0 9 0 16
2KUR_A 0.90 0.86 0.95 18 1109 1 0 1 0 3
2KUU_A 0.87 0.81 0.94 17 1110 2 0 1 1 4
2KUV_A 0.88 0.82 0.95 18 1109 1 0 1 0 4
2KUW_A 0.90 0.86 0.95 18 1109 1 0 1 0 3
2KX8_A 0.94 0.89 1.00 16 845 0 0 0 0 2
2KXM_A - 0.83 0.70 1.00 7 344 0 0 0 0 3
2KZL_A - 0.86 0.74 1.00 14 1471 0 0 0 0 5
2L1F_B 0.98 0.96 1.00 24 2121 0 0 0 0 1
2L1F_A 0.98 0.96 1.00 23 2057 0 0 0 0 1
2L2K_A - 0.94 0.89 1.00 17 844 0 0 0 0 2
2L3C_B - 0.91 0.82 1.00 14 547 0 0 0 0 3
2L3E_A - 0.96 0.92 1.00 12 583 1 0 0 1 1
2L3J_B 0.94 0.88 1.00 30 2455 0 0 0 0 4
2L5Z_A - 0.86 0.75 1.00 9 316 0 0 0 0 3
2L94_A 0.97 0.95 1.00 19 971 0 0 0 0 1
2LA5_A - 0.39 0.26 0.63 5 622 3 1 2 0 14
2LBS_A - 1.00 1.00 1.00 14 482 0 0 0 0 0
2LC8_A -0.01 0.00 0.00 0 1525 15 2 13 0 20
2LDL_A - 0.90 0.82 1.00 9 342 1 0 0 1 2
2LDT_A - 0.85 0.73 1.00 11 454 0 0 0 0 4
2LHP_A - 0.97 0.94 1.00 15 651 0 0 0 0 1
2LI4_A - 0.93 0.88 1.00 14 482 0 0 0 0 2
2LJJ_A - 1.00 1.00 1.00 8 343 2 0 0 2 0
2LK3_A - 0.95 0.90 1.00 9 267 0 0 0 0 1
2LKR_A - 0.81 0.74 0.88 29 6072 6 0 4 2 10
2LQZ_A - 0.85 0.82 0.90 9 341 1 1 0 0 2
2LWK_A - 0.88 0.85 0.92 11 484 1 0 1 0 2
2RRC_A - 0.70 0.50 1.00 5 226 0 0 0 0 5
2WRQ_Y 0.53 0.53 0.53 9 2833 13 5 3 5 8
2WW9_F - 0.69 0.70 0.70 7 290 3 0 3 0 3
2WW9_E - -0.01 0.00 0.00 0 556 5 0 5 0 14
2WW9_D - 0.14 0.12 0.19 3 1937 13 0 13 0 23
2WWQ_V 0.76 0.64 0.90 18 2906 3 0 2 1 10
2XKV_B 0.51 0.50 0.53 10 4541 24 0 9 15 10
2XQD_Y 0.88 0.78 1.00 21 2829 0 0 0 0 6
2XXA_G 0.25 0.24 0.27 10 5114 27 1 26 0 32
2Y8W_B - 1.00 1.00 1.00 6 184 1 0 0 1 0
2Y9C_V - 0.65 0.63 0.68 15 2256 8 1 6 1 9
2YIE_X - 0.36 0.33 0.40 4 1368 6 2 4 0 8
2YIE_Z - -0.01 0.00 0.00 0 1525 15 3 12 0 12
3A2K_C 0.42 0.39 0.46 11 2902 13 2 11 0 17
3ADB_C - 0.80 0.74 0.88 28 4154 4 0 4 0 10
3AKZ_H 0.84 0.75 0.95 21 2679 1 1 0 0 7
3AM1_B - 0.78 0.71 0.86 25 3211 4 0 4 0 10
3AMU_B 0.64 0.59 0.70 16 2980 9 0 7 2 11
3GX2_A 0.77 0.63 0.96 25 4345 2 0 1 1 15
3IAB_R - 0.92 0.86 1.00 12 1023 0 0 0 0 2
3IVN_B 0.78 0.61 1.00 19 2327 0 0 0 0 12
3IWN_A 0.74 0.67 0.81 22 4251 6 0 5 1 11
3IYQ_A 0.29 0.31 0.27 29 60617 84 19 61 4 65
3IZ4_A 0.54 0.46 0.64 61 70781 39 3 31 5 71
3IZF_C 0.71 0.61 0.83 33 6863 7 1 6 0 21
3J0L_1 - 0.76 0.63 0.92 12 1212 2 0 1 1 7
3J0L_8 - 0.93 0.88 1.00 7 183 0 0 0 0 1
3J0L_a - 0.39 0.31 0.50 5 1118 5 2 3 0 11
3J0L_h - 0.81 0.65 1.00 28 6077 2 0 0 2 15
3J0L_7 - -0.01 0.00 0.00 0 1213 12 0 12 0 17
3J0L_g - -0.01 0.00 0.00 0 461 4 1 3 0 4
3J0L_2 - 0.25 0.24 0.26 8 6185 25 2 21 2 25
3J16_L 0.29 0.23 0.37 7 2756 12 0 12 0 23
3J20_2 0.49 0.44 0.56 276 1116272 221 10 207 4 357
3J20_1 0.73 0.70 0.76 16 2905 5 0 5 0 7
3J20_0 0.44 0.40 0.50 12 2826 13 0 12 1 18
3J2C_M - 0.38 0.31 0.47 64 106354 76 10 63 3 143
3J2C_O - 0.63 0.51 0.78 32 10255 10 0 9 1 31
3J2L_3 0.62 0.53 0.74 28 7837 12 0 10 2 25
3JYV_7 0.41 0.31 0.56 10 2832 8 0 8 0 22
3JYX_3 0.64 0.63 0.65 17 6302 19 1 8 10 10
3JYX_4 0.31 0.30 0.31 10 12214 34 4 18 12 23
3KIY_A - 0.54 0.45 0.65 553 4053271 314 23 281 10 688
3KTW_C - 0.54 0.47 0.65 20 4529 12 2 9 1 23
3LA5_A 0.78 0.62 1.00 21 2464 0 0 0 0 13
3NDB_M - 0.78 0.69 0.89 42 9133 6 0 5 1 19
3NKB_B - 0.59 0.54 0.67 14 1995 7 0 7 0 12
3NMU_E - -0.01 0.00 0.00 0 553 10 1 7 2 6
3NPB_A 0.71 0.61 0.82 28 6987 9 0 6 3 18
3O58_2 0.75 0.76 0.74 29 7221 11 3 7 1 9
3O58_3 0.35 0.34 0.35 12 12369 35 2 20 13 23
3OVB_D - 1.00 1.00 1.00 11 584 1 0 0 1 0
3OVS_D - 0.96 0.92 1.00 12 549 0 0 0 0 1
3P22_G - 0.96 0.92 1.00 11 730 0 0 0 0 1
3PDR_A 0.76 0.63 0.94 45 12832 5 0 3 2 27
3R4F_A - 0.96 0.92 1.00 23 2122 0 0 0 0 2
3R9X_C - 0.84 0.80 0.89 8 586 2 0 1 1 2
3RKF_A 0.76 0.59 1.00 20 2191 0 0 0 0 14
3SD1_A 0.58 0.48 0.71 20 3888 8 1 7 0 22
3SIU_F - 0.85 0.73 1.00 8 370 0 0 0 0 3
3SN2_B 1.00 1.00 1.00 12 394 0 0 0 0 0
3TRZ_Z - 0.91 0.83 1.00 5 205 1 0 0 1 1
3TS0_U - 1.00 1.00 1.00 6 247 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 271 0 0 0 0 0
3U4M_B - 0.45 0.35 0.59 13 3138 9 1 8 0 24
3UZL_B 0.77 0.62 0.96 23 3546 1 0 1 0 14
3VJR_D - 0.96 0.92 1.00 12 618 0 0 0 0 1
3W3S_B 0.88 0.78 1.00 31 4722 1 0 0 1 9
3ZEX_G - 0.77 0.66 0.91 49 16417 11 0 5 6 25
3ZEX_D 0.69 0.61 0.79 30 6983 8 1 7 0 19
3ZEX_B - 0.30 0.26 0.35 147 1071957 289 29 247 13 411
3ZEX_E - 0.00 0.00 0.00 0 21893 54 2 50 2 77
3ZEX_C 0.41 0.27 0.64 14 14174 16 2 6 8 38
3ZEX_F - 0.00 0.00 0.00 0 2617 11 2 9 0 12
3ZEX_H - 0.19 0.18 0.19 7 9009 29 5 24 0 31
4A1C_2 0.14 0.15 0.14 5 11746 41 4 26 11 28
4A1C_3 0.69 0.57 0.84 31 7103 6 0 6 0 23
4A4U_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
4AOB_A 0.50 0.40 0.63 17 4344 11 2 8 1 25
4ATO_G - 0.38 0.30 0.50 3 522 3 0 3 0 7
4ENB_A 0.76 0.58 1.00 11 1264 0 0 0 0 8
4ENC_A 0.31 0.26 0.38 5 1313 8 1 7 0 14
4FNJ_A - 0.79 0.63 1.00 10 585 0 0 0 0 6
4FRG_B 0.30 0.22 0.41 7 3469 10 1 9 0 25
4FRN_A 0.45 0.36 0.57 13 5128 10 2 8 0 23
4HXH_A - 1.00 1.00 1.00 6 319 0 0 0 0 0

^top



Performance of Sfold - scored lower in this pairwise comparison

1. Total counts & total scores for Sfold

Total Base Pair Counts
Total TP 2625
Total TN 6831284
Total FP 1596
Total FP CONTRA 142
Total FP INCONS 1301
Total FP COMP 153
Total FN 3085
Total Scores
MCC 0.544
Average MCC ± 95% Confidence Intervals 0.683 ± 0.049
Sensitivity 0.460
Positive Predictive Value 0.645
Nr of predictions 118

^top



2. Individual counts for Sfold [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
2KE6_A 0.92 0.89 0.94 17 1110 2 0 1 1 2
2KPV_A - 0.96 0.92 1.00 12 549 0 0 0 0 1
2KRL_A - 0.76 0.60 0.96 24 5126 5 0 1 4 16
2KU0_A - 0.83 0.69 1.00 11 692 0 0 0 0 5
2KUR_A 0.90 0.86 0.95 18 1109 1 0 1 0 3
2KUU_A 0.87 0.81 0.94 17 1110 2 0 1 1 4
2KUV_A 0.88 0.82 0.95 18 1109 1 0 1 0 4
2KUW_A 0.90 0.86 0.95 18 1109 1 0 1 0 3
2KX8_A 0.94 0.89 1.00 16 845 0 0 0 0 2
2KXM_A - 0.89 0.80 1.00 8 343 0 0 0 0 2
2KZL_A - 0.86 0.74 1.00 14 1471 0 0 0 0 5
2L1F_B 0.98 0.96 1.00 24 2121 0 0 0 0 1
2L1F_A 0.98 0.96 1.00 23 2057 0 0 0 0 1
2L2K_A - 0.94 0.89 1.00 17 844 0 0 0 0 2
2L3C_B - 0.91 0.82 1.00 14 547 0 0 0 0 3
2L3E_A - 0.96 0.92 1.00 12 583 1 0 0 1 1
2L3J_B 0.94 0.88 1.00 30 2455 0 0 0 0 4
2L5Z_A - 0.86 0.75 1.00 9 316 0 0 0 0 3
2L94_A 0.97 0.95 1.00 19 971 0 0 0 0 1
2LA5_A - 0.39 0.26 0.63 5 622 3 1 2 0 14
2LBS_A - 1.00 1.00 1.00 14 482 0 0 0 0 0
2LC8_A 0.45 0.35 0.58 7 1528 5 0 5 0 13
2LDL_A - 0.90 0.82 1.00 9 342 1 0 0 1 2
2LDT_A - 0.85 0.73 1.00 11 454 0 0 0 0 4
2LHP_A - 0.97 0.94 1.00 15 651 0 0 0 0 1
2LI4_A - 0.93 0.88 1.00 14 482 0 0 0 0 2
2LJJ_A - 1.00 1.00 1.00 8 343 2 0 0 2 0
2LK3_A - 0.95 0.90 1.00 9 267 0 0 0 0 1
2LKR_A - 0.89 0.79 1.00 31 6074 1 0 0 1 8
2LQZ_A - 0.85 0.82 0.90 9 341 1 1 0 0 2
2LWK_A - 0.88 0.85 0.92 11 484 1 0 1 0 2
2RRC_A - 0.83 0.70 1.00 7 224 0 0 0 0 3
2WRQ_Y 0.53 0.53 0.53 9 2833 13 5 3 5 8
2WW9_F - 0.69 0.70 0.70 7 290 3 0 3 0 3
2WW9_E - 0.53 0.29 1.00 4 557 0 0 0 0 10
2WW9_D - 0.29 0.12 0.75 3 1949 1 0 1 0 23
2WWQ_V 0.80 0.68 0.95 19 2906 2 0 1 1 9
2XKV_B 0.51 0.50 0.53 10 4541 23 0 9 14 10
2XQD_Y 0.77 0.67 0.90 18 2830 2 0 2 0 9
2XXA_G 0.36 0.31 0.43 13 5121 17 1 16 0 29
2Y8W_B - 1.00 1.00 1.00 6 184 1 0 0 1 0
2Y9C_V - 0.65 0.63 0.68 15 2256 8 1 6 1 9
2YIE_X - 0.58 0.33 1.00 4 1374 0 0 0 0 8
2YIE_Z - -0.01 0.00 0.00 0 1525 15 3 12 0 12
3A2K_C 0.42 0.39 0.46 11 2902 13 2 11 0 17
3ADB_C - 0.62 0.55 0.70 21 4156 9 0 9 0 17
3AKZ_H 0.44 0.39 0.50 11 2679 11 2 9 0 17
3AM1_B - 0.69 0.63 0.76 22 3211 7 1 6 0 13
3AMU_B 0.70 0.59 0.84 16 2984 5 0 3 2 11
3GX2_A 0.72 0.55 0.96 22 4348 2 0 1 1 18
3IAB_R - 0.92 0.86 1.00 12 1023 0 0 0 0 2
3IVN_B 0.78 0.61 1.00 19 2327 0 0 0 0 12
3IWN_A 0.63 0.58 0.70 19 4251 9 0 8 1 14
3IYQ_A 0.33 0.34 0.32 32 60627 71 19 48 4 62
3IZ4_A 0.52 0.42 0.65 56 70790 31 3 27 1 76
3IZF_C 0.71 0.61 0.83 33 6863 7 0 7 0 21
3J0L_1 - 0.76 0.63 0.92 12 1212 3 0 1 2 7
3J0L_8 - 0.93 0.88 1.00 7 183 0 0 0 0 1
3J0L_a - 0.51 0.50 0.53 8 1113 7 3 4 0 8
3J0L_h - 0.75 0.60 0.93 26 6077 4 0 2 2 17
3J0L_7 - -0.01 0.00 0.00 0 1222 3 0 3 0 17
3J0L_g - -0.01 0.00 0.00 0 460 5 1 4 0 4
3J0L_2 - 0.36 0.33 0.39 11 6188 19 2 15 2 22
3J16_L 0.29 0.23 0.37 7 2756 12 0 12 0 23
3J20_2 0.47 0.41 0.55 261 1116287 220 14 203 3 372
3J20_1 0.73 0.70 0.76 16 2905 6 0 5 1 7
3J20_0 0.43 0.40 0.48 12 2825 14 0 13 1 18
3J2C_M - 0.63 0.49 0.80 102 106363 31 3 23 5 105
3J2C_O - 0.64 0.48 0.86 30 10261 6 0 5 1 33
3J2L_3 0.62 0.53 0.74 28 7837 12 0 10 2 25
3JYV_7 -0.01 0.00 0.00 0 2834 16 0 16 0 32
3JYX_3 0.62 0.63 0.61 17 6300 19 1 10 8 10
3JYX_4 0.35 0.30 0.42 10 12222 23 3 11 9 23
3KIY_A - 0.43 0.34 0.55 424 4053360 352 20 324 8 817
3KTW_C - 0.74 0.63 0.87 27 4529 5 2 2 1 16
3LA5_A 0.78 0.62 1.00 21 2464 0 0 0 0 13
3NDB_M - 0.81 0.69 0.95 42 9136 3 0 2 1 19
3NKB_B - 0.64 0.54 0.78 14 1998 4 0 4 0 12
3NMU_E - 0.61 0.50 0.75 3 557 4 0 1 3 3
3NPB_A 0.77 0.65 0.91 30 6988 6 1 2 3 16
3O58_2 0.74 0.74 0.74 28 7222 11 3 7 1 10
3O58_3 0.43 0.34 0.55 12 12381 17 0 10 7 23
3OVB_D - 1.00 1.00 1.00 11 584 1 0 0 1 0
3OVS_D - 0.96 0.92 1.00 12 549 0 0 0 0 1
3P22_G - 0.96 0.92 1.00 11 730 0 0 0 0 1
3PDR_A 0.69 0.56 0.85 40 12833 9 0 7 2 32
3R4F_A - 0.96 0.92 1.00 23 2122 0 0 0 0 2
3R9X_C - 0.84 0.80 0.89 8 586 2 0 1 1 2
3RKF_A 0.76 0.59 1.00 20 2191 0 0 0 0 14
3SD1_A 0.65 0.50 0.84 21 3891 4 1 3 0 21
3SIU_F - 0.85 0.73 1.00 8 370 0 0 0 0 3
3SN2_B 0.96 0.92 1.00 11 395 0 0 0 0 1
3TRZ_Z - 0.91 0.83 1.00 5 205 2 0 0 2 1
3TS0_U - 1.00 1.00 1.00 6 247 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 271 0 0 0 0 0
3U4M_B - 0.38 0.32 0.46 12 3134 14 0 14 0 25
3UZL_B 0.51 0.32 0.80 12 3555 3 2 1 0 25
3VJR_D - 0.96 0.92 1.00 12 618 0 0 0 0 1
3W3S_B 0.57 0.53 0.64 21 4720 13 1 11 1 19
3ZEX_G - 0.78 0.68 0.91 50 16416 9 0 5 4 24
3ZEX_D 0.74 0.63 0.86 31 6985 5 0 5 0 18
3ZEX_B - 0.31 0.27 0.36 152 1071959 280 27 242 11 406
3ZEX_E - 0.00 0.00 0.00 0 21891 56 3 51 2 77
3ZEX_C 0.32 0.21 0.48 11 14173 26 1 11 14 41
3ZEX_F - 0.00 0.00 0.00 0 2623 7 2 3 2 12
3ZEX_H - 0.19 0.18 0.19 7 9009 29 5 24 0 31
4A1C_2 0.16 0.15 0.17 5 11751 34 2 23 9 28
4A1C_3 0.69 0.57 0.84 31 7103 6 1 5 0 23
4A4U_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
4AOB_A 0.50 0.40 0.63 17 4344 11 2 8 1 25
4ATO_G - 0.31 0.10 1.00 1 527 0 0 0 0 9
4ENB_A 0.76 0.58 1.00 11 1264 0 0 0 0 8
4ENC_A 0.51 0.26 1.00 5 1321 0 0 0 0 14
4FNJ_A - 0.79 0.63 1.00 10 585 0 0 0 0 6
4FRG_B 0.62 0.47 0.83 15 3468 3 1 2 0 17
4FRN_A 0.63 0.44 0.89 16 5133 2 2 0 0 20
4HXH_A - 1.00 1.00 1.00 6 319 0 0 0 0 0

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.