CompaRNA - on-line benchmarks of RNA structure prediction methods
Home

Methods
Datasets
Rankings
RNA 2D Atlas

Help
FAQ

Contact us
RSS feeds
Twitter

Table of contents:

  1. Overview

  2. Performance Plots

  3. Performance of MaxExpect - scored higher in this pairwise comparison

  4. Performance of RNAshapes - scored lower in this pairwise comparison

  5. Compile and download dataset for MaxExpect & RNAshapes [.zip] - may take several seconds...


Overview

Metric MaxExpect RNAshapes
MCC 0.610 > 0.600
Average MCC ± 95% Confidence Intervals 0.666 ± 0.056 < 0.673 ± 0.053
Sensitivity 0.539 > 0.533
Positive Predictive Value 0.695 > 0.679
Total TP 1766 > 1747
Total TN 589537 > 589506
Total FP 914 < 963
Total FP CONTRA 94 < 102
Total FP INCONS 682 < 724
Total FP COMP 138 > 137
Total FN 1512 < 1531
P-value 4.48494289621e-08

^top




Performance plots


  1. Comparison of performance of MaxExpect and RNAshapes. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for MaxExpect and RNAshapes).

  2. Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for MaxExpect and RNAshapes).

  3. Comparison of average Matthews Correlation Coefficients (MCCs) for MaxExpect and RNAshapes. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for MaxExpect and RNAshapes).

^top





Performance of MaxExpect - scored higher in this pairwise comparison

1. Total counts & total scores for MaxExpect

Total Base Pair Counts
Total TP 1766
Total TN 589537
Total FP 914
Total FP CONTRA 94
Total FP INCONS 682
Total FP COMP 138
Total FN 1512
Total Scores
MCC 0.610
Average MCC ± 95% Confidence Intervals 0.666 ± 0.056
Sensitivity 0.539
Positive Predictive Value 0.695
Nr of predictions 115

^top



2. Individual counts for MaxExpect [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
2KE6_A 0.92 0.89 0.94 17 1110 2 0 1 1 2
2KPV_A - 0.96 0.92 1.00 12 549 0 0 0 0 1
2KRL_A - 0.76 0.60 0.96 24 5126 5 0 1 4 16
2KU0_A - -0.02 0.00 0.00 0 694 9 0 9 0 16
2KUR_A 0.90 0.86 0.95 18 1109 1 0 1 0 3
2KUU_A 0.87 0.81 0.94 17 1110 2 0 1 1 4
2KUV_A 0.88 0.82 0.95 18 1109 1 0 1 0 4
2KUW_A 0.90 0.86 0.95 18 1109 1 0 1 0 3
2KX8_A 0.94 0.89 1.00 16 845 0 0 0 0 2
2KXM_A - 0.83 0.70 1.00 7 344 0 0 0 0 3
2KZL_A - 0.86 0.74 1.00 14 1471 0 0 0 0 5
2L1F_A 0.98 0.96 1.00 23 2057 0 0 0 0 1
2L1F_B 0.98 0.96 1.00 24 2121 0 0 0 0 1
2L2K_A - 0.94 0.89 1.00 17 844 0 0 0 0 2
2L3C_B - 0.91 0.82 1.00 14 547 0 0 0 0 3
2L3E_A - 0.96 0.92 1.00 12 583 1 0 0 1 1
2L3J_B 0.94 0.88 1.00 30 2455 0 0 0 0 4
2L5Z_A - 0.86 0.75 1.00 9 316 0 0 0 0 3
2L94_A 0.97 0.95 1.00 19 971 0 0 0 0 1
2LA5_A - 0.39 0.26 0.63 5 622 3 1 2 0 14
2LBS_A - 1.00 1.00 1.00 14 482 0 0 0 0 0
2LC8_A -0.01 0.00 0.00 0 1525 15 2 13 0 20
2LDL_A - 0.90 0.82 1.00 9 342 1 0 0 1 2
2LDT_A - 0.85 0.73 1.00 11 454 0 0 0 0 4
2LHP_A - 0.97 0.94 1.00 15 651 0 0 0 0 1
2LI4_A - 0.93 0.88 1.00 14 482 0 0 0 0 2
2LJJ_A - 1.00 1.00 1.00 8 343 2 0 0 2 0
2LK3_A - 0.95 0.90 1.00 9 267 0 0 0 0 1
2LKR_A - 0.81 0.74 0.88 29 6072 6 0 4 2 10
2LQZ_A - 0.85 0.82 0.90 9 341 1 1 0 0 2
2LWK_A - 0.88 0.85 0.92 11 484 1 0 1 0 2
2RRC_A - 0.70 0.50 1.00 5 226 0 0 0 0 5
2WRQ_Y 0.53 0.53 0.53 9 2833 13 5 3 5 8
2WW9_D - 0.14 0.12 0.19 3 1937 13 0 13 0 23
2WW9_E - -0.01 0.00 0.00 0 556 5 0 5 0 14
2WW9_F - 0.69 0.70 0.70 7 290 3 0 3 0 3
2WWQ_V 0.76 0.64 0.90 18 2906 3 0 2 1 10
2XKV_B 0.51 0.50 0.53 10 4541 24 0 9 15 10
2XQD_Y 0.88 0.78 1.00 21 2829 0 0 0 0 6
2XXA_G 0.25 0.24 0.27 10 5114 27 1 26 0 32
2Y8W_B - 1.00 1.00 1.00 6 184 1 0 0 1 0
2Y9C_V - 0.65 0.63 0.68 15 2256 8 1 6 1 9
2YIE_X - 0.36 0.33 0.40 4 1368 6 2 4 0 8
2YIE_Z - -0.01 0.00 0.00 0 1525 15 3 12 0 12
3A2K_C 0.42 0.39 0.46 11 2902 13 2 11 0 17
3ADB_C - 0.80 0.74 0.88 28 4154 4 0 4 0 10
3AKZ_H 0.84 0.75 0.95 21 2679 1 1 0 0 7
3AM1_B - 0.78 0.71 0.86 25 3211 4 0 4 0 10
3AMU_B 0.64 0.59 0.70 16 2980 9 0 7 2 11
3GX2_A 0.77 0.63 0.96 25 4345 2 0 1 1 15
3IAB_R - 0.92 0.86 1.00 12 1023 0 0 0 0 2
3IVN_B 0.78 0.61 1.00 19 2327 0 0 0 0 12
3IWN_A 0.74 0.67 0.81 22 4251 6 0 5 1 11
3IYQ_A 0.29 0.31 0.27 29 60617 84 19 61 4 65
3IZ4_A 0.54 0.46 0.64 61 70781 39 3 31 5 71
3IZF_C 0.71 0.61 0.83 33 6863 7 1 6 0 21
3J0L_g - -0.01 0.00 0.00 0 461 4 1 3 0 4
3J0L_7 - -0.01 0.00 0.00 0 1213 12 0 12 0 17
3J0L_h - 0.81 0.65 1.00 28 6077 2 0 0 2 15
3J0L_1 - 0.76 0.63 0.92 12 1212 2 0 1 1 7
3J0L_a - 0.39 0.31 0.50 5 1118 5 2 3 0 11
3J0L_8 - 0.93 0.88 1.00 7 183 0 0 0 0 1
3J0L_2 - 0.25 0.24 0.26 8 6185 25 2 21 2 25
3J16_L 0.29 0.23 0.37 7 2756 12 0 12 0 23
3J20_0 0.44 0.40 0.50 12 2826 13 0 12 1 18
3J20_1 0.73 0.70 0.76 16 2905 5 0 5 0 7
3J2C_O - 0.63 0.51 0.78 32 10255 10 0 9 1 31
3J2C_M - 0.38 0.31 0.47 64 106354 76 10 63 3 143
3J2L_3 0.62 0.53 0.74 28 7837 12 0 10 2 25
3JYV_7 0.41 0.31 0.56 10 2832 8 0 8 0 22
3JYX_4 0.31 0.30 0.31 10 12214 34 4 18 12 23
3JYX_3 0.64 0.63 0.65 17 6302 19 1 8 10 10
3KTW_C - 0.54 0.47 0.65 20 4529 12 2 9 1 23
3LA5_A 0.78 0.62 1.00 21 2464 0 0 0 0 13
3NDB_M - 0.78 0.69 0.89 42 9133 6 0 5 1 19
3NKB_B - 0.59 0.54 0.67 14 1995 7 0 7 0 12
3NMU_E - -0.01 0.00 0.00 0 553 10 1 7 2 6
3NPB_A 0.71 0.61 0.82 28 6987 9 0 6 3 18
3O58_2 0.75 0.76 0.74 29 7221 11 3 7 1 9
3O58_3 0.35 0.34 0.35 12 12369 35 2 20 13 23
3OVB_D - 1.00 1.00 1.00 11 584 1 0 0 1 0
3OVS_D - 0.96 0.92 1.00 12 549 0 0 0 0 1
3P22_G - 0.96 0.92 1.00 11 730 0 0 0 0 1
3PDR_A 0.76 0.63 0.94 45 12832 5 0 3 2 27
3R4F_A - 0.96 0.92 1.00 23 2122 0 0 0 0 2
3R9X_C - 0.84 0.80 0.89 8 586 2 0 1 1 2
3RKF_A 0.76 0.59 1.00 20 2191 0 0 0 0 14
3SD1_A 0.58 0.48 0.71 20 3888 8 1 7 0 22
3SIU_F - 0.85 0.73 1.00 8 370 0 0 0 0 3
3SN2_B 1.00 1.00 1.00 12 394 0 0 0 0 0
3TRZ_Z - 0.91 0.83 1.00 5 205 1 0 0 1 1
3TS0_U - 1.00 1.00 1.00 6 247 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 271 0 0 0 0 0
3U4M_B - 0.45 0.35 0.59 13 3138 9 1 8 0 24
3UZL_B 0.77 0.62 0.96 23 3546 1 0 1 0 14
3VJR_D - 0.96 0.92 1.00 12 618 0 0 0 0 1
3W3S_B 0.88 0.78 1.00 31 4722 1 0 0 1 9
3ZEX_F - 0.00 0.00 0.00 0 2617 11 2 9 0 12
3ZEX_C 0.41 0.27 0.64 14 14174 16 2 6 8 38
3ZEX_G - 0.77 0.66 0.91 49 16417 11 0 5 6 25
3ZEX_D 0.69 0.61 0.79 30 6983 8 1 7 0 19
3ZEX_E - 0.00 0.00 0.00 0 21893 54 2 50 2 77
3ZEX_H - 0.19 0.18 0.19 7 9009 29 5 24 0 31
4A1C_2 0.14 0.15 0.14 5 11746 41 4 26 11 28
4A1C_3 0.69 0.57 0.84 31 7103 6 0 6 0 23
4A4U_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
4AOB_A 0.50 0.40 0.63 17 4344 11 2 8 1 25
4ATO_G - 0.38 0.30 0.50 3 522 3 0 3 0 7
4ENB_A 0.76 0.58 1.00 11 1264 0 0 0 0 8
4ENC_A 0.31 0.26 0.38 5 1313 8 1 7 0 14
4FNJ_A - 0.79 0.63 1.00 10 585 0 0 0 0 6
4FRG_B 0.30 0.22 0.41 7 3469 10 1 9 0 25
4FRN_A 0.45 0.36 0.57 13 5128 10 2 8 0 23
4HXH_A - 1.00 1.00 1.00 6 319 0 0 0 0 0

^top



Performance of RNAshapes - scored lower in this pairwise comparison

1. Total counts & total scores for RNAshapes

Total Base Pair Counts
Total TP 1747
Total TN 589506
Total FP 963
Total FP CONTRA 102
Total FP INCONS 724
Total FP COMP 137
Total FN 1531
Total Scores
MCC 0.600
Average MCC ± 95% Confidence Intervals 0.673 ± 0.053
Sensitivity 0.533
Positive Predictive Value 0.679
Nr of predictions 115

^top



2. Individual counts for RNAshapes [ download as .csv ]

RNA Chain Rfam family MCC SENS PPV TP TN FP FP CONTRA FP INCONS FP COMP FN
2KD8_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
2KE6_A 0.97 0.95 1.00 18 1110 1 0 0 1 1
2KPV_A - 0.96 0.92 1.00 12 549 0 0 0 0 1
2KRL_A - 0.76 0.60 0.96 24 5126 5 0 1 4 16
2KU0_A - 0.83 0.69 1.00 11 692 0 0 0 0 5
2KUR_A 0.95 0.90 1.00 19 1109 0 0 0 0 2
2KUU_A 0.92 0.86 1.00 18 1110 1 0 0 1 3
2KUV_A 0.93 0.86 1.00 19 1109 0 0 0 0 3
2KUW_A 0.95 0.90 1.00 19 1109 0 0 0 0 2
2KX8_A 0.91 0.83 1.00 15 846 0 0 0 0 3
2KXM_A - 0.83 0.70 1.00 7 344 0 0 0 0 3
2KZL_A - 0.86 0.74 1.00 14 1471 0 0 0 0 5
2L1F_A 0.87 0.83 0.91 20 2058 2 0 2 0 4
2L1F_B 0.96 0.92 1.00 23 2122 0 0 0 0 2
2L2K_A - 0.92 0.84 1.00 16 845 0 0 0 0 3
2L3C_B - 0.91 0.82 1.00 14 547 0 0 0 0 3
2L3E_A - 0.96 0.92 1.00 12 583 1 0 0 1 1
2L3J_B 0.92 0.85 1.00 29 2456 0 0 0 0 5
2L5Z_A - 0.86 0.75 1.00 9 316 0 0 0 0 3
2L94_A 0.97 0.95 1.00 19 971 0 0 0 0 1
2LA5_A - 0.39 0.26 0.63 5 622 3 1 2 0 14
2LBS_A - 1.00 1.00 1.00 14 482 0 0 0 0 0
2LC8_A 0.61 0.55 0.69 11 1524 5 0 5 0 9
2LDL_A - 0.90 0.82 1.00 9 342 1 0 0 1 2
2LDT_A - 0.85 0.73 1.00 11 454 0 0 0 0 4
2LHP_A - 0.97 0.94 1.00 15 651 0 0 0 0 1
2LI4_A - 0.93 0.88 1.00 14 482 0 0 0 0 2
2LJJ_A - 1.00 1.00 1.00 8 343 2 0 0 2 0
2LK3_A - 0.95 0.90 1.00 9 267 0 0 0 0 1
2LKR_A - 0.93 0.87 1.00 34 6071 1 0 0 1 5
2LQZ_A - 0.85 0.82 0.90 9 341 1 1 0 0 2
2LWK_A - 0.83 0.77 0.91 10 485 1 0 1 0 3
2RRC_A - 0.70 0.50 1.00 5 226 0 0 0 0 5
2WRQ_Y 0.59 0.59 0.59 10 2833 12 4 3 5 7
2WW9_D - 0.16 0.12 0.25 3 1941 9 4 5 0 23
2WW9_E - 0.53 0.29 1.00 4 557 0 0 0 0 10
2WW9_F - 0.59 0.60 0.60 6 290 4 0 4 0 4
2WWQ_V 0.76 0.68 0.86 19 2904 5 0 3 2 9
2XKV_B 0.51 0.50 0.53 10 4541 22 0 9 13 10
2XQD_Y 0.64 0.56 0.75 15 2830 5 0 5 0 12
2XXA_G 0.93 0.86 1.00 36 5115 0 0 0 0 6
2Y8W_B - 1.00 1.00 1.00 6 184 1 0 0 1 0
2Y9C_V - 0.65 0.63 0.68 15 2256 8 1 6 1 9
2YIE_X - 0.25 0.25 0.27 3 1367 11 1 7 3 9
2YIE_Z - -0.01 0.00 0.00 0 1525 15 3 12 0 12
3A2K_C 0.42 0.39 0.46 11 2902 13 2 11 0 17
3ADB_C - 0.67 0.61 0.74 23 4155 8 0 8 0 15
3AKZ_H 0.68 0.61 0.77 17 2679 6 1 4 1 11
3AM1_B - 0.64 0.60 0.70 21 3210 9 1 8 0 14
3AMU_B 0.70 0.59 0.84 16 2984 5 0 3 2 11
3GX2_A 0.76 0.63 0.93 25 4344 3 0 2 1 15
3IAB_R - 0.92 0.86 1.00 12 1023 0 0 0 0 2
3IVN_B 0.78 0.61 1.00 19 2327 0 0 0 0 12
3IWN_A 0.76 0.67 0.88 22 4253 3 0 3 0 11
3IYQ_A 0.26 0.28 0.24 26 60619 83 21 60 2 68
3IZ4_A 0.52 0.45 0.60 60 70776 45 6 34 5 72
3IZF_C 0.59 0.52 0.68 28 6862 13 1 12 0 26
3J0L_g - -0.01 0.00 0.00 0 461 4 1 3 0 4
3J0L_7 - -0.01 0.00 0.00 0 1213 12 0 12 0 17
3J0L_h - 0.75 0.60 0.93 26 6077 4 0 2 2 17
3J0L_1 - 0.76 0.63 0.92 12 1212 2 0 1 1 7
3J0L_a - 0.23 0.19 0.30 3 1118 7 1 6 0 13
3J0L_8 - 0.93 0.88 1.00 7 183 0 0 0 0 1
3J0L_2 - 0.38 0.36 0.40 12 6186 20 3 15 2 21
3J16_L 0.50 0.40 0.63 12 2756 7 0 7 0 18
3J20_0 0.43 0.40 0.48 12 2825 14 0 13 1 18
3J20_1 0.96 0.91 1.00 21 2905 0 0 0 0 2
3J2C_O - 0.39 0.33 0.47 21 10251 24 0 24 0 42
3J2C_M - 0.47 0.37 0.61 77 106364 53 6 44 3 130
3J2L_3 0.59 0.51 0.69 27 7836 14 0 12 2 26
3JYV_7 -0.01 0.00 0.00 0 2830 20 0 20 0 32
3JYX_4 0.31 0.30 0.31 10 12214 35 5 17 13 23
3JYX_3 0.63 0.63 0.63 17 6301 21 1 9 11 10
3KTW_C - 0.39 0.35 0.45 15 4527 19 1 17 1 28
3LA5_A 0.76 0.59 1.00 20 2465 0 0 0 0 14
3NDB_M - 0.82 0.72 0.94 44 9133 4 0 3 1 17
3NKB_B - 0.61 0.54 0.70 14 1996 6 0 6 0 12
3NMU_E - 0.54 0.50 0.60 3 556 5 0 2 3 3
3NPB_A 0.77 0.63 0.94 29 6990 4 1 1 2 17
3O58_2 0.60 0.61 0.59 23 7221 18 4 12 2 15
3O58_3 0.34 0.34 0.34 12 12368 37 2 21 14 23
3OVB_D - 1.00 1.00 1.00 11 584 1 0 0 1 0
3OVS_D - 0.96 0.92 1.00 12 549 0 0 0 0 1
3P22_G - 0.96 0.92 1.00 11 730 0 0 0 0 1
3PDR_A 0.67 0.56 0.80 40 12830 12 0 10 2 32
3R4F_A - 0.96 0.92 1.00 23 2122 0 0 0 0 2
3R9X_C - 0.84 0.80 0.89 8 586 2 0 1 1 2
3RKF_A 0.76 0.59 1.00 20 2191 0 0 0 0 14
3SD1_A 0.64 0.52 0.79 22 3888 6 1 5 0 20
3SIU_F - 0.85 0.73 1.00 8 370 0 0 0 0 3
3SN2_B 0.96 0.92 1.00 11 395 0 0 0 0 1
3TRZ_Z - 0.91 0.83 1.00 5 205 1 0 0 1 1
3TS0_U - 1.00 1.00 1.00 6 247 1 0 0 1 0
3TS2_V - 1.00 1.00 1.00 5 271 0 0 0 0 0
3U4M_B - 0.38 0.32 0.46 12 3134 14 0 14 0 25
3UZL_B 0.39 0.32 0.48 12 3545 13 2 11 0 25
3VJR_D - 0.96 0.92 1.00 12 618 0 0 0 0 1
3W3S_B 0.55 0.48 0.63 19 4723 12 1 10 1 21
3ZEX_F - 0.00 0.00 0.00 0 2617 11 2 9 0 12
3ZEX_C 0.23 0.21 0.25 11 14152 41 1 32 8 41
3ZEX_G - 0.52 0.45 0.61 33 16417 24 2 19 3 41
3ZEX_D 0.76 0.63 0.91 31 6987 3 0 3 0 18
3ZEX_E - 0.00 0.00 0.00 0 21888 59 3 54 2 77
3ZEX_H - 0.20 0.21 0.21 8 9006 31 5 26 0 30
4A1C_2 0.14 0.15 0.14 5 11746 41 5 25 11 28
4A1C_3 0.68 0.57 0.82 31 7102 7 1 6 0 23
4A4U_A - 1.00 1.00 1.00 9 222 0 0 0 0 0
4AOB_A 0.50 0.40 0.63 17 4344 11 2 8 1 25
4ATO_G - 0.32 0.30 0.38 3 520 5 1 4 0 7
4ENB_A 0.70 0.58 0.85 11 1262 2 1 1 0 8
4ENC_A 0.32 0.26 0.42 5 1314 7 1 6 0 14
4FNJ_A - 0.79 0.63 1.00 10 585 0 0 0 0 6
4FRG_B 0.32 0.28 0.38 9 3462 15 0 15 0 23
4FRN_A 0.52 0.44 0.62 16 5125 10 2 8 0 20
4HXH_A - 1.00 1.00 1.00 6 319 0 0 0 0 0

^top


Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.