CompaRNA - on-line benchmarks of RNA structure prediction methods

FAQ

Metric	PETfold_pre2.0(20)		MXScarna(seed)
MCC	0.642	>	0.595
Average MCC ± 95% Confidence Intervals	0.620 ± 0.120	>	0.562 ± 0.107
Sensitivity	0.524	>	0.476
Positive Predictive Value	0.791	>	0.748
Total TP	345	>	314
Total TN	107984	<	108000
Total FP	119	<	151
Total FP CONTRA	4	<	16
Total FP INCONS	87	<	90
Total FP COMP	28	<	45
Total FN	314	<	345
P-value	3.27388928538e-08

Comparison of performance of PETfold_pre2.0(20) and MXScarna(seed). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for PETfold_pre2.0(20) and MXScarna(seed)).
Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for PETfold_pre2.0(20) and MXScarna(seed)).
Comparison of average Matthews Correlation Coefficients (MCCs) for PETfold_pre2.0(20) and MXScarna(seed). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for PETfold_pre2.0(20) and MXScarna(seed)).

Total Scores
MCC	0.642
Average MCC ± 95% Confidence Intervals	0.620 ± 0.120
Sensitivity	0.524
Positive Predictive Value	0.791
Nr of predictions	17

RNA Chain	Rfam family	MCC	SENS	PPV	TP	TN	FP	FP CONTRA	FP INCONS	FP COMP	FN
3J20_1	tRNA links to Rfam database...	0.96	0.91	1.00	21	2905	0	0	0	0	2
3J20_0	tRNA links to Rfam database...	0.84	0.70	1.00	21	2829	0	0	0	0	9
3J2L_3	5S_rRNA links to Rfam database...	0.78	0.62	0.97	33	7841	3	0	1	2	20
3J3E_8	5_8S_rRNA links to Rfam database...	0.32	0.21	0.50	7	7489	11	0	7	4	26
3J3E_7	5S_rRNA links to Rfam database...	0.74	0.63	0.87	34	7101	5	1	4	0	20
3J3F_8	5_8S_rRNA links to Rfam database...	0.30	0.25	0.38	9	12222	22	0	15	7	27
3J3F_7	5S_rRNA links to Rfam database...	0.78	0.68	0.89	34	7222	5	1	3	1	16
3J3V_B	5S_rRNA links to Rfam database...	0.65	0.51	0.83	29	6986	6	1	5	0	28
3ZEX_D	5S_rRNA links to Rfam database...	0.81	0.71	0.92	35	6983	3	0	3	0	14
3ZEX_C	5_8S_rRNA links to Rfam database...	0.24	0.17	0.35	9	14170	18	0	17	1	43
3ZND_W	tRNA links to Rfam database...	0.47	0.43	0.53	10	2984	11	0	9	2	13
4A1C_2	5_8S_rRNA links to Rfam database...	0.19	0.15	0.25	5	11761	25	0	15	10	28
4A1C_3	5S_rRNA links to Rfam database...	0.82	0.69	0.97	37	7102	1	0	1	0	17
4AOB_A	SAM links to Rfam database...	0.74	0.60	0.93	25	4344	3	0	2	1	17
4ENB_A	crcB links to Rfam database...	0.61	0.42	0.89	8	1266	1	0	1	0	11
4ENC_A	crcB links to Rfam database...	0.55	0.42	0.73	8	1315	3	0	3	0	11
4FRG_B	AdoCbl-variant links to Rfam database...	0.75	0.63	0.91	20	3464	2	1	1	0	12

Total Scores
MCC	0.595
Average MCC ± 95% Confidence Intervals	0.562 ± 0.107
Sensitivity	0.476
Positive Predictive Value	0.748
Nr of predictions	17

RNA Chain	Rfam family	MCC	SENS	PPV	TP	TN	FP	FP CONTRA	FP INCONS	FP COMP	FN
3J20_1	tRNA links to Rfam database...	0.93	0.87	1.00	20	2906	0	0	0	0	3
3J20_0	tRNA links to Rfam database...	0.79	0.63	1.00	19	2831	0	0	0	0	11
3J2L_3	5S_rRNA links to Rfam database...	0.60	0.49	0.74	26	7840	11	1	8	2	27
3J3E_8	5_8S_rRNA links to Rfam database...	0.16	0.09	0.27	3	7492	14	1	7	6	30
3J3E_7	5S_rRNA links to Rfam database...	0.70	0.59	0.82	32	7101	7	2	5	0	22
3J3F_8	5_8S_rRNA links to Rfam database...	0.45	0.33	0.60	12	12226	18	1	7	10	24
3J3F_7	5S_rRNA links to Rfam database...	0.78	0.70	0.88	35	7220	6	2	3	1	15
3J3V_B	5S_rRNA links to Rfam database...	0.56	0.46	0.70	26	6984	11	1	10	0	31
3ZEX_D	5S_rRNA links to Rfam database...	0.76	0.69	0.83	34	6980	7	1	6	0	15
3ZEX_C	5_8S_rRNA links to Rfam database...	0.32	0.19	0.53	10	14177	20	2	7	11	42
3ZND_W	tRNA links to Rfam database...	0.44	0.39	0.50	9	2985	11	0	9	2	14
4A1C_2	5_8S_rRNA links to Rfam database...	0.35	0.24	0.50	8	11765	20	1	7	12	25
4A1C_3	5S_rRNA links to Rfam database...	0.75	0.65	0.88	35	7100	5	1	4	0	19
4AOB_A	SAM links to Rfam database...	0.66	0.55	0.79	23	4342	7	0	6	1	19
4ENB_A	crcB links to Rfam database...	0.48	0.32	0.75	6	1267	2	0	2	0	13
4ENC_A	crcB links to Rfam database...	0.43	0.32	0.60	6	1316	4	1	3	0	13
4FRG_B	AdoCbl-variant links to Rfam database...	0.41	0.31	0.56	10	3468	8	2	6	0	22

Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.