CompaRNA - on-line benchmarks of RNA structure prediction methods

FAQ

Metric	RNASampler(20)		Afold
MCC	0.541	>	0.489
Average MCC ± 95% Confidence Intervals	0.582 ± 0.101	>	0.502 ± 0.126
Sensitivity	0.411	<	0.481
Positive Predictive Value	0.715	>	0.499
Total TP	526	<	616
Total TN	742331	>	741833
Total FP	255	<	728
Total FP CONTRA	34	<	70
Total FP INCONS	176	<	548
Total FP COMP	45	<	110
Total FN	755	>	665
P-value	2.8150742666e-08

Comparison of performance of RNASampler(20) and Afold. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for RNASampler(20) and Afold).
Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for RNASampler(20) and Afold).
Comparison of average Matthews Correlation Coefficients (MCCs) for RNASampler(20) and Afold. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for RNASampler(20) and Afold).

Total Scores
MCC	0.541
Average MCC ± 95% Confidence Intervals	0.582 ± 0.101
Sensitivity	0.411
Positive Predictive Value	0.715
Nr of predictions	15

RNA Chain	Rfam family	MCC	SENS	PPV	TP	TN	FP	FP CONTRA	FP INCONS	FP COMP	FN
ASE_00064	RNaseP_bact_a links to Rfam database...	0.44	0.34	0.58	30	45399	23	1	21	1	59
ASE_00090	RNaseP_bact_a links to Rfam database...	0.54	0.40	0.74	40	55557	14	0	14	0	61
ASE_00135	RNaseP_bact_a links to Rfam database...	0.54	0.36	0.83	39	63143	16	0	8	8	70
ASE_00153	RNaseP_nuc links to Rfam database...	0.38	0.33	0.44	24	57576	38	7	23	8	49
ASE_00215	RNaseP_bact_a links to Rfam database...	0.38	0.23	0.62	23	48479	16	0	14	2	76
ASE_00328	RNaseP_bact_b links to Rfam database...	0.70	0.57	0.85	64	72696	14	5	6	3	48
ASE_00361	RNaseP_bact_a links to Rfam database...	0.34	0.21	0.55	27	75417	26	3	19	4	100
ASE_00441	RNaseP_bact_a links to Rfam database...	0.46	0.31	0.69	35	64210	16	0	16	0	77
PDB_00571	tRNA links to Rfam database...	0.87	0.80	0.95	20	3300	1	1	0	0	5
PDB_00828	Purine links to Rfam database...	0.86	0.74	1.00	20	2465	0	0	0	0	7
PDB_00829	Purine links to Rfam database...	0.86	0.75	1.00	18	2260	0	0	0	0	6
TMR_00046	tmRNA links to Rfam database...	0.48	0.40	0.58	38	62769	34	5	23	6	58
TMR_00048	tmRNA links to Rfam database...	0.61	0.46	0.80	44	64925	16	3	8	5	51
TMR_00458	tmRNA links to Rfam database...	0.55	0.42	0.72	39	63492	15	7	8	0	54
TMR_00568	tmRNA links to Rfam database...	0.72	0.66	0.78	65	60643	26	2	16	8	34

Total Scores
MCC	0.489
Average MCC ± 95% Confidence Intervals	0.502 ± 0.126
Sensitivity	0.481
Positive Predictive Value	0.499
Nr of predictions	15

RNA Chain	Rfam family	MCC	SENS	PPV	TP	TN	FP	FP CONTRA	FP INCONS	FP COMP	FN
ASE_00064	RNaseP_bact_a links to Rfam database...	0.43	0.43	0.44	38	45364	54	5	44	5	51
ASE_00090	RNaseP_bact_a links to Rfam database...	0.46	0.47	0.47	47	55510	59	6	48	5	54
ASE_00135	RNaseP_bact_a links to Rfam database...	0.45	0.44	0.45	48	63084	63	8	50	5	61
ASE_00153	RNaseP_nuc links to Rfam database...	0.66	0.67	0.65	49	57555	68	2	24	42	24
ASE_00215	RNaseP_bact_a links to Rfam database...	0.54	0.51	0.57	50	48429	39	4	33	2	49
ASE_00328	RNaseP_bact_b links to Rfam database...	0.40	0.41	0.40	46	72656	75	3	66	6	66
ASE_00361	RNaseP_bact_a links to Rfam database...	0.51	0.49	0.53	62	75349	59	7	48	4	65
ASE_00441	RNaseP_bact_a links to Rfam database...	0.77	0.72	0.83	81	64163	30	1	16	13	31
PDB_00571	tRNA links to Rfam database...	-0.01	0.00	0.00	0	3302	20	5	14	1	25
PDB_00828	Purine links to Rfam database...	0.88	0.78	1.00	21	2464	0	0	0	0	6
PDB_00829	Purine links to Rfam database...	0.82	0.71	0.94	17	2260	1	0	1	0	7
TMR_00046	tmRNA links to Rfam database...	0.47	0.48	0.46	46	62734	61	10	45	6	50
TMR_00048	tmRNA links to Rfam database...	0.35	0.37	0.34	35	64876	73	9	60	4	60
TMR_00458	tmRNA links to Rfam database...	0.22	0.22	0.23	20	63458	76	7	61	8	73
TMR_00568	tmRNA links to Rfam database...	0.57	0.57	0.58	56	60629	50	3	38	9	43

Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.