CompaRNA - on-line benchmarks of RNA structure prediction methods

FAQ

Metric	CentroidAlifold(seed)		RNASampler(20)
MCC	0.702	>	0.526
Average MCC ± 95% Confidence Intervals	0.743 ± 0.088	>	0.595 ± 0.129
Sensitivity	0.526	>	0.384
Positive Predictive Value	0.939	>	0.723
Total TP	489	>	357
Total TN	493787	<	493814
Total FP	40	<	165
Total FP CONTRA	2	<	16
Total FP INCONS	30	<	121
Total FP COMP	8	<	28
Total FN	441	<	573
P-value	1.5677163434e-08

Comparison of performance of CentroidAlifold(seed) and RNASampler(20). Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for CentroidAlifold(seed) and RNASampler(20)).
Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for CentroidAlifold(seed) and RNASampler(20)).
Comparison of average Matthews Correlation Coefficients (MCCs) for CentroidAlifold(seed) and RNASampler(20). The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for CentroidAlifold(seed) and RNASampler(20)).

Total Scores
MCC	0.702
Average MCC ± 95% Confidence Intervals	0.743 ± 0.088
Sensitivity	0.526
Positive Predictive Value	0.939
Nr of predictions	12

RNA Chain	Rfam family	MCC	SENS	PPV	TP	TN	FP	FP CONTRA	FP INCONS	FP COMP	FN
ASE_00064	RNaseP_bact_a links to Rfam database...	0.77	0.60	1.00	53	45398	0	0	0	0	36
ASE_00090	RNaseP_bact_a links to Rfam database...	0.75	0.56	1.00	57	55554	0	0	0	0	44
ASE_00135	RNaseP_bact_a links to Rfam database...	0.53	0.39	0.74	42	63133	15	0	15	0	67
ASE_00153	RNaseP_nuc links to Rfam database...	0.67	0.53	0.85	39	57584	7	1	6	0	34
ASE_00215	RNaseP_bact_a links to Rfam database...	0.51	0.32	0.82	32	48477	7	0	7	0	67
ASE_00328	RNaseP_bact_b links to Rfam database...	0.83	0.70	1.00	78	72693	5	0	0	5	34
ASE_00361	RNaseP_bact_a links to Rfam database...	0.65	0.43	0.98	55	75410	1	1	0	0	72
ASE_00441	RNaseP_bact_a links to Rfam database...	0.65	0.44	0.96	49	64210	3	0	2	1	63
PDB_00828	Purine links to Rfam database...	0.88	0.78	1.00	21	2464	0	0	0	0	6
PDB_00829	Purine links to Rfam database...	0.91	0.83	1.00	20	2258	0	0	0	0	4
PDB_01020	Purine links to Rfam database...	0.91	0.83	1.00	19	2259	1	0	0	1	4
PDB_01073	SAM links to Rfam database...	0.84	0.71	1.00	24	4347	1	0	0	1	10

Total Scores
MCC	0.526
Average MCC ± 95% Confidence Intervals	0.595 ± 0.129
Sensitivity	0.384
Positive Predictive Value	0.723
Nr of predictions	12

RNA Chain	Rfam family	MCC	SENS	PPV	TP	TN	FP	FP CONTRA	FP INCONS	FP COMP	FN
ASE_00064	RNaseP_bact_a links to Rfam database...	0.44	0.34	0.58	30	45399	23	1	21	1	59
ASE_00090	RNaseP_bact_a links to Rfam database...	0.54	0.40	0.74	40	55557	14	0	14	0	61
ASE_00135	RNaseP_bact_a links to Rfam database...	0.54	0.36	0.83	39	63143	16	0	8	8	70
ASE_00153	RNaseP_nuc links to Rfam database...	0.38	0.33	0.44	24	57576	38	7	23	8	49
ASE_00215	RNaseP_bact_a links to Rfam database...	0.38	0.23	0.62	23	48479	16	0	14	2	76
ASE_00328	RNaseP_bact_b links to Rfam database...	0.70	0.57	0.85	64	72696	14	5	6	3	48
ASE_00361	RNaseP_bact_a links to Rfam database...	0.34	0.21	0.55	27	75417	26	3	19	4	100
ASE_00441	RNaseP_bact_a links to Rfam database...	0.46	0.31	0.69	35	64210	16	0	16	0	77
PDB_00828	Purine links to Rfam database...	0.86	0.74	1.00	20	2465	0	0	0	0	7
PDB_00829	Purine links to Rfam database...	0.86	0.75	1.00	18	2260	0	0	0	0	6
PDB_01020	Purine links to Rfam database...	0.86	0.74	1.00	17	2261	1	0	0	1	6
PDB_01073	SAM links to Rfam database...	0.77	0.59	1.00	20	4351	1	0	0	1	14

Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.