CompaRNA - on-line benchmarks of RNA structure prediction methods

FAQ

Metric	RNASampler(20)		NanoFolder
MCC	0.634	>	0.100
Average MCC ± 95% Confidence Intervals	0.664 ± 0.102	>	0.106 ± 0.067
Sensitivity	0.523	>	0.123
Positive Predictive Value	0.771	>	0.084
Total TP	441	>	104
Total TN	519154	>	518481
Total FP	166	<	1176
Total FP CONTRA	48	<	208
Total FP INCONS	83	<	933
Total FP COMP	35	=	35
Total FN	403	<	740
P-value	1.24019667966e-08

Comparison of performance of RNASampler(20) and NanoFolder. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for RNASampler(20) and NanoFolder).
Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for RNASampler(20) and NanoFolder).
Comparison of average Matthews Correlation Coefficients (MCCs) for RNASampler(20) and NanoFolder. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for RNASampler(20) and NanoFolder).

Total Scores
MCC	0.634
Average MCC ± 95% Confidence Intervals	0.664 ± 0.102
Sensitivity	0.523
Positive Predictive Value	0.771
Nr of predictions	10

RNA Chain	Rfam family	MCC	SENS	PPV	TP	TN	FP	FP CONTRA	FP INCONS	FP COMP	FN
PDB_00571	tRNA links to Rfam database...	0.87	0.80	0.95	20	3300	1	1	0	0	5
PDB_00828	Purine links to Rfam database...	0.86	0.74	1.00	20	2465	0	0	0	0	7
TMR_00123	tmRNA links to Rfam database...	0.60	0.46	0.79	46	66372	17	3	9	5	55
TMR_00443	tmRNA links to Rfam database...	0.56	0.41	0.75	43	67104	14	6	8	0	61
TMR_00469	tmRNA links to Rfam database...	0.72	0.61	0.86	61	64549	11	4	6	1	39
TMR_00519	tmRNA links to Rfam database...	0.50	0.40	0.62	38	63129	27	14	9	4	57
TMR_00528	tmRNA links to Rfam database...	0.49	0.40	0.59	39	63124	31	12	15	4	58
TMR_00571	tmRNA links to Rfam database...	0.77	0.71	0.83	70	60642	22	2	12	8	29
TMR_00584	tmRNA links to Rfam database...	0.71	0.65	0.79	63	60995	25	3	14	8	34
TMR_00703	tmRNA links to Rfam database...	0.56	0.41	0.76	41	67474	18	3	10	5	58

Total Scores
MCC	0.100
Average MCC ± 95% Confidence Intervals	0.106 ± 0.067
Sensitivity	0.123
Positive Predictive Value	0.084
Nr of predictions	10

RNA Chain	Rfam family	MCC	SENS	PPV	TP	TN	FP	FP CONTRA	FP INCONS	FP COMP	FN
PDB_00571	tRNA links to Rfam database...	-0.01	0.00	0.00	0	3286	36	6	29	1	25
PDB_00828	Purine links to Rfam database...	0.30	0.33	0.28	9	2453	23	0	23	0	18
TMR_00123	tmRNA links to Rfam database...	0.19	0.23	0.16	23	66283	126	20	104	2	78
TMR_00443	tmRNA links to Rfam database...	0.17	0.20	0.15	21	67018	126	24	98	4	83
TMR_00469	tmRNA links to Rfam database...	0.10	0.12	0.08	12	64469	145	27	112	6	88
TMR_00519	tmRNA links to Rfam database...	0.04	0.05	0.03	5	63046	141	33	106	2	90
TMR_00528	tmRNA links to Rfam database...	0.13	0.15	0.11	15	63051	131	20	104	7	82
TMR_00571	tmRNA links to Rfam database...	0.05	0.06	0.04	6	60577	149	19	124	6	93
TMR_00584	tmRNA links to Rfam database...	0.01	0.01	0.01	1	60922	159	33	119	7	96
TMR_00703	tmRNA links to Rfam database...	0.10	0.12	0.08	12	67376	140	26	114	0	87

Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.