CompaRNA - on-line benchmarks of RNA structure prediction methods

FAQ

Metric	RNASampler(20)		NanoFolder
MCC	0.602	>	0.121
Average MCC ± 95% Confidence Intervals	0.584 ± 0.090	>	0.124 ± 0.062
Sensitivity	0.450	>	0.148
Positive Predictive Value	0.805	>	0.102
Total TP	587	>	193
Total TN	752688	>	751519
Total FP	192	<	1749
Total FP CONTRA	48	<	319
Total FP INCONS	94	<	1386
Total FP COMP	50	>	44
Total FN	716	<	1110
P-value	2.41358941668e-08

Comparison of performance of RNASampler(20) and NanoFolder. Positive Predictive Value (PPV) is plotted against sensitivity. Each dot represents a single test of each method. See tables below for raw data (individual counts for RNASampler(20) and NanoFolder).
Average Matthews Correlation Coefficients (MCC) with 95% confidence intervals (CIs) were plotted for different RNA families, for which at least 3 members were present in the benchmarking dataset. 'n' denotes the number of MCCs used to calculate the average and CI. See tables below for raw data (individual counts for RNASampler(20) and NanoFolder).
Comparison of average Matthews Correlation Coefficients (MCCs) for RNASampler(20) and NanoFolder. The whiskers correspond to 95% confidence intervals (CIs). 'n' denotes the number of MCCs used to calculate average MCCs and CIs. See tables below for raw data (individual counts for RNASampler(20) and NanoFolder).

Total Scores
MCC	0.602
Average MCC ± 95% Confidence Intervals	0.584 ± 0.090
Sensitivity	0.450
Positive Predictive Value	0.805
Nr of predictions	14

RNA Chain	Rfam family	MCC	SENS	PPV	TP	TN	FP	FP CONTRA	FP INCONS	FP COMP	FN
ASE_00427	RNaseP_nuc links to Rfam database...	0.57	0.46	0.72	21	40726	18	1	7	10	25
SRP_00103	Metazoa_SRP links to Rfam database...	0.80	0.64	1.00	65	45386	0	0	0	0	37
SRP_00234	Bacteria_large_SRP links to Rfam database...	0.47	0.23	0.95	20	36294	1	0	1	0	66
SRP_00308	Bacteria_large_SRP links to Rfam database...	0.28	0.08	1.00	7	36308	0	0	0	0	81
SRP_00317	Metazoa_SRP links to Rfam database...	0.76	0.61	0.95	60	45087	8	0	3	5	38
SRP_00337	Bacteria_large_SRP links to Rfam database...	0.38	0.14	1.00	13	35498	0	0	0	0	78
TMR_00123	tmRNA links to Rfam database...	0.60	0.46	0.79	46	66372	17	3	9	5	55
TMR_00443	tmRNA links to Rfam database...	0.56	0.41	0.75	43	67104	14	6	8	0	61
TMR_00469	tmRNA links to Rfam database...	0.72	0.61	0.86	61	64549	11	4	6	1	39
TMR_00519	tmRNA links to Rfam database...	0.50	0.40	0.62	38	63129	27	14	9	4	57
TMR_00528	tmRNA links to Rfam database...	0.49	0.40	0.59	39	63124	31	12	15	4	58
TMR_00571	tmRNA links to Rfam database...	0.77	0.71	0.83	70	60642	22	2	12	8	29
TMR_00584	tmRNA links to Rfam database...	0.71	0.65	0.79	63	60995	25	3	14	8	34
TMR_00703	tmRNA links to Rfam database...	0.56	0.41	0.76	41	67474	18	3	10	5	58

Total Scores
MCC	0.121
Average MCC ± 95% Confidence Intervals	0.124 ± 0.062
Sensitivity	0.148
Positive Predictive Value	0.102
Nr of predictions	14

RNA Chain	Rfam family	MCC	SENS	PPV	TP	TN	FP	FP CONTRA	FP INCONS	FP COMP	FN
ASE_00427	RNaseP_nuc links to Rfam database...	0.06	0.09	0.04	4	40651	105	35	65	5	42
SRP_00103	Metazoa_SRP links to Rfam database...	0.00	0.00	0.00	0	45319	132	14	118	0	102
SRP_00234	Bacteria_large_SRP links to Rfam database...	0.21	0.24	0.18	21	36200	97	19	75	3	65
SRP_00308	Bacteria_large_SRP links to Rfam database...	0.14	0.17	0.12	15	36189	112	18	93	1	73
SRP_00317	Metazoa_SRP links to Rfam database...	0.14	0.16	0.12	16	45018	116	16	100	0	82
SRP_00337	Bacteria_large_SRP links to Rfam database...	0.42	0.46	0.38	42	35400	70	15	54	1	49
TMR_00123	tmRNA links to Rfam database...	0.19	0.23	0.16	23	66283	126	20	104	2	78
TMR_00443	tmRNA links to Rfam database...	0.17	0.20	0.15	21	67018	126	24	98	4	83
TMR_00469	tmRNA links to Rfam database...	0.10	0.12	0.08	12	64469	145	27	112	6	88
TMR_00519	tmRNA links to Rfam database...	0.04	0.05	0.03	5	63046	141	33	106	2	90
TMR_00528	tmRNA links to Rfam database...	0.13	0.15	0.11	15	63051	131	20	104	7	82
TMR_00571	tmRNA links to Rfam database...	0.05	0.06	0.04	6	60577	149	19	124	6	93
TMR_00584	tmRNA links to Rfam database...	0.01	0.01	0.01	1	60922	159	33	119	7	96
TMR_00703	tmRNA links to Rfam database...	0.10	0.12	0.08	12	67376	140	26	114	0	87

Matthews Correlation Coeffient, Sensitivity and Positive Predictive Value have been calculated based on the paper by Gardener & Giegerich, 2004.