clarin-pl/combo-nlp-models

COMBO vs Trankit vs Stanza — UD Evaluation Report

42 treebanks · F1 scores · COMBO on UD 2.17 · Trankit on UD 2.5 · Stanza on UD 2.12 · highest lowest

Treebank	System	Tokens	Sents	Words	UPOS	XPOS	UFeats	AllTags	Lemmas	UAS	LAS	CLAS	MLAS	BLEX
Afrikaans AFRIBOOMS	COMBO[base]	99.87	99.29	99.87	98.75	96.03	98.26	95.68	98.41	90.09	86.87	81.45	77.26	79.36
	Trankit[large]	99.84	100.00	99.84	98.87	96.63	98.68	96.54	97.47	91.61	89.35	85.30	82.64	81.65
	Trankit[base]	99.72	100.00	99.72	98.55	96.10	98.26	95.89	97.39	91.03	88.79	84.46	81.31	80.91
	Stanza	99.99	99.65	99.99	98.60	95.77	98.41	95.66	98.29	89.54	86.14	80.10	76.71	77.53
Ancient Greek PROIEL	COMBO[base]	99.90	67.37	99.90	97.53	97.66	90.15	89.09	97.47	85.14	81.63	75.69	64.85	73.51
	Trankit[large]	99.91	67.60	99.91	97.86	97.93	93.03	92.02	97.50	85.63	82.31	78.16	68.27	75.76
	Trankit[base]	99.83	58.62	99.83	97.30	97.37	91.59	90.33	97.31	83.21	79.68	74.96	64.13	72.80
	Stanza	100.00	52.00	100.00	97.42	97.82	92.30	91.06	97.18	82.77	79.02	72.47	63.94	70.00
Ancient Hebrew PTNK	COMBO[base]	99.50	94.60	88.82	86.81	86.96	85.00	84.05	87.32	74.99	72.91	65.15	59.40	63.47
	Stanza	100.00	98.18	93.60	90.20	90.53	85.40	84.63	92.14	76.12	72.72	63.10	54.54	61.98
Arabic PADT	COMBO[base]	99.78	73.33	97.39	94.75	92.50	92.59	92.01	93.03	83.86	80.76	78.77	73.01	74.72
	Trankit[large]	99.95	96.79	99.39	96.80	94.93	94.99	94.61	94.92	89.74	85.98	83.92	78.47	79.96
	Trankit[base]	99.93	96.59	99.22	96.31	94.08	94.28	93.69	94.65	88.39	84.68	82.35	76.46	78.46
	Stanza	99.98	83.56	98.03	95.18	92.40	92.52	92.18	93.54	83.99	79.77	76.72	71.10	73.25
Armenian ARMTDP	COMBO[base]	97.64	88.89	96.94	95.20	96.94	89.25	88.72	94.52	87.23	83.86	81.47	71.76	79.15
	Trankit[large]	98.52	99.46	97.26	95.03	97.26	90.42	89.56	93.45	86.47	83.01	80.91	72.45	77.07
	Trankit[base]	98.41	98.92	97.23	94.36	97.23	88.89	87.84	93.28	84.22	80.14	77.47	67.92	73.94
	Stanza	99.92	99.46	99.48	83.05	99.48	66.22	64.30	91.35	74.04	61.80	52.52	29.58	49.05
Basque BDT	COMBO[base]	99.86	99.92	99.86	96.34	99.86	89.08	87.42	96.82	88.94	85.73	83.71	70.52	80.77
	Trankit[large]	99.89	100.00	99.89	96.95	99.89	93.96	92.30	96.69	90.22	87.47	86.54	78.10	83.21
	Trankit[base]	99.81	99.92	99.81	96.23	99.81	92.35	90.40	96.46	88.24	85.25	84.29	74.20	80.92
	Stanza	99.98	100.00	99.98	96.07	99.98	92.95	91.16	96.38	86.65	83.20	80.80	73.42	77.71
Bulgarian BTB	COMBO[base]	99.80	96.02	99.80	99.30	97.68	98.11	97.19	98.44	95.74	93.52	91.29	88.41	89.33
	Trankit[large]	99.78	98.79	99.78	99.29	97.75	98.30	97.29	97.37	96.30	94.21	92.19	89.64	88.79
	Trankit[base]	99.84	98.25	99.84	99.18	97.42	98.10	96.91	97.36	95.81	93.47	91.23	88.28	87.80
	Stanza	99.96	96.50	99.96	99.16	96.97	97.84	96.62	97.28	93.98	91.09	88.00	84.88	84.59
Catalan ANCORA	COMBO[base]	99.80	99.76	99.78	98.96	97.04	98.40	96.65	99.34	94.67	93.29	90.21	87.72	89.71
	Trankit[large]	99.94	99.76	99.94	99.11	99.07	98.66	98.24	98.49	95.15	93.83	91.41	89.31	90.27
	Trankit[base]	99.94	100.00	99.93	99.02	98.97	98.57	98.09	98.46	94.61	93.01	90.09	87.89	88.99
	Stanza	99.97	99.57	99.95	98.70	95.93	98.27	95.67	99.06	93.55	91.66	87.24	84.72	86.41
Chinese GSD	COMBO[base]	80.00	97.49	80.00	77.66	77.79	79.51	77.25	79.44	58.59	56.88	52.54	49.14	51.93
	Trankit[large]	97.75	99.40	97.75	95.13	94.94	97.28	94.45	97.75	87.38	84.82	83.44	79.82	83.44
	Trankit[base]	97.01	99.70	97.01	94.21	94.02	96.59	93.56	97.01	85.19	82.54	80.91	77.42	80.91
Chinese GSDSIMP	COMBO[base]	86.66	97.49	86.66	84.19	84.20	86.14	83.69	86.08	68.02	66.05	61.81	58.51	61.22
	Trankit[large]	97.66	98.20	97.66	94.99	94.78	97.24	94.32	97.66	87.01	84.63	83.23	79.84	83.23
	Trankit[base]	96.94	99.70	96.94	94.17	93.98	96.51	93.52	96.94	84.64	81.96	80.14	76.30	80.14
	Stanza	92.14	99.10	92.14	88.82	88.61	91.47	87.95	92.07	73.41	70.65	67.39	63.96	67.35
Classical Chinese KYOTO	COMBO[base]	97.17	86.54	97.17	90.57	89.44	91.73	87.15	10.97	80.80	76.06	75.36	72.19	8.19
	Trankit[large]	99.70	70.58	99.70	92.89	91.90	94.43	89.98	99.67	82.03	76.82	75.60	73.16	75.60
	Trankit[base]	99.63	61.82	99.63	92.07	91.03	93.88	88.92	99.60	78.79	73.23	72.18	69.27	72.18
	Stanza	97.85	49.57	97.85	89.16	88.17	91.38	85.89	97.63	71.32	65.80	64.54	61.94	64.43
Coptic SCRIPTORIUM	COMBO[base]	99.36	47.58	78.99	15.46	10.15	43.29	1.32	17.49	1.45	0.39	1.12	0.03	0.00
	Stanza	100.00	34.09	88.63	85.77	85.69	86.37	84.77	85.95	66.37	64.43	56.39	53.08	55.19
Croatian SET	COMBO[base]	99.80	95.78	99.80	98.52	96.17	96.54	95.70	97.85	93.13	90.29	88.43	82.87	85.91
	Trankit[large]	99.93	99.08	99.93	98.58	96.55	96.85	95.99	96.71	93.86	89.74	87.77	82.19	83.56
	Trankit[base]	99.92	99.16	99.92	98.38	96.08	96.52	95.44	96.60	93.34	89.36	87.16	81.12	82.91
	Stanza	99.96	97.08	99.96	98.21	94.89	95.27	94.60	96.60	90.29	86.95	84.14	78.19	80.25
Czech CAC	COMBO[base]	99.78	98.10	99.72	99.34	97.55	96.52	96.12	98.67	94.72	93.36	92.17	87.49	90.86
	Trankit[large]	99.99	100.00	99.98	99.64	98.17	98.01	97.65	98.30	95.48	93.97	93.06	90.13	90.92
	Trankit[base]	99.96	100.00	99.95	99.42	97.40	97.12	96.68	98.13	94.97	93.27	92.08	88.13	89.85
	Stanza	99.82	100.00	99.57	98.52	94.59	93.64	92.92	97.46	91.09	88.71	86.78	80.27	84.87
Czech CLTT	COMBO[base]	88.05	60.17	88.06	86.98	77.81	77.75	75.73	83.31	72.70	70.76	72.89	61.07	68.61
	Trankit[large]	99.89	98.89	99.85	99.17	93.80	94.02	93.53	97.81	92.53	90.76	88.90	81.07	86.35
	Trankit[base]	99.82	100.00	99.76	98.93	93.36	93.68	93.21	97.66	90.20	88.01	85.13	77.66	82.80
	Stanza	99.23	98.23	99.23	97.38	88.25	88.22	87.42	95.78	83.53	79.71	76.52	66.32	74.12
Czech FICTREE	COMBO[base]	99.68	94.11	99.59	98.65	96.63	97.10	96.03	98.69	93.86	92.45	91.34	87.09	90.12
	Trankit[large]	99.98	99.50	99.98	99.11	97.26	97.99	97.06	98.64	96.50	94.85	93.44	89.79	91.46
	Trankit[base]	99.97	99.38	99.97	98.94	96.47	97.09	96.12	98.61	95.85	93.86	92.10	87.13	90.16
	Stanza	99.98	99.07	99.97	98.45	95.15	96.00	94.77	98.32	93.28	90.80	88.55	82.85	86.53
Danish DDT	COMBO[base]	99.64	93.50	99.64	98.37	99.64	96.97	96.55	97.94	89.27	87.60	85.44	80.31	83.39
	Trankit[large]	99.81	95.46	99.81	98.71	99.81	98.33	97.87	97.35	91.75	90.33	88.81	85.66	85.42
	Trankit[base]	99.79	95.19	99.79	98.35	99.79	97.79	97.19	97.18	90.41	88.78	86.94	82.58	83.38
	Stanza	99.97	91.81	99.97	98.30	99.97	97.87	97.14	97.88	88.61	86.61	83.80	80.06	81.27
Dutch ALPINO	COMBO[base]	99.67	90.86	99.67	97.70	96.67	97.43	96.03	95.22	93.63	91.96	89.24	85.14	82.63
	Trankit[large]	99.43	90.65	99.43	96.67	95.01	96.55	94.75	96.39	94.41	92.49	89.56	84.22	85.29
	Trankit[base]	99.22	89.88	99.22	96.55	94.92	96.22	94.56	96.23	93.28	91.28	87.88	82.58	83.86
	Stanza	99.87	90.08	99.87	96.75	95.50	97.00	94.97	95.33	91.42	88.67	83.94	79.65	78.12
Dutch LASSYSMALL	COMBO[base]	99.73	76.73	99.73	97.93	97.06	97.68	96.59	96.00	92.26	90.45	86.86	83.16	81.45
	Trankit[large]	99.36	92.60	99.36	96.52	95.57	96.63	94.99	97.37	92.25	89.68	86.51	82.56	84.19
	Trankit[base]	99.21	91.09	99.21	96.20	95.18	96.25	94.53	97.19	91.09	88.18	84.63	80.47	82.44
	Stanza	99.87	78.61	99.87	96.67	95.92	96.83	95.20	95.86	88.25	85.32	80.00	76.72	75.15
English ATIS	COMBO[base]	99.48	7.90	99.48	98.26	99.48	97.34	96.79	99.11	64.87	63.57	55.72	48.78	55.46
	Stanza	100.00	87.55	100.00	99.15	100.00	98.74	98.19	99.65	94.45	92.61	90.08	87.49	89.72
English EWT	COMBO[base]	98.86	83.79	98.57	96.59	96.20	96.77	95.39	96.85	91.40	89.80	87.27	84.32	85.46
	Trankit[large]	98.67	90.49	98.67	96.65	96.47	96.75	95.49	97.25	91.29	89.40	87.41	83.45	85.73
	Trankit[base]	98.48	88.35	98.48	95.95	95.71	96.26	94.62	96.84	90.14	87.96	85.75	81.02	83.83
	Stanza	99.17	89.16	98.92	96.20	95.75	96.39	94.64	96.63	88.90	86.77	84.11	80.27	81.54
Estonian EDT	COMBO[base]	99.78	92.32	99.78	97.88	98.30	96.72	95.85	96.77	91.00	88.92	88.07	84.03	84.71
	Trankit[large]	99.75	96.58	99.75	97.87	98.35	97.10	96.04	96.09	91.71	89.52	88.57	84.68	84.10
	Trankit[base]	99.72	96.55	99.72	97.53	98.13	96.56	95.37	95.98	90.65	88.31	87.15	82.82	82.81
	Stanza	99.97	92.88	99.97	97.19	97.94	95.64	94.39	95.93	86.60	83.81	82.23	77.02	78.05
Finnish TDT	COMBO[base]	99.54	91.69	99.35	97.82	98.33	96.41	95.77	95.78	92.82	91.22	90.49	86.20	86.39
	Trankit[large]	99.71	97.22	99.72	98.48	98.78	96.84	96.33	95.59	94.98	93.77	92.92	88.97	88.02
	Trankit[base]	99.62	95.98	99.62	97.99	98.44	96.52	95.76	95.39	93.47	91.94	90.78	86.55	86.00
	Stanza	99.73	90.70	99.70	97.51	98.09	96.13	95.39	95.98	90.37	88.31	87.15	83.01	83.42
French GSD	COMBO[base]	99.35	96.52	99.01	97.80	99.01	97.33	97.04	98.01	93.93	92.48	89.61	86.69	88.29
	Trankit[large]	99.77	98.80	99.72	97.89	99.72	97.34	96.66	97.90	94.97	93.37	90.17	86.35	87.73
	Trankit[base]	99.70	96.63	99.66	97.85	99.66	97.16	96.60	97.80	94.00	92.34	88.66	84.76	86.08
	Stanza	99.71	95.03	99.50	97.63	99.50	97.74	97.05	98.20	93.27	91.28	87.44	84.45	85.78
German GSD	COMBO[base]	99.48	83.63	99.47	96.77	97.41	91.41	89.03	97.45	87.68	84.78	81.04	68.63	78.29
	Trankit[large]	99.71	89.72	99.72	95.23	97.68	91.68	87.21	96.58	89.01	85.20	81.49	65.82	77.20
	Trankit[base]	99.75	92.72	99.75	95.04	97.57	91.51	86.86	96.60	88.73	84.77	80.78	64.76	76.58
	Stanza	99.62	83.84	99.60	95.61	97.36	89.78	87.19	97.23	85.80	81.80	77.08	62.81	74.04
Hebrew HTB	COMBO[base]	99.65	99.59	56.55	55.11	55.09	53.23	52.94	54.94	39.57	37.97	37.82	30.34	36.34
	Trankit[large]	99.81	99.69	96.31	94.32	94.32	93.03	92.39	93.48	88.41	86.04	82.23	74.92	78.35
	Trankit[base]	99.79	100.00	96.03	93.75	93.75	91.96	91.24	93.21	87.02	84.55	80.34	72.38	76.52
	Stanza	99.98	100.00	92.29	89.99	89.99	88.68	88.03	89.71	79.38	77.18	71.95	65.35	68.81
Icelandic ICEPAHC	COMBO[base]	99.77	94.33	99.71	97.00	93.78	90.62	86.51	96.55	88.52	85.16	80.38	67.52	77.29
	Stanza	99.90	93.07	99.84	96.36	92.27	90.42	85.48	95.78	86.79	82.85	77.27	64.78	73.63
Irish IDT	COMBO[base]	99.68	97.92	99.68	96.01	94.57	85.40	82.44	95.34	87.13	81.72	76.87	59.21	72.35
	Trankit[large]	99.47	98.24	99.47	94.72	93.74	80.90	77.94	92.64	83.47	76.86	70.64	48.55	64.06
	Trankit[base]	99.32	97.25	99.32	93.88	92.46	80.36	76.72	92.34	82.52	74.91	67.96	46.29	61.34
	Stanza	99.75	97.24	99.75	96.01	94.55	89.41	86.52	95.32	86.81	81.51	76.34	63.38	71.94
Italian ISDT	COMBO[base]	99.77	99.07	99.64	98.46	98.38	97.89	97.47	98.41	95.02	93.74	90.89	87.63	88.96
	Trankit[large]	99.88	99.07	99.86	98.72	98.63	98.32	97.79	98.33	95.73	94.45	91.97	89.08	89.45
	Trankit[base]	99.88	98.76	99.87	98.58	98.46	98.20	97.60	98.23	95.31	93.87	90.93	87.81	88.45
	Stanza	99.93	99.07	99.79	98.41	98.32	98.14	97.69	98.23	93.29	91.61	87.32	85.05	85.17
Japanese GSD	COMBO[base]	95.78	94.95	95.78	94.86	94.05	95.77	93.76	94.79	86.78	85.86	80.75	78.90	79.48
	Trankit[large]	95.25	95.88	95.25	93.66	93.47	95.23	93.44	94.68	86.67	85.56	78.00	76.02	77.62
	Trankit[base]	94.57	95.49	94.57	92.86	92.44	94.56	92.42	93.99	84.58	83.38	75.60	73.67	75.14
	Stanza	97.37	99.72	97.37	96.38	95.72	97.36	95.45	96.02	90.01	89.29	84.99	83.86	83.34
Korean KAIST	COMBO[base]	99.91	99.93	99.91	96.58	89.25	99.91	89.06	93.16	90.16	88.22	86.96	84.47	80.66
	Trankit[large]	98.70	99.87	98.70	95.13	88.07	98.70	88.07	92.36	90.00	88.22	86.37	83.56	80.16
	Trankit[base]	98.79	99.14	98.79	94.99	87.62	98.79	87.62	92.44	88.72	86.96	84.99	81.84	78.90
	Stanza	100.00	99.93	100.00	95.74	85.81	100.00	85.81	94.10	87.63	85.50	83.65	80.93	78.78
Latin ITTB	COMBO[base]	99.86	94.27	99.85	99.12	96.95	97.72	96.15	99.29	91.54	89.94	88.30	84.80	87.91
	Trankit[large]	100.00	94.54	100.00	98.97	97.29	97.98	96.41	99.13	93.25	91.87	90.75	87.86	90.00
	Trankit[base]	100.00	94.57	100.00	98.76	96.74	97.54	95.68	99.07	92.42	90.91	89.45	86.12	88.71
	Stanza	99.99	80.66	99.99	98.76	95.72	96.69	94.63	99.09	88.82	86.80	85.71	81.47	85.31
Lithuanian ALKSNIS	COMBO[base]	99.93	91.51	99.93	97.29	93.12	93.73	91.61	95.06	87.70	84.10	82.49	75.23	77.58
	Trankit[large]	99.84	95.72	99.84	97.45	93.98	94.46	93.30	94.30	90.48	87.67	86.66	79.86	80.20
	Trankit[base]	99.82	95.10	99.82	97.03	92.35	93.00	91.54	94.05	88.30	84.96	83.59	75.11	77.35
	Stanza	99.94	89.93	99.94	93.71	86.48	88.16	85.54	92.95	77.82	72.75	70.23	60.53	65.21
Norwegian NYNORSK	COMBO[base]	99.73	94.81	99.73	98.42	99.01	96.35	95.72	98.47	94.01	92.76	90.52	85.42	88.89
	Trankit[large]	99.84	98.97	99.84	98.52	99.84	97.79	97.13	98.01	95.23	93.82	92.35	89.01	89.79
	Trankit[base]	99.81	98.71	99.81	98.20	99.81	97.20	96.48	97.89	94.15	92.58	90.70	86.61	88.15
	Stanza	99.97	95.48	99.97	98.13	98.91	96.87	95.93	97.87	91.63	89.94	86.84	82.33	84.30
Persian PERDT	COMBO[base]	99.77	99.83	99.51	97.52	97.51	97.68	95.74	99.02	94.47	92.53	90.99	87.56	90.44
	Stanza	99.96	99.73	99.46	97.17	97.14	97.31	95.35	98.55	92.90	90.80	88.87	85.48	87.83
Polish PDB	COMBO[base]	99.42	98.26	99.36	98.63	95.82	95.77	94.93	97.61	95.15	93.62	92.45	87.36	90.29
	Trankit[large]	99.93	98.71	99.92	99.16	96.92	97.11	96.54	97.56	96.43	94.88	93.77	89.78	90.53
	Trankit[base]	99.91	98.53	99.89	99.06	96.29	96.44	95.77	97.52	95.52	93.86	92.50	87.67	89.34
	Stanza	99.85	98.19	99.83	98.63	94.59	94.78	93.89	97.47	93.73	91.73	89.82	83.50	86.95
Portuguese CINTIL	COMBO[base]	99.85	83.02	99.66	97.66	96.29	95.70	93.55	97.33	85.44	82.42	78.02	71.81	75.68
	Stanza	100.00	85.34	99.93	97.19	95.74	95.48	92.94	98.56	84.08	80.09	74.75	68.90	73.19
Romanian NONSTANDARD	COMBO[base]	98.66	98.86	98.66	96.23	91.86	90.25	88.79	95.00	90.26	86.84	83.53	70.19	79.53
	Trankit[large]	98.74	98.00	98.74	96.23	91.64	90.51	89.07	94.66	90.68	86.89	83.13	70.10	78.50
	Trankit[base]	98.68	98.57	98.68	96.04	91.48	90.33	88.89	94.57	90.14	86.40	82.40	69.46	77.93
	Stanza	98.96	97.53	98.96	95.60	90.85	89.71	88.32	94.96	88.97	84.88	80.51	67.17	76.58
Russian SYNTAGRUS	COMBO[base]	99.52	98.77	99.52	98.68	99.52	97.30	97.08	98.50	94.52	93.27	92.82	89.47	91.62
	Trankit[large]	99.71	99.45	99.71	99.06	99.71	98.20	97.99	97.99	95.66	94.65	93.78	91.68	91.51
	Trankit[base]	99.71	99.14	99.71	98.94	99.71	97.85	97.59	97.89	95.19	94.08	93.13	90.59	90.77
	Stanza	99.74	98.43	99.74	98.06	99.74	92.78	92.44	97.65	93.26	90.90	89.47	80.36	87.17
Spanish ANCORA	COMBO[base]	99.85	97.97	99.83	98.97	96.12	98.57	95.61	99.38	93.82	92.27	89.60	87.18	89.16
	Trankit[large]	99.91	99.39	99.91	99.06	99.00	98.85	98.36	99.15	94.77	93.29	91.10	89.23	90.14
	Trankit[base]	99.94	99.13	99.93	99.02	98.94	98.80	98.27	99.17	94.11	92.41	89.66	87.60	88.71
	Stanza	99.97	98.49	99.96	98.96	96.10	98.54	95.65	99.24	93.09	91.30	87.80	85.49	86.94
Swedish TALBANKEN	COMBO[base]	99.81	97.34	99.81	98.92	97.65	97.24	96.40	98.43	93.56	91.71	90.51	85.92	88.74
	Trankit[large]	99.91	99.26	99.91	99.06	98.09	98.04	97.43	97.87	94.41	92.97	92.08	88.67	89.28
	Trankit[base]	99.87	99.38	99.87	98.76	97.77	97.73	97.03	97.82	93.61	91.87	90.72	86.97	88.03
	Stanza	99.97	97.49	99.97	98.28	97.13	97.29	96.24	97.92	90.32	87.85	85.91	81.88	83.42
Turkish PENN	COMBO[base]	99.84	94.97	99.84	95.63	99.84	93.65	92.77	93.74	85.72	72.78	69.63	61.49	64.25
	Stanza	100.00	94.52	100.00	96.24	100.00	94.37	93.46	94.58	85.09	71.95	68.83	61.15	64.27