LINGUIST List 7.802

Fri May 31 1996

FYI: Multilingual Evaluation Tool

Editor for this issue: Ljuba Veselinova <lveselinemunix.emich.edu>

Directory

1. Balkan L, Multilingual Evaluation Tool

 Multilingual Evaluation Tool

We would like to draw your attention to a multilingual evaluation tool
that is now available. It consists of a database of test suites for
English, French and German that have been constructed for evaluating
Natural Language Processing Systems, but which may be useful for other
purposes. The database consists of over 14,000 examples in English,
French and German, which have been very systematically constructed
with detailed annotations about various grammatical and other
information. The test suites, support software, user documentation,
and background documentation are available free from:

 http://tsnlp.dfki.uni-sb.de/tsnlp/ (WWW)
 tsnlp.dfki.uni-sb.de/tsnlp/ (anonymous ftp)

They have been produced by the University of Essex (UK), ISSCO
(Switzerland), Deutsches Forschungszentrum fuer Kuenstliche
Intelligenz (DFKI) (Germany), and Aerospatiale (France) as part
of LRE Project 62-089, Test Suites for Natural Language Processing
(TSNLP).

Here is a short report on the project, including some more information.

Lorna Balkan,
University of Essex

 ----------------------------------------------------------------

			 LRE Project 62-089
	 Test Suites for Natural Language Processing
			 (TSNLP)

TSNLP has resulted in a database of test suites in 3 languages,
English, French and German with over 4000 test items in each language.
The test items illustrate central syntactic phenomena and are relevant
to any developer or user of systems with grammar components who wish
to test, benchmark or evaluate those systems. The data have
been validated in a testing phase. The project was coordinated
by the University of Essex, with partners in Switzerland,
France and Germany as follows:

University of Essex, CL/MT Group, Department of Language
and Linguistics, University of Essex, Colchester, C04 3SQ, UK.
Contact person: Lorna Balkan, Tel: (44) 1206 872092
Fax: (44) 1206 872085, email: balkaessex.ac.uk

ISSCO, University of Geneva, CH - 1227 Geneva, Switzerland.
Contact person: Sabine Lehmann, Tel. +41 22 705 79 33,
Fax. +41 22 300 10 86, email: Sabine.Lehmannissco.unige.ch

Aerospatiale, Centre commun de recherches Louis-Bleriot,
Departement Information-Documentation, 12, Rue Pasteur BP 76,
92152 Suresnes Cedex, France.
Contact person: Sylvie Regnier-Prost, Tel: +33 1 46 97 3061
Fax: +33 1 46 97 3518, email: sylvie.regniersiege.aerospatiale.fr

Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH,
Computational Linguistics Department, Stuhlsatzenhausweg 3,
D-66123 Saarbruecken (Germany)
Contact persons: Klaus Netter, Stephan Oepen, Tel +49 681 302 52 82
Fax: +49 681 302 53 41, email: {netter|oe}cl.dfki.uni-sb.de

Major project results are documented in a user manual, which provides
a description of the test data, the methodology which underlies their
construction, and the tools which were developed in the project to aid
test suite construction and use. The user manual is available in three
volumes:

Volume1: Background, methodology, customisation and testing
A description of the background to TSNLP,
the methodology that underlies the
TSNLP data, how the data can be customised, and
how the data was used in practice to test a grammar
checker.

Volume 2:Core Test suite technology:
A description of the annotation scheme on which the data is
constructed, plus a description of the database
(ANSI C and Access), and the test suite construction tool

Volume 2b:Test suite technology:
A description of the automatic test suite generation tool, and the
lexical replacement tool

Volume 3:Test Data documentation
A detailed description of the data in the database.

TSNLP results are being made available free of charge to the academic
and industrial community, in order that they might be widely used and
accepted as an evaluation tool. Users are encouraged to
report back any comments or criticisms. They are also encouraged to
offer any extensions they make for inclusion in possible future
releases of the database. The consortium is also happy to discuss
producing customised test suites for specific users, at a cost to be
arranged, and developing some of the construction tools.

TSNLP test data, software, and reports, including the user manual, are
available from the coordinator or via the following ftp and www
address:

FTP access:
anonymoustsnlp.dfki.uni-sb.de/tsnlp/

World Wide Web Access:
http://tsnlp.dfki.uni-sb.de/tsnlp/

Contact point:
Lorna Balkan,
CL/MT Group,
Department of Language and Linguistics,
University of Essex, Colchester,
C04 3SQ
UK

Tel: (44) 1206 872092
Fax: (44) 1206 872085
Email: balkaessex.ac.uk

Project results will also shortly be available from
the European Language Resources Association (ELRA)
at the following address:

Tel. +33 1 45 86 53 00
Fax. +33 1 45 86 44 88
87, Avenue D'ITALIE, 75013 PARIS

Email: elracalvanet.calvacom.fr

Web: http://www.icp.grenet.fr/ELRA/home.html