LINGUIST List 8.778

Sat May 24 1997

Sum: Natural lg dialogue systems evaluation

Editor for this issue: Ann Dizdar <annlinguistlist.org>

Directory

1. gas0, Natural language dialogue systems evaluation

Ramon Lopez-Cozar Delgado
Dpto. Electronica y Tecnologia de Computadores
Universidad de Granada
18071 Granada, Espana (Spain)
e-mail: gas0elvira.ugr.es
Fax: +34-58-243230

Dear LINGUIST Subscribers,

A few weeks ago I sent out an E-mail inquiry on
natural language dialogue systems evaluation.
I would like to thank the following people for
their kind replies. I enclose a summary of their
answers.

Nicholas Ostler - nostlerchibcha.demon.co.uk
Jean Caelen - Jean.Caelenimag.fr
Frederik Fouvry - fouvryllsun10.essex.ac.uk
Hugo Zaragoza - Hugo.Zaragozalaforia.ibp.fr
Gerhard Hanrieder - hanriedevs.dasa.de
Gavin E Churcher - gavinscs.leeds.ac.uk
Philips Bouwman - bouwmanpfa.research.philips.com
Eric Atwell - ericscs.leeds.ac.uk
Matthias Denecke - deneckecs.cmu.edu
Robert John Gaizauskas - R.Gaizauskasdcs.shef.ac.uk
Boda Peter - peter.bodaresearch.nokia.com
Doug Arnold - dougessex.ac.uk
Bill Manaris - manarisucs.usl.edu
Maria Teresa Taboada - taboada+andrew.cmu.edu
Alan Biermann - awbcs.duke.edu
Mai Kuha - mkuhaindiana.edu
Hans Dybkjaer - dybkjaercog.ruc.dk

Special thanks to Nicholas Ostler for sending "Lying in
Wait at the Heart of the Web", a very interesting
report on the development of linguistic technology in
Japan.

The summary follows:

******* Interesting www sites ***********************

- The DIET project:
http://www2.echo.lu/langeng/en/le3/diet/diet.html

- Dialogue systems evaluation
http://xxx.lanl.gov/cmp-lg

- A Generic Template to evaluate integrated components
in spoken dialogue
http://www.scs.leed.ac.uk/gavin/papers/acl.htm

- PARADISE: A framework for Evaluating Spoken Dialogue
Agents
http://xxx.lanl.gov/abs/cmp-lg/?9704004

- Metrics for evaluating Dialogue Strategies in a Spoken
Language System
http://xxx.lanl.gov/abs/cmp-lg/?9612003

- EAGLES
http://www.ilc.pi.cnr.it/EAGLES/home.html

- Tests suites for Natural Language Processing
http://clwww.essex.ac.uk/group/projects/tsnlp/

- "End-to-end evaluation in Janus: a speech-to-speech
translation system"
http://www.is.cs.cmu.edu/ISL.speech.publications.html

********** Interesting ftp addresses *****************

ftp://ftp.ilc.pi.cnr.it/pub/eagles/evaluation/ewg95.ps
ftp://agora.leeds.ac.uk/scs/doc/reports/1997/97_06.ps.Z

************ Interesting reports **********************

- Neal J. Feit E. et al (1992): "An Evaluation Methodology
for Natural Processing Systems", Final Technical Report,
Rome Laboratory.

*************** Interesting books *********************

- "Evaluating Natural Language Processing Systems"
Karen Sparck-Jones and Julia Galliers, Springer, 1995

- "An Architecture for Voice-Dialog"
Smith, Hipp and Biermann,
Computational Linguistics, vol. 21, no. 3

Best regards,
Ramon.