LINGUIST List 8.1706

Thu Nov 27 1997

Confs: Word Sense Disambiguation Evaluation

Editor for this issue: Anita Huang <anitalinguistlist.org>

Directory

1. senseval-coord, SENSEVAL WSD Evaluation Exercise

(This document is http://www.itri.brighton.ac.uk/events/senseval/cfp.txt)

 ****************************************
 * CALL FOR PARTICIPATION *
 * WSD EVALUATION EXERCISE *
 * (SENSEVAL) *
 * *
 * WORKSHOP ANNOUNCEMENT *
 * "SENSEVAL AND THE LEXICOGRAPHY LOOP" *
 ****************************************

 Sponsored by ACL SIGLEX

There are now many automatic Word Sense Disambiguation (WSD) programs
but it is currently very hard to determine which are better, which
worse, and where the strengths and weaknesses of each lie. There is
widespread agreement that the field urgently needs an evaluation
framework. Under the auspices of ACL SIGLEX, a pilot will take place
in the course of 1998. As in ARPA evaluations, the framework comprises:

1) definition of task and scoring metric
2) preparation of a set of manually tagged correct answers
3) a dry run, with sample data distributed to participants
4) distribution of test data to participants
5) participants sense-tag and return the data
6) participants' taggings scored against correct answers
7) workshop to discuss results, lessons learned, way forward

We shall be undertaking evaluation for English, French, German,
Italian and Spanish. The workshop will be held at

 Herstmonceux Castle, Sussex, UK
 on
 Sept 2-4 1998

If you have a working WSD program (or will have one by Summer 1998),
and would like to subject it to objective, quantitative evaluation, or
if you have skills or resources that you would like to contribute to
the exercise, first look at

 http://www.itri.bton.ac.uk/events/senseval/exp-interest.html

and then mail your expression of interest to

 senseval-coorditri.bton.ac.uk


 Provisional timetable
 =====================
By end of
Dec 97: expressions of interest to be registered
Mar 98: 'dry run' data samples distributed to participants
May 98: test data distributed to participants, and, one week
 later, returned with taggings
Jul 98: correct answers and scores made available
Sep 98: workshop

The workshop will comprise:

 (1) reports/papers on the manual tagging and other aspects of
 the evaluation exercise
 (2) reports/papers from participants on the design and
 performance of their system
 (3) other related research papers
 (4) working sessions on the way ahead for WSD evaluation

The deadline for "other related research papers" is

 16th April 1998

* Preference will be given to papers discussing WSD, with particular
 attention to evaluation issues
* Maximum submission length: 6 pages
* First page to include title, abstract, and author's name(s) and
 contact details
* Electronic submission of postscript documents permitted but must be
 supported by hard copy to arrive not later than 7th April
 (in case of printing problems).
 email: senseval-submissionsitri.bton.ac.uk
 hard copy: SENSEVAL Submissions
 ITRI
 University of Brighton
 Lewes Road
 Brighton BN2 4GJ, UK

A background paper is available on

 ftp://ftp.itri.bton.ac.uk/pub/reports/ITRI-97-11.ps.gz


 Planning and Programme Committee
 ================================

Robert Amsler Science and Engineering Associates, Inc.
Susan Armstrong ISSCO, University of Geneva
Roberto Basili University of Rome, Tor Vergata
Henri B\'{e}joint University of Lyon II
Bran Boguraev Apple Computers
Nicoletta Calzolari Istituto di Linguistica Computazionale, Pisa
John Carroll University of Sussex
Steve Crowdy Addison Wesley Longman
Roger Evans ITRI, University of Brighton
Helmut Feldweg University of T\"{u}bingen
Christiane Fellbaum WordNet, Princeton University
Patrick Hanks Oxford University Press
Marti Hearst University of California, Berkeley
Lynette Hirschman MITRE Corporation
Nancy Ide Vassar College
Adam Kilgarriff (co-ordinator) ITRI, University of Brighton
Judith Klavans Columbia University
Ramesh Krishnamurthy University of Birmingham
Robert Krovetz NEC Research Institute, New Jersey
Marc Light University of Stuttgart
Dan Melamed University of Pennsylvania
Rosamund Moon Collins and University of Birmingham
Boyan Onyshkevych US Department of Defense
Martha Palmer (ACL SIGLEX chair) University of Pennsylvania
Philip Resnik University of Maryland
Michael Rundell Lexicographer
Hinrich Sch\"{u}tze Xerox Parc and Stanford University
Fr\'{e}d\'{e}rique Segond Xerox Research Centre, Grenoble
Jean V\'{e}ronis Universit\'{e} de Provence
Evelyne Viegas New Mexico State University
Clare Voss ARL (US Army Research Lab)
Piek Vossen University of Amsterdam
Jan Wiebe New Mexico State University
Yorick Wilks University of Sheffield
David Yarowsky Johns Hopkins University
Antonio Zampolli University of Pisa


(This document is http://www.itri.brighton.ac.uk/events/senseval/cfp.txt)