LINGUIST List 17.1577: SemEval-1 / SensEval-4: Call for Task Proposals

LINGUIST List 17.1577

Wed May 24 2006

FYI: SemEval-1 / SensEval-4: Call for Task Proposals

Editor for this issue: Svetlana Aksenova <svetlanalinguistlist.org>

Directory 1. Eneko Agirre, SemEval-1 / SensEval-4: Call for Task Proposals

Message 1: SemEval-1 / SensEval-4: Call for Task Proposals
Date: 23-May-2006
From: Eneko Agirre <e.agirreehu.es>
Subject: SemEval-1 / SensEval-4: Call for Task Proposals

SEMEVAL-1 / SENSEVAL-4

CALL FOR TASK PROPOSALS

The Senseval Committee invites proposals for tasks to be run as partof SemEval-1 and Senseval-4. As the nature of the tasks in Sensevalhas evolved to include semantic analysis tasks outside of word sensedisambiguation, the Senseval Committee is pleased to announce thedebut of ''SemEval: Evaluating Semantics''. In 2007, the SemEvalworkshop will retain the Senseval name to help ease the transition tothe new name.

We welcome proposals for any tasks that can test an automatic systemfor semantic analysis of text, be it application dependent orindependent. We especially encourage tasks for different languages,cross-lingual tasks, and tasks that are relevant to particular NLPapplications such as machine translation, information retrieval andinformation extraction.

It will help us if tasks are designed in such a way that they can berun and scored automatically from a central website (as inSenseval-3). We will be happy to help with task design, dataformatting, and so on.

Information on the design of tasks from Senseval-1, -2, and -3 can befound at:http://www.senseval.org/senseval3/http://193.133.140.102/senseval2/http://www.itri.brighton.ac.uk/events/senseval/ARCHIVE/index.html

The time period for SemEval-1/Senseval-4 has not yet been finalised,but it will likely be held over a 45-day period sometime during thefirst quarter of 2007.

SUBMISSION DETAILS

Proposals for tasks will ideally contain:

- A description of the task (max 1 page)- How the training/testing data will be built and/or procured- The evaluation methodology to be used, including clear evaluation criteria- The anticipated availability of necessary resources to the participants (copyright, etc)- The resources required to prepare the task (computation and annotation time, etc)

If you are not yet at a point to provide outlines of all of these,that is acceptable, but please give some thought to each, and presenta sketch of your first ideas. We will gladly give feedback.

Please submit proposals as soon as possible, preferably by electronicmail in plain ASCII text to the SemEval-1 / Senseval-4 email address:semevalnlp.cs.swarthmore.edu

IMPORTANT DATES

July 1, 2006 Submission deadline for task proposalsJuly 15, 2006 Notification of acceptance

TENTATIVE SCHEDULE

November 1, 2006 Preliminary call for participationJanuary 1, 2007 Call for participation, release of trial dataMarch 2007 Estimated Evaluation Period (45 days in total)Summer 2007 Workshop, possibly co-located with ACL or NAACL

SEMEVAL-1 / SENSEVAL-4 CHAIRS

Richard Wicentowski, Swarthmore CollegeLluís Màrquez, Technical University of Catalonia (UPC)Eneko Agirre, University of the Basque Country (UPV-EHU)

SEMEVAL-1 / SENSEVAL-4 WEBSITE

http://nlp.cs.swarthmore.edu/semeval/

Linguistic Field(s): Computational Linguistics; Semantics