LINGUIST List 19.3929
|
Sun Dec 21 2008
Qs: ResPubliQA2009: Call for Participation
Editor for this issue: Dan Parker
<dan linguistlist.org>
|
We'd like to remind readers that the responses to queries are usually best posted to the individual asking the question. That individual is then strongly encouraged to post a summary to the list. This policy was
instituted to help control the huge volume of mail on LINGUIST; so we would appreciate your cooperating with it whenever it seems appropriate.
In addition to posting a summary, we'd like to remind people that it is usually a good idea to personally thank those individuals who have taken the trouble to respond to the query.
To post to LINGUIST, use our convenient web form at http://linguistlist.org/LL/posttolinguist.html.
|
Directory
1. Pamela
Forner,
ResPubliQA2009: Call for Participation
Message 1: ResPubliQA2009: Call for Participation
|
Date: 19-Dec-2008
From: Pamela Forner <fornercelct.it>
Subject: ResPubliQA2009: Call for Participation
 E-mail this message to a friend
ResPubliQA exercise at the
Multilingual Question Answering Track at CLEF 2009
We are glad to announce that a new exercise will be proposed this year
within the Question Answering track at CLEF. For more information and
instructions visit the new ResPubliQA website at:
http://celct.isti.cnr.it/ResPubliQA/index.php
We invite participation, both from academic institutions and industrial
organizations, on this new task.
Guidelines describing the task will be distributed among the participants
and will be downloadable from the ResPubliQA website. Participants will
also be provided with training data in order to have the opportunity to
test the systems with procedures to be used in the formal evaluation
campaign. The results of the evaluation will be disseminated at the final
workshop which will be organized in Corfu in conjunction with ECDL 2009.
ResPubliQA 2009: Task Overview
Task Description: Systems receive natural language questions as input, and
must return one paragraph containing the answer from the document
collection. No exact answer is required neither multiple responses.
Document Collection: The subset of JRC-Acquis documents that have parallel
aligned translations into all the languages involved will be used, namely
Bulgarian, Dutch, English, French, German, Italian, Portuguese, Romanian
and Spanish. The sub-collection is available at the ResPubliQA website:
http://celct.isti.cnr.it/ResPubliQA/index.php?page=Pages/downloads.php&option=newTrackSetup
Questions: a pool of 500 independent questions (factoid, definition,
reason, purpose and procedure) is provided
- No list questions
- No topic related questions (questions linked to the same topic)
- No NIL questions
Answers: one of the following two responses must be returned
a) one single paragraph containing the candidate answer. Multi-paragraph
answers are not considered in this task
b) the string NOA to indicate that the system prefers not to answer the
question.
Systems that give no answers (NOA) instead of wrong answers will be
rewarded by the evaluation measure. Answer Validation techniques (including
Machine Learning) are expected to be used for taking this final decision.
Languages Involved: Basque (EU), Bulgarian (BG), Dutch (NL), English (EN),
French (FR), German (DE), Italian (IT), Portuguese (PT), Romanian (RO) and
Spanish (ES).
A monolingual English (EN) task will also be activated this year, as both
the exercise and the collection are different from TREC.
Basque has been included exclusively as a source language, as there is no
Basque collection available.
Schedule:
Corpora Release: December, 19 2008
Preliminary Track Guidelines: December, 19 2008
Training data: December 19, 2008
Registration Open: February 1, 2009
Final Track Guidelines: February 1, 2009
Test Sets Release: May 25, 2009
Submissions of Runs by Participants: June 5, 2009
Release of Individual Results: from July 15, 2009
Submission of Papers for Working Notes: August 14, 2009
CLEF Workshop (in Corfu, Greece): 30 September – 02 October 2009
The participants will have 5 DAYS to upload their submissions, starting
from the moment when the questions are downloaded, and not later than June
5, 2009.
Track Coordinators and Organizers:
- UNED (coordinator)
Spanish Distance Learning University, Spain
Anselmo Peñas
- CELCT (coordinator)
Center for the Evaluation of Language and Communication Technology, Italy
Pamela Forner and Danilo Giampiccolo
- ELDA/ELRA
Evaluations and Language Resources Distribution Agency, France
Nicolas Moreau
- University of Limerick, Ireland
Richard Sutcliffe
- BTB
Bulgarian Academy of Science, Bulgaria
Petya Osenova
- UAIC and RACAI, Romania
Alexandru Ioan Cuza University and Romanian Academy Research Institute for
Artificial Intelligence, Romania
Corina Forascu
- UBC
University of Basque Country, Spain
Iñaki Alegria
Linguistic Field(s):
General Linguistics
Read more issues|LINGUIST home page|Top of issue
|
|
Please report any bad links or misclassified data
LINGUIST Homepage | Read
LINGUIST | Contact us
While the LINGUIST List makes every effort to ensure the linguistic relevance of sites listed
on its pages, it cannot vouch for their contents.
|
|
