* * * * * * * * * * * * * * * * * * * * * * * *
LINGUIST List logo Eastern Michigan University Wayne State University *
* People & Organizations * Jobs * Calls & Conferences * Publications * Language Resources * Text & Computer Tools * Teaching & Learning * Mailing Lists * Search *
* *
LINGUIST List 19.3929

Sun Dec 21 2008

Qs: ResPubliQA2009: Call for Participation

Editor for this issue: Dan Parker <danlinguistlist.org>


We'd like to remind readers that the responses to queries are usually best posted to the individual asking the question. That individual is then strongly encouraged to post a summary to the list. This policy was instituted to help control the huge volume of mail on LINGUIST; so we would appreciate your cooperating with it whenever it seems appropriate.

In addition to posting a summary, we'd like to remind people that it is usually a good idea to personally thank those individuals who have taken the trouble to respond to the query.

To post to LINGUIST, use our convenient web form at http://linguistlist.org/LL/posttolinguist.html.
Directory
        1.    Pamela Forner, ResPubliQA2009: Call for Participation


Message 1: ResPubliQA2009: Call for Participation
Date: 19-Dec-2008
From: Pamela Forner <fornercelct.it>
Subject: ResPubliQA2009: Call for Participation
E-mail this message to a friend

ResPubliQA exercise at the

Multilingual Question Answering Track at CLEF 2009


We are glad to announce that a new exercise will be proposed this year
within the Question Answering track at CLEF. For more information and
instructions visit the new ResPubliQA website at:

http://celct.isti.cnr.it/ResPubliQA/index.php

We invite participation, both from academic institutions and industrial
organizations, on this new task.

Guidelines describing the task will be distributed among the participants
and will be downloadable from the ResPubliQA website. Participants will
also be provided with training data in order to have the opportunity to
test the systems with procedures to be used in the formal evaluation
campaign. The results of the evaluation will be disseminated at the final
workshop which will be organized in Corfu in conjunction with ECDL 2009.

ResPubliQA 2009: Task Overview

Task Description: Systems receive natural language questions as input, and
must return one paragraph containing the answer from the document
collection. No exact answer is required neither multiple responses.

Document Collection: The subset of JRC-Acquis documents that have parallel
aligned translations into all the languages involved will be used, namely
Bulgarian, Dutch, English, French, German, Italian, Portuguese, Romanian
and Spanish. The sub-collection is available at the ResPubliQA website:

http://celct.isti.cnr.it/ResPubliQA/index.php?page=Pages/downloads.php&option=newTrackSetup

Questions: a pool of 500 independent questions (factoid, definition,
reason, purpose and procedure) is provided
- No list questions
- No topic related questions (questions linked to the same topic)
- No NIL questions

Answers: one of the following two responses must be returned
a) one single paragraph containing the candidate answer. Multi-paragraph
answers are not considered in this task
b) the string NOA to indicate that the system prefers not to answer the
question.
Systems that give no answers (NOA) instead of wrong answers will be
rewarded by the evaluation measure. Answer Validation techniques (including
Machine Learning) are expected to be used for taking this final decision.

Languages Involved: Basque (EU), Bulgarian (BG), Dutch (NL), English (EN),
French (FR), German (DE), Italian (IT), Portuguese (PT), Romanian (RO) and
Spanish (ES).

A monolingual English (EN) task will also be activated this year, as both
the exercise and the collection are different from TREC.
Basque has been included exclusively as a source language, as there is no
Basque collection available.

Schedule:

Corpora Release: December, 19 2008

Preliminary Track Guidelines: December, 19 2008

Training data: December 19, 2008

Registration Open: February 1, 2009

Final Track Guidelines: February 1, 2009

Test Sets Release: May 25, 2009

Submissions of Runs by Participants: June 5, 2009

Release of Individual Results: from July 15, 2009

Submission of Papers for Working Notes: August 14, 2009

CLEF Workshop (in Corfu, Greece): 30 September – 02 October 2009

The participants will have 5 DAYS to upload their submissions, starting
from the moment when the questions are downloaded, and not later than June
5, 2009.

Track Coordinators and Organizers:

- UNED (coordinator)
Spanish Distance Learning University, Spain
Anselmo Peñas

- CELCT (coordinator)
Center for the Evaluation of Language and Communication Technology, Italy
Pamela Forner and Danilo Giampiccolo

- ELDA/ELRA
Evaluations and Language Resources Distribution Agency, France
Nicolas Moreau

- University of Limerick, Ireland
Richard Sutcliffe

- BTB
Bulgarian Academy of Science, Bulgaria
Petya Osenova

- UAIC and RACAI, Romania
Alexandru Ioan Cuza University and Romanian Academy Research Institute for
Artificial Intelligence, Romania
Corina Forascu

- UBC
University of Basque Country, Spain
Iñaki Alegria

Linguistic Field(s): General Linguistics

Read more issues|LINGUIST home page|Top of issue




Please report any bad links or misclassified data

LINGUIST Homepage | Read LINGUIST | Contact us

NSF Logo

While the LINGUIST List makes every effort to ensure the linguistic relevance of sites listed
on its pages, it cannot vouch for their contents.