* * * * * * * * * * * * * * * * * * * * * * * *
LINGUIST List logo Eastern Michigan University Wayne State University *
* People & Organizations * Jobs * Calls & Conferences * Publications * Language Resources * Text & Computer Tools * Teaching & Learning * Mailing Lists * Search *
* *
LINGUIST List 16.3173

Thu Nov 03 2005

FYI: PASCAL RTE2 Challenge; Temporary Mirror for RTE2

Editor for this issue: Svetlana Aksenova <svetlanalinguistlist.org>


To post to LINGUIST, use our convenient web form at http://linguistlist.org/LL/posttolinguist.html.
Directory
        1.    Roy Bar-Haim, Call for Participation: PASCAL RTE2 Challenge
        2.    Roy Bar-Haim, Temporary Mirror of the PASCAL RTE2 Challenge Website


Message 1: Call for Participation: PASCAL RTE2 Challenge
Date: 02-Nov-2005
From: Roy Bar-Haim <barhaircs.biu.ac.il>
Subject: Call for Participation: PASCAL RTE2 Challenge


The Second PASCAL Recognising Textual Entailment Challenge

CALL FOR PARTICIPACION

A fundamental phenomenon of natural language is the variability of
semantic expression, where the same meaning can be expressed by
or inferred from different texts. Many natural language processing
applications, such as Question Answering (QA), Information Retrieval
(IR), Information Extraction (IE), and (multi) document summarization
need to model this variability in order to recognize that a particular
target meaning can be inferred from different text variants. Even though
many applications face similar underlying semantic problems, these
problems are usually addressed in an application-oriented manner.

Textual Entailment Recognition was proposed recently as a generic task
and evaluation framework that captures major semantic inference needs
across natural language processing applications. The current challenge
considers an applied notion of textual entailment, defined as a
directional relation between two text fragments, termed T - the
entailing text, and H - the entailed text. We say that T entails H if,
typically, a human reading T would infer that H is most likely true (see
examples below). This operational definition is based on (and assumes)
common human understanding of language as well as common background
knowledge.

The last two years have seen rapidly growing interest in textual
entailment within the natural language processing community.
The First PASCAL Recognising Textual Entailment (RTE) Challenge
provided the first benchmark for evaluating entailment systems. The
challenge raised noticeable attention in the research community,
attracting 17 submissions from diverse groups. The relatively low
accuracy achieved by the participating systems suggests that the
entailment task is indeed a challenging one, with a wide room for
improvement. It was followed by an ACL 2005 Workshop on Empirical
Modeling of Semantic Equivalence and Entailment. The challenge and its
dataset motivated further research on empirical entailment, which
resulted in a number of publications in recent main conferences as well
as the inclusion of this topic in some recent calls for papers.

By introducing a second challenge we hope to keep the momentum going,
and to further promote the formation of a research community around the
applied entailment task. As in the previous challenge, the main task is
judging whether a hypothesis (H) is entailed by a text (T). One of the
main goals for the RTE-2 dataset is to provide more ''realistic''
text-hypothesis examples, based mostly on outputs of actual systems. We
focus on the four application settings mentioned above: QA, IR, IE and
multi-document summarization. Each portion of the dataset includes
typical T-H examples that correspond to success and failure cases of such
applications. The examples represent different levels of entailment
reasoning, such as lexical, syntactic, morphological and logical.
The data collection procedure for each application setting can be found
in the challenge website. The development subset, which represents the
different types of test examples, is released first, but systems are
likely to use external and unsupervised knowledge resources as well.
The development set consists of 800 examples, 200 for each application
setting. The test set will contain 1000-1200 examples. To make the
challenge data more accessible, we also provide some pre-processing for
the text and hypothesis, including sentence splitting and dependency
parsing.

RTE-2 was organized by Bar-Ilan University (Israel), CELCT (Trento,
Italy), Microsoft Research (USA) and MITRE (USA). Data collection and
annotation processes were improved this year, including cross-annotation
of the examples across the organizing sites.


T-H EXAMPLES

Text: The drugs that slow down or halt Alzheimer's disease work best
the earlier you administer them.

Hypothesis: Alzheimer's disease is treated using drugs.

Entailment: Yes

* * *
Text: Drew Walker, NHS Tayside's public health director, said: ''It is
important to stress that this is not a confirmed case of rabies.''

Hypothesis: A case of rabies was confirmed.

Entailment: No

* * *
Text:Yoko Ono unveiled a bronze statue of her late husband, John
Lennon, to complete the official renaming of England's Liverpool
Airport as Liverpool John Lennon Airport.

Hypothesis: Yoko Ono is John Lennon's widow.

Entailment: Yes

* * *
Text: Arabic, for example, is used densely across North Africa
and from the Eastern Mediterranean to the Philippines, as the key
language of the Arab world and the primary vehicle of Islam.

Hypothesis: Arabic is the primary language of the Philippines.

Entailment: No


IMPORTANT DATES

Release of Development Set October 26, 2005
Release of Test Set January 12, 2006
Deadline for participants' Submissions February 2, 2006
Release of individual results February 7, 2006
Deadline for participants' reports February 21, 2006
Camera-ready version of reports March 14, 2006
PASCAL Challenges Workshop April 10, 2006
(in Venice, Italy)

Note: the workshop is scheduled right after EACL.


CHALLENGE ORGANISERS

Bar-Ilan University, Israel (Coordinator):
Ido Dagan,
Roy Bar-Haim,
Idan Szpektor

CELCT, Trento - Italy:
Bernardo Magnini,
Danilo Giampiccolo

Microsoft Research, USA:
Bill Dolan

MITRE, USA:
Lisa Ferro


SUPPORT

The preparation and running of this challenge has been supported by the
EU-funded PASCAL Network of Excellence on Pattern Analysis, Statistical
Modelling and Computational Learning.


REGISTRATION

For registration, further information and inquiries - please visit the
challenge web site:

http://www.pascal-network.org/Challenges/RTE2


CONTACT

Roy Bar-Haim cs.biu.ac.il>



Linguistic Field(s): Applied Linguistics
Computational Linguistics
Semantics
Text/Corpus Linguistics
Message 2: Temporary Mirror of the PASCAL RTE2 Challenge Website
Date: 03-Nov-2005
From: Roy Bar-Haim <barhaircs.biu.ac.il>
Subject: Temporary Mirror of the PASCAL RTE2 Challenge Website


Due to a serious fire at the University of Southampton, UK, the PASCAL Network
of Excellence website (which hosts the RTE 2 website), is temporarily
unavailable. A temporary mirror of the RTE 2 website is available here:

http://www.cs.biu.ac.il/~barhair/RTE2/

The RTE 2 development set, released October 26th, can be downloaded from this
website, as well as RTE 1 datasets.

We apologise for any confusion that this may have caused.

Information about the fire can be found at:

http://news.bbc.co.uk/1/hi/england/hampshire/4390048.stm

No persons are believed to have been injured in the fire.



Linguistic Field(s): Not Applicable




Respond to list|Read more issues|LINGUIST home page|Top of issue




Please report any bad links or misclassified data

LINGUIST Homepage | Read LINGUIST | Contact us

NSF Logo

While the LINGUIST List makes every effort to ensure the linguistic relevance of sites listed
on its pages, it cannot vouch for their contents.