LINGUIST List 21.2188
|
Tue May 11 2010
FYI: Natural Language Generation Challenge (GIVE 2.5)
Editor for this issue: Rachelle Felzien
<rachelle linguistlist.org>
|
To post to LINGUIST, use our convenient web form at http://linguistlist.org/LL/posttolinguist.cfm.
|
Directory
1. Alexander
Koller,
Natural Language Generation Challenge (GIVE 2.5)
Message 1: Natural Language Generation Challenge (GIVE 2.5)
|
Date: 07-May-2010
From: Alexander Koller <kollermmci.uni-saarland.de>
Subject: Natural Language Generation Challenge (GIVE 2.5)
 E-mail this message to a friend
Second Natural Language Generation (NLG) Challenge on Generating
Instructions in Virtual Environments (GIVE-2.5).
Early Announcement:
http://www.give-challenge.org/research/
In the past two years, we have been organizing the Challenges on
Generating Instructions in Virtual Environments (GIVE). In 2008-09,
we evaluated five natural language generation systems; the Second GIVE
Challenge (GIVE-2), in which we are evaluating seven systems, is
currently underway.
We are now announcing that we will organize the Second Second GIVE
Challenge (GIVE-2.5) in the winter of 2010-11. The task in GIVE-2.5
will be basically identical to the one we now have in GIVE-2; this is
so GIVE-2 systems can be improved based on experiences from the
evaluation, and to allow more people to participate in the same task.
We invite you to consider participating in GIVE-2.5. For more
information and to try out the GIVE-2 software, see
http://www.give-challenge.org/research.
If you are potentially interested in participating, please email us at
kollermmci.uni-saarland.de so we know to keep you updated.
Overview
--------
The Challenge on Generating Instructions in Virtual Environments
(GIVE) is a novel approach to the notoriously hard problem of
evaluating NLG systems. In this scenario, a human user performs a
"treasure hunt" task in a virtual 3D environment. The NLG system's job
is to generate, in real time, a sequence of natural-language
instructions that will help the user perform this task. The crucial
thing is that users connect to the generation systems over the
Internet. By logging how well they were able to follow the system's
instructions, we can evaluate the quality of these instructions in
terms of task completion rates and times, subjective measures such as
helpfulness and friendliness, and runtime performance. Because the
user and the system don't need to be physically in the same place,
access to experimental subjects over the Internet becomes easy.
GIVE is a theory-neutral, end-to-end evaluation effort for NLG
systems. It involves research opportunities in text planning, sentence
planning, realization, and situated communication. One particularly
interesting aspect of situating the generation problem in a virtual
environment is that spatial and relational expressions play a bigger
role than in other NLG tasks. Beyond NLG, GIVE can be interesting as a
testbed for improving the NLG components of dialogue systems, and for
computational semanticists working on spatial language.
The GIVE-2 Task
---------------
In the GIVE-1 Challenge, which we ran last year, five NLG systems were
evaluated using data from almost 1200 game runs. To our knowledge,
this made GIVE-1 the largest ever NLG evaluation effort in terms of
the number of experimental subjects. We presented the results of the
evaluation at the ENLG Workshop, and have verified that these results
are consistent with (but more detailed than) the results that could be
obtained from a traditional lab-based evaluation.
In GIVE-2 we are evaluating seven systems; the public evaluation is
currently underway (see www.give-challenge.org). The main novelty in
GIVE-2 is that where GIVE-1 used discrete worlds (which were based on
square tiles, and the user could only jump from the center of one tile
to the center of the next, and turn in 90 degree steps), GIVE-2
permits free, continuous movements in the worlds. This makes the
generation task more challenging because simple instructions of the
form "walk three steps forward" are no longer possible. The results
of GIVE-2 will be presented at the INLG conference this year.
Anyone is invited to submit an NLG system to participate in the
GIVE-2.5 Challenge. We particularly invite contributions from students
and student teams. To get an idea of what this involves, you may want
to go to the GIVE website mentioned above and take a look at our EACL
2009 demo paper describing the software architecture, or download the
GIVE-2 software and look at it in more detail.
Provisional Timeline
--------------------
We plan to use essentially the same software for GIVE-2.5 that we used
in GIVE-2. This means that GIVE-2 systems should be adaptable to
GIVE-2.5 with minimal effort; that is, you can start implementing your
GIVE-2.5 system based on the GIVE-2 software right now. While we
don't have a precise schedule yet, we hope to present the results of
GIVE-2.5 at ENLG 2011. This will probably entail that the public
evaluation phase will be at some point in the winter 2010-11.
We will distribute a call for participation with more details and
provide the GIVE-2.5 software in due time.
Organizing committee
--------------------
Donna Byron, Northeastern University
Justine Cassell, Northwestern University
Robert Dale, Macquarie University
Alexander Koller, Saarland University
Johanna Moore, University of Edinburgh
Jon Oberlander, University of Edinburgh
Kristina Striegnitz, Union College
Linguistic Field(s): Computational Linguistics
Read more issues|LINGUIST home page|Top of issue
|
|
Please report any bad links or misclassified data
LINGUIST Homepage | Read
LINGUIST | Contact us
While the LINGUIST List makes every effort to ensure the linguistic relevance of sites listed
on its pages, it cannot vouch for their contents.
|
|
