LINGUIST List 21.2188

Tue May 11 2010

FYI: Natural Language Generation Challenge (GIVE 2.5)

Editor for this issue: Rachelle Felzien <rachellelinguistlist.org>

        1.    Alexander Koller, Natural Language Generation Challenge (GIVE 2.5)
Message 1: Natural Language Generation Challenge (GIVE 2.5)
Date: 07-May-2010
From: Alexander Koller <kollermmci.uni-saarland.de>
Subject: Natural Language Generation Challenge (GIVE 2.5)
E-mail this message to a friend

Second Natural Language Generation (NLG) Challenge on GeneratingInstructions in Virtual Environments (GIVE-2.5).

Early Announcement:

http://www.give-challenge.org/research/

In the past two years, we have been organizing the Challenges onGenerating Instructions in Virtual Environments (GIVE). In 2008-09,we evaluated five natural language generation systems; the Second GIVEChallenge (GIVE-2), in which we are evaluating seven systems, iscurrently underway.

We are now announcing that we will organize the Second Second GIVEChallenge (GIVE-2.5) in the winter of 2010-11. The task in GIVE-2.5will be basically identical to the one we now have in GIVE-2; this isso GIVE-2 systems can be improved based on experiences from theevaluation, and to allow more people to participate in the same task.

We invite you to consider participating in GIVE-2.5. For moreinformation and to try out the GIVE-2 software, seehttp://www.give-challenge.org/research.

If you are potentially interested in participating, please email us atkollermmci.uni-saarland.de so we know to keep you updated.

Overview--------

The Challenge on Generating Instructions in Virtual Environments(GIVE) is a novel approach to the notoriously hard problem ofevaluating NLG systems. In this scenario, a human user performs a"treasure hunt" task in a virtual 3D environment. The NLG system's jobis to generate, in real time, a sequence of natural-languageinstructions that will help the user perform this task. The crucialthing is that users connect to the generation systems over theInternet. By logging how well they were able to follow the system'sinstructions, we can evaluate the quality of these instructions interms of task completion rates and times, subjective measures such ashelpfulness and friendliness, and runtime performance. Because theuser and the system don't need to be physically in the same place,access to experimental subjects over the Internet becomes easy.

GIVE is a theory-neutral, end-to-end evaluation effort for NLGsystems. It involves research opportunities in text planning, sentenceplanning, realization, and situated communication. One particularlyinteresting aspect of situating the generation problem in a virtualenvironment is that spatial and relational expressions play a biggerrole than in other NLG tasks. Beyond NLG, GIVE can be interesting as atestbed for improving the NLG components of dialogue systems, and forcomputational semanticists working on spatial language.

The GIVE-2 Task---------------

In the GIVE-1 Challenge, which we ran last year, five NLG systems wereevaluated using data from almost 1200 game runs. To our knowledge,this made GIVE-1 the largest ever NLG evaluation effort in terms ofthe number of experimental subjects. We presented the results of theevaluation at the ENLG Workshop, and have verified that these resultsare consistent with (but more detailed than) the results that could beobtained from a traditional lab-based evaluation.

In GIVE-2 we are evaluating seven systems; the public evaluation iscurrently underway (see www.give-challenge.org). The main novelty inGIVE-2 is that where GIVE-1 used discrete worlds (which were based onsquare tiles, and the user could only jump from the center of one tileto the center of the next, and turn in 90 degree steps), GIVE-2permits free, continuous movements in the worlds. This makes thegeneration task more challenging because simple instructions of theform "walk three steps forward" are no longer possible. The resultsof GIVE-2 will be presented at the INLG conference this year.

Anyone is invited to submit an NLG system to participate in theGIVE-2.5 Challenge. We particularly invite contributions from studentsand student teams. To get an idea of what this involves, you may wantto go to the GIVE website mentioned above and take a look at our EACL2009 demo paper describing the software architecture, or download theGIVE-2 software and look at it in more detail.

Provisional Timeline--------------------

We plan to use essentially the same software for GIVE-2.5 that we usedin GIVE-2. This means that GIVE-2 systems should be adaptable toGIVE-2.5 with minimal effort; that is, you can start implementing yourGIVE-2.5 system based on the GIVE-2 software right now. While wedon't have a precise schedule yet, we hope to present the results ofGIVE-2.5 at ENLG 2011. This will probably entail that the publicevaluation phase will be at some point in the winter 2010-11.We will distribute a call for participation with more details andprovide the GIVE-2.5 software in due time.

Organizing committee--------------------

Donna Byron, Northeastern UniversityJustine Cassell, Northwestern UniversityRobert Dale, Macquarie UniversityAlexander Koller, Saarland UniversityJohanna Moore, University of EdinburghJon Oberlander, University of EdinburghKristina Striegnitz, Union College

Linguistic Field(s): Computational Linguistics

Page Updated: