LINGUIST List 10.5

Tue Jan 5 1999

Calls: Semantics/Pragmatics, Ergo's Parsing Contest

Editor for this issue: Jody Huellmantel <jodylinguistlist.org>


As a matter of policy, LINGUIST discourages the use of abbreviations or acronyms in conference announcements unless they are explained in the text.

Directory

  1. Robert van Rooy, Amstelogue'99 Second Call for papers
  2. Philip A. Bralich, Ph.D., Ergo's 1st ANNUAL PARSING CONTEST

Message 1: Amstelogue'99 Second Call for papers

Date: Mon, 04 Jan 1999 18:56:09 +0100
From: Robert van Rooy <vanrooybs20.bs.uva.nl>
Subject: Amstelogue'99 Second Call for papers

 SECOND CALL FOR PAPERS


 AMSTELOGUE'99

Amsterdam Workshop on the Semantics and Pragmatics of Dialogue

 May 7-9, 1999
 University of Amsterdam


Invited Speakers:
 Hans Kamp (Stuttgart, Germany)
 Hannes Rieser (Bielefeld, Germany)
 Munindar Singh (North Carolina, USA)
 David Traum (Maryland, USA)

 **

Amstelogue '99 will be a sequel to the successful dialogue workshops
Mundial'97 (Muenchen) and Twendial'98 (Twente). Like its predecessors,
Amstelogue '99 aims at bringing together researchers from different
fields on the topic of semantics and pragmatics of dialogue. These
fields include artificial intelligence, formal semantics/pragmatics
and computational/applied linguistics.

In order to increase the cohesion of the talks across the three
themes, we put forward, in addition, the over-all topic Speech
Acts/Dialogue Moves.

 **

This year, the workshop will be organised around the following
three themes:

I Formal Semantics of Dialogue

This covers topics such as models of common ground/ mutual belief, the
semantics of goals, intentions and commitments in communication,
treatment of dialogue moves in a formal semantic framework, the
semantics of cross-speaker anaphora.

II Dialogue Systems

This covers e.g. knowledge representation for multi-agent interaction,
dialogue management in practical implementations, semantics and
pragmatics of natural language in automated dialogue systems.

III Dialogue Analysis (Empirical)

This covers topics such as turn-taking, categorisation of dialogue
moves or speech acts in real (i.e., non-constructed) dialogues,
aspects of institutional interaction, characteristics of
multi-participant conversations, the role of nonlinguistic interaction
in communication.

 **

Program committee:
 Laila Dybkjaer (Odense, Denmark)
 Herman Hendriks (Amsterdam, The Netherlands)
 Gerd Jaeger (Berlin, Germany)
 Hans Kamp (Stuttgart, Germany)
 Jan van Kuppevelt (Stuttgart, Germany) (Chair)
 Hannes Rieser (Bielefeld, Germany)
 Candy Sidner (Cambridge MA, USA)
 Munindar Singh (North Carolina, USA)
 David Traum (Maryland, USA)
 Bonnie Webber (Edinburgh, Scotland)

 **


The collected abstracts of the talks will be provided as workshop
proceedings at the start of the workshop. Furthermore, we will have a
proceedings of the full papers of accepted abstracts. With the Journal
of Semantics we have agreed to publish a special issue, subject to
their double reviewing procedure.

 **

Related event: the day before the workshop, 6 May, will be occupied to
the TRINDI-project workshop. For further information see their webside
http://www.ling.gu.se/research/projects/trindi/

 **

Procedure: Anyone who is working on the semantics or pragmatics of
dialogue is kindly invited to send us an ABSTRACT of (at most) 5 pages
(400 a 500 words/page) in latex or ascii (talks will be 45 minutes) on
one of the above mentioned topics, or a related topic. Please indicate
for what theme you are opting. Abstracts clearly relating to the
over-all topic of Speech Acts will be preferred.

deadline: 1 February 1999
send to: amsteloghum.uva.nl

 **

Registration and Accommodation

The registration fee will be Dfl 150 (70 ECU). This fee includes a
copy of the Proceedings of the conference, and lunch. A special
student fee is available on request.

 **

Organisation:

Noor van Leusen, Dept. of Computational Linguistics, U.v.A,
noorai.let.uva.nl
Robert van Rooy, Dept. of Philosophy, U.v.A, vanrooyphilo.uva.nl
Henk Zeevat, Dept. of Computational Linguistics, U.v.A,
henkai.let.uva.nl

More information on website: http://earth.hum.uva.nl/~amstelog

Sponsors:

Dutch Research School in Logic
Dutch Research Foundation
Faculty of Humanities, University of Amsterdam
TRINDI
DISC
Mail to author|Respond to list|Read more issues|LINGUIST home page|Top of issue

Message 2: Ergo's 1st ANNUAL PARSING CONTEST

Date: Mon, 4 Jan 1999 15:35:45 -1000 (HST)
From: Philip A. Bralich, Ph.D. <bralichhawaii.edu>
Subject: Ergo's 1st ANNUAL PARSING CONTEST


Ergo Linguistic Technologies would like to announce its first annual
parsing contest based on a fixed set of sentences and a fixed set of
tasks to be performed on that set of sentences. The area of NLP to be
explored is that of increased syntactic analysis to provide: 1)
improvements in navigation and control technology through more complex
grammar, 2) improvements in the implementation of question/answer,
statement/response dialogs with computers and computer characters, and
3) improvements in web and database searching using natural language
queries.

The contest will be based on a comparison of results for parses of a
fixed set of sentences (included at end of this message) and various
tasks that can be performed as a result of those parses. That is, the
comparison will be based on the actual parse tree and the ability to
use that parsed output to generate theory independent parse trees and
output and to perform various NLP tasks. The judging will be based on
the standards for evaluating NLP that have been proposed previously on
this list by myself and Derek Bickerton and which are currently being
developed into an ISO standard for the Virtual Reality Modeling
Language (VRML) as part of the VRML Consortium's development efforts
(http://www.vrml.org/WorkingGroups/ NLP-ANIM). The standards proposed
are theory and field independent standards which allow both linguists
and non-linguists to evaluate NLP systems in the areas of navigation
and control, question/answer dialogues, and database and web
searching. I will also be at the annual meeting of the Linguistic
Society of America this week in Los Angeles for those who would like
to discuss this in more detail.

The sentences chosen for this contest are rather simple, but as we
find more and more parsers that can accomplish the tasks on this list,
we will add more complex sentences and tasks to the list. Please, be
aware that systems that may be designed for large corpora of
unrestricted text actually cannot work in this domain. Thus, while
such systems may be useful for certain searching tasks, they are not
useful in the domain explored in this contest and this is
evidenced by their inability to perform on tests such as the one
provide here.

The full contest instructions and an HTML document of Ergo's results
in this area can be found at http://www.ergo-ling.com . The standards
were designed to allow the developers of a parsing system (statistical
or syntactic) to demonstrate the thoroughness and accuracy of the
parses they produce by using the parsed output to perform a number of
straightforward, traditional syntactic tasks such as changing a
statement to a question or an active to a passive as well as
demonstrating an ability to create standard trees (Using the Penn
Treebank II guidelines) and standard grammatical analyses. All the
standards chosen were chosen to be theory independent measures of the
accuracy of a parse through the use of standard and ordinary
grammatical and syntactic output.

The contest officially begins on January 15th and will be closed on
March 31st. This will allow developers 2.5 months to develop tools
and to work with trouble spots that they may have with the set of
sentences offered in this contest. The contest will be offered in
subsequent years from January to March. As time develops we hope the
parsers, the contest rules, and the test sentences will all grow in
sophistication and scope. However, as most parsers have existed many
more years than ours, it is reasonable to think these tools exist
already.

THE CONTEST RULES:
Anyone who joins must submit an HTML document and the parser (source
code only) that created it. The parser can be in any format but it
must require a minimum of effort for the contest judges to set up and
run. For example, a WIN95 Interface that takes input files and
produces the html output file would be considered a minimum effort
parser. There will be tests to ensure that the output is genuine
parsed output rather than a synthesis such as a series of print calls
that merely present the correct output for a particular string rather
than generating it.

The HTML files of all contestants will be made available at the Ergo
web site (http://www.ergo-ling.com). Those who wish to join even
though their parsing system is not robust or complete enough for all
the tasks or all the sentences in the contest are also welcome to
join. Reviewers will then look at these documents as promising
parsers for future contests. Their results will be posted on our web
site as well.

Judging will be based on the percentage of sentences that parsed, the
percentage of the tasks that are completed and on the accuracy of the
parses that result and the success on the parsing tasks. Currently,
the judges will be Derek Bickerton and myself, but we will welcome
others to join in the task. Because of the home court advantage of
the judges, there will be printed reports of the judging available on
the Ergo web site for review by the overall community of professionals
in this area. Complaints or criticisms will also be posted.

Anyone who would like to review the judging and the comments on the
judging are welcome to do so. Anyone who wishes to be a volunteer
judge may also contact us. However, the criteria for all judging will
be the accuracy of the parser in creating a correct parse of all the
sentences and completing all the tasks set forth in the test
materials.

We would like this contest to remain open not only to challengers but
also to those who would like to design and improve the contest itself
through the addition of more sentences or more tasks added to the
parsing task. There is one condition, however, on being able to this,
we will hold rigidly to the rule that those who would improve on or
add to the contest must first meet the original challenge at a minimum
level of 75% accuracy before being allowed to contribute. We are
starting with a small set of relatively simple sentences to make this
as available as possible to as many people as possible. In this
manner researchers in industry, academia, and government will be able
to compare their results without exposing any proprietary or
confidential information. We also do not want the contest to be
unduly influenced by those who would like to target some ideal of
parsing that is not thoroughly grounded in what is currently possible
in these domains.

At a Virtual Reality and Multi-Media Conference in Japan (VSMM
98), Ergo was awarded the "Best Technical Award" for its NLP
technology. I believe the main reason that judges and others were
able to notice this is because I was able to point out that "THE
ENTIRE FIELD OF VIRTUAL REALITY AND MULTI-MEDIA IS BEING HELD HOSTAGE
BY GRAMMAR." And then I went on to explain that the main reason many
VR and Multi-Meida sites and programs are not catching on is because
their users cannot ask even a simple question of the characters or
about the objects they encounter. Thus, a UNESCO virtual world such
as reconstructed cathedral will receive many visitors but they will
not stay and explore because they cannot ask even the simplest
questions like "How many stairs in this Cathedral?" "When was the
Nave built?" and so on. I then pointed out that while speech and
graphics were actually ready to work with such projects, the fact that
their grammatical abilities is so limited, no one is using them with
these products. The missing link between speech, VR and multi- media
and users actually talking to avatars and sites is GRAMMAR. When I
then demonstrated that this was so with the use of the Ergo tools, we
won the award. The main reason I am sponsoring this contest is so
that all linguists and NLP researchers who would like to paticipate in
this very large future source of jobs can do so as soon as possible.
So in order to stimulate research and interest this contest is
proposed.

WE WOULD ESPECIALLY LIKE TO INVITE PROFESSORS, STUDENTS, AND STAFF AT
CARNEGIE MELON, STANFORD, XEROX PARC, MICROSOFT, IBM, DRAGON, LEARNOUT
AND HAUSPIE, PHILIPS, MIT, SUN MICROSYSTEMS (JAVASPEECH GROUP), NEW
YORK UNIVERSITY, AND GEORGETOWN UNIVERSITY TO SUBMIT ENTRIES TO THIS
CONTEST. WE WILL BE HAPPY TO POST THEIR RESULTS AND WOULD ALSO BE
HAPPY TO TELL THE WORLD IF THEY CAN GENERATE A PARSE THAT IS BETTER
THAN OURS ON THE STANDARDS PROVIDED HERE. THIS IS A GREAT OPPORTUNITY
FOR STUDENTS AND JUNIOR STAFF TO WORK WITH EXTANT PARSERS TO COMBINE
AND EXTEND TOOLS INTO THESE VERY USEFUL AND PRACTICAL AREAS.

	THE SENTENCES

The full set of sentences for the contest is available at the
http://www.ergo-ling.com web site. This list contains five from each
of the three sections: 1) theory independent parsing, 2) navigation
and control, and 3) Question/answer, statement/response repartee. The
full list contains 105 sentences and will grow and be modified over
the years as this annual contest takes root.

Section 1:	Theory independent parsing. 
	1.	there is a dog on the porch
	2.	John's house is bigger than mary's house
	3.	the tall thin man in the office is reading a technical report
	4.	the man who mary likes is reading the book that john gave her
	5.	learning how to cope with stress is of primary importance in 
 the work world

Section 2:	Navigation and Control.
	1.	Erase all files that end in .doc
	2.	print the file called teach.doc
	3.	send an email to bob that says "meeting at eight"
	4.	send a fax to bob that says "there is a meeting at eight 
 tonight"
	5.	go to yahoo and find information about golf courses in Georgia

Section 3:	Question and Answer/Statement Response Repartee. 
	1.	bill's email is billserver.com
		what is bill's email address
		what is bill's email
	2.	john has romantic books
		what kind of books does john have
	3.	My appointment with bob is at six o'clock
		what time is my appointment
		what time are my appointments
	4.	the tall thin man in the office is reading a technical 
 report book
		what is the man reading
		what is the man doing
		is the man reading a report
		who is reading a report
	5.	John gave mary a book because it was her birthday
		who gave mary a book
		who did john give a book
		what did john give mary
		why did john give mary a book
		did john give mary a book
		did john give mary a book because it was her birthday
		did john give mary a pencil
		did john give mary a book because it was bob's birthday

Philip A. Bralich, Ph.D.
President and CEO
Ergo Linguistic Technologies
2800 Woodlawn Drive, Suite 175
Honolulu, HI 96822

Tel: (808)539-3920
Fax: (808)539-3924
bralichhawaii.edu
http://www.ergo-ling.com
Mail to author|Respond to list|Read more issues|LINGUIST home page|Top of issue