LINGUIST List 11.268

Mon Feb 7 2000

Calls: Tests/Lang Understanding Systems, Machine Trans

Editor for this issue: Karen Milligan <karenlinguistlist.org>

Directory

1. Priscilla Rasmussen, ANLP/NAACL2000 2nd Workshop Call for Papers
2. Priscilla Rasmussen, AMTA-2000 Call for Participation

			Second Call for Papers

	Workshop on Reading Comprehension Tests as Evaluation for
		Computer-Based Language Understanding Systems

	 Thursday, May 4th, 2000, Seattle, Washington, USA 
	(post-conference workshop in conjunction with ANLP-NAACL2000)


Reading Comprehension tests, such as the one below, are designed to help
evaluate a reader's understanding of a text passage.

 How Maple Syrup is Made

 Maple syrup comes from sugar maple trees. At one time, maple
 syrup was used to make sugar. This is why the tree is called a
 "sugar" maple tree.

 Sugar maple trees make sap. Farmers collect the sap. The best
 time to collect sap is in February and March. The nights must be
 cold and the days warm.

 The farmer drills a few small holes in each tree. He puts a
 spout in each hole. Then he hangs a bucket on the end of each
 spout. The bucket has a cover to keep rain and snow out. The sap
 drips into the bucket. About 10 gallons of sap come from each
 hole.

 1. Who collects maple sap? (Farmers)
 2. What does the farmer hang from a spout? (A bucket)
 3. When is sap collected? (February and March)
 4. Where does the maple sap come from? (Sugar maple trees)
 5. Why is the bucket covered? (to keep rain and snow out)

Such tests exist in many languages, have human performance benchmarks
associated with them, and come in a variety of types (short-answer,
multiple choice) and levels of difficulty. In addition, they are
generally written to make each story and set of questions
self-contained, in order to require as little outside knowledge as
possible to answer the questions.

The focus of the proposed workshop will be to explore the following
questions:

- Can such exams be used to evaluate computer-based language
 understanding effectively and efficiently? 
- Would they provide an impetus and test bed for interesting and
 useful research? 
- Are they too hard for current technology? 
- Or are they too easy, such that simple hacks can score high,
 although there is clearly no understanding involved? 

The most direct method of exploring these questions is to choose a set
of tests and build a system that takes these tests. Some preliminary
results indicate that such tests are tractable, but not trivial and
that linguistic processing is helpful (Hirschman, et al. ACL-99). A
test set, evaluation routines, prototype system, and documentation are
available upon request to lightmitre.org.

We hope that a number of submissions will present results based on
actual reading comprehension systems. In addition, we encourage
submissions that report on other kinds of tests or similar tests in
other languages, or that address our list of questions by other
means. Note that submissions are encouraged that describe work in 
progress with preliminary empirical results.


Invited speaker:

Karen Kukich (Educational Testing Service)

"NLP Tools for Identifying Reading Comprehension Skills"


Format for Submission

Authors are asked to submit previously unpublished papers only; a
workshop proceedings will be published. Our target submission length
is 2000 words but both shorter and longer submissions will also be
considered. Electronic submission of postscript will be accepted.
Hard copy submissions should include 4 copies of the paper. Since the
papers will be reviewed anonymously, please do not place the author
name on the paper. Instead include a separate title page with title,
abstract, author, and e-mail address. Unless requested otherwise,
notification of acceptance will be sent electronically to the first
author. Parallel submission is unproblematic; however if your paper
is accepted to this workshop and you decide to present it here, we
will ask you to withdraw it from any other events.

Important Dates

Deadline for submission: February 11th, 2000
Notification of authors: March 1st, 2000
Final versions due: March 10th, 2000


Address for Submission and Further Information 

Marc Light 
The MITRE Corporation
202 Burlington Rd.
M/S K329
Bedford, MA 01730
USA
Phone: 1-781-271-5579
lightmitre.org

(The mailing list, read-complinus.mitre.org, has been set up to
discuss reading comprehension tests as evaluation for computer-based
language understanding systems. It is open subscription and
unmoderated. To subscribe, send email to majordomolinus.mitre.org
with 'subscribe read-comp' in the body.)



Program Committee:

Eric Brill
Eugene Charniak
Mary Harper
Marc Light (chair)
Ellen Riloff
Ellen Voorhees

		--- PRELIMINARY CALL FOR PARTICIPATION ---
	The Association for Machine Translation in the Americas

AMTA-2000 Conference
Location: Cuernavaca, Mexico
Dates: October 10-14, 2000


Envisioning Machine Translation in the Information Future


The Association for Machine Translation in the Americas (AMTA) is happy to
announce the plans for the fourth biennial conference, planned for October
10-14 at Mision del Sol, near Cuernavaca, Mexico.

The theme of AMTA-2000 is "Envisioning MT in the Information Future." The
focus will be on the articulation of future visions of MT: in the '00
decade, the 21st century, and even in the third millennium. Ubiquitous,
instant internet access will be available very soon from a host of
appliances and apparel. Later on, ways of thinking about the universe of
information will transcend our current metaphors of networks, clients,
servers, and communication. How will these and other possible paths into
the future affect our exponential need for translation? Will the process of
translation become transparent? How long before we each have a true
babelfish in our ear? Will the quality ceiling finally be broken by
incremental improvements, or by an as yet unimagined breakthrough? Will
translation even be necessary - will globalization lead to a single
language, or will translation allow for the growth of local languages?

Every current topic in multilingual information processing is germane to
this discourse, especially as it points the way to the near term and long
term role of MT in the information world of the future.

As in the past, the conference will feature a lively and engaging variety of
invited speakers, panel discussions, demonstrations, workshops, tutorials,
and technical papers by researchers, developers, and users.

AMTA invites everyone interested in machine translation to participate in
this conference - developers, researchers, users, professional translators,
managers, marketing experts - anyone who has a stake in the vision of an
information world in which language issues become transparent to the
information consumer. We especially invite users to share their experiences,
developers to describe what is happening in the internet marketplace,
researchers looking to new capabilities, and visionaries to describe the
future.

Colleagues from Latin America who are involved in or interested in MT or
other human language technologies, are especially welcome to participate and
submit papers, workshops, tutorials, and demonstrations.

We also welcome and encourage participation by members of AMTA's sister
organizations, AAMT in Asia and EAMT in Europe. We also urge people working
in related areas in information processing to participate as well.

For complete information about the conference, please see the web site at:
http://www.isi.edu/natural-language/conferences/
<http://www.isi.edu/natural-language/conferences/>; 


David Farwell, General Chair
John S. White, Program Chair