linguistlist.org>
Editor for this issue: Jody Huellmantel <jodylinguistlist.org>
cs.rutgers.edu>ACL/EACL 2001 Workshop 8th EUROPEAN WORKSHOP ON NATURAL LANGUAGE GENERATION 6-7 July 2001 Toulouse, France http://www.cs.unca.edu/~bruce/acl01/NLG.html - ---------------------------------------------------------------------- Natural language generation (NLG) constitutes the production of meaningful texts in natural languages from some underlying non-linguistic representation of information. Accomplishing this goal may be envisioned for a number of different purposes, including standardized and/or multi-lingual reports, summaries, machine translation, dialog applications, and embedding in multi-media and hypertext environments. Consequently, the automated production of language is associated with a large number of highly diverse tasks whose appropriate orchestration in high quality poses a variety of theoretical and practical problems. Relevant issues include content selection, text organization, the production of referring expressions, aggregation, lexicalization, and surface realization, as well as coordination with other media. This workshop is part of a bi-annual series of workshops about natural language generation that runs since 1987. Previous European workshops have been held at Royaumont, Edinburgh, Judenstein, Pisa, Leiden, Duisburg, and Toulouse. The goal of the workshop is to be an informal meeting which facilitates the dissemination of knowledge and expertise in the field. The workshop will focus on the following topics: * Search methods for NLG (in content planning and realization) There seems to be a substantial discrepancy between application-oriented systems and principled approaches to NLG. Accomodating a standard pipeline architecture with suitable heuristic preferences to the intended functionality of a system stands in contrast to several principled approaches to searching which have been tried out so far. These include blackboard architectures, constraint propagation and, more recently genetic algorithms and statistical techniques. A comparison of these methods in terms of their potential and limitations is likely to improve understanding about this issue. Gained insights could prove fruitful for building applications in a more general and, thus, better reusable way, especially in large-scale applications such as summarization and machine translation. * Differences in information organization between source and presentation specifications (and methods to bridge between these) Whether the generation task is to verbally express contents of some knowledge base or to produce multi-lingual presentations from language-neutral or similar representations, there are strong similarities in building the target representations: In the overwhelming number of cases, the ordering and embedding of elements in the source representation is reflected by the ordering and embedding of their corresponding realizations at the surface. Often, this reflection is systematic, many times even simple. But a few cases prove complex and involve a major restructuring of the surface structure when compared to the source structure. A major emphasis of this topic is on collecting such complex cases, identifying commonalities between them and discussing restructuring techniques. Accepted papers on these topics will be scheduled for presentation. The majority of the time will be devoted to discussions, either in sequence or in parallel, depending on the number of participants. We are considering organizing a panel. For the focus topics above, we will contact a number of competent researchers to address the topic from a specific perspective according to their experience. In addition, we will ask some of them to prepare material / concrete examples for discussions. WORKSHOP CHAIRS Helmut Horacek Univ. of the Saarland Nicolas Nicolov IBM T.J. Watson Research Center Leo Wanner Univ. of Stuttgart PROGRAMME COMMITTEE John Bateman Univ. of Bremen Dan Cristea Univ. of Iasi Robert Dale Macquarie University Laurence Danlos Universite Paris 7 Marc Dymetman Xerox Research Centre Europe, Grenoble Michael Elhadad Ben-Gurion Univ. Kristiina Jokinen Univ. of Art and Design Helsinki Richard Kittredge Univ. of Montreal & CoGenTex Daniel Marcu ISI, Univ. of Southern California Chris Mellish Univ. of Edinburgh Sergei Nirenburg CRL, New Mexico Owen Rambow AT&T Research Ehud Reiter Univ. of Aberdeen Manfred Stede Technical University of Berlin Michael Zock LIMSI, CNRS SUBMISSIONS Papers describing original work in the area of NLG in particular related to the workshop focus topics above should be submitted electronically. Papers should be 6-8 pages long in PDF format. We recommend a A4, two-column format like the ACL proceedings: http://acl2001.dfki.de/style/ We also invite poster submissions (free format, up to 6 page, PDF). The submissions should be associated with a cover email containing the following information (ASCII text): # TITLE: <title of the paper> # AUTHORS: <list of authors> # EMAIL: <email of author(s) for correspondence> # KEYWORDS: <keywords, topic sub-areas, ...> # ABSTRACT: <abstract of the paper> Send your submission to Leo Wanner <wannerloMail to author|Respond to list|Read more issues|LINGUIST home page|Top of issue
cs.rutgers.edu>Call for Papers Workshop on Evaluation for Language and Dialogue Systems ACL/EACL 2001 Toulouse, France July 6-7, 2001 WORKSHOP GOALS The aim of this two day workshop is to identify and to synthesize current needs for language-technology evaluation. The first day of the workshop will focus on one of the most challenging current issues in language engineering: the evaluation of dialogue systems and models. The second day will extend the discussion to address the problem of evaluation in language engineering more broadly and on more theoretical grounds. The space of possible dialogues is enormous, even for limited domains like travel information servers. The generalization of evaluation methodologies across different application domains and languages is an open problem. Review of published evaluations of dialogue models and systems suggests that usability techniques are the standard method. Dialogue-based system are often evaluated in terms of standard, objective usability metrics, such as task-completion time and number of user actions. In the past, researchers have proposed and debated theory-based methods for modifying and testing the underlying dialogue model, but the most widely used method of evaluation is usability testing, although more precise and empirical methods for evaluating the effectiveness of dialogue models have been proposed. For task-based interaction, typical measures of effectiveness are time-to-completion and task outcome, but the evaluation should focus on user satisfaction rather than on arbitrary effectiveness measurements.Indeed, the problems faced in current approaches to measurement of effectiveness dialogue models and systems include: Direct measures are unhelpful because efficient performance on the nominal task may not represent the most effective interaction Indirect measures usually rely on judgment and are vulnerable to weak relationships between the inputs and outputs Subjective measures are unreliable and domain-specific For its first day, the workshop organizers solicit papers on these issues, with particular emphasis on methods that go beyond usability testing to address the underlying dialogue model. Representative questions to be addressed include: o How do we deal with the combinatorial explosion of dialogue states? o How can satisfaction be measured with respect to underlying dialogue models? o Are there useful direct measures of dialogue properties that do not depend on task efficiency? o What is the role of agent-based simulation in evaluation of dialogue models? Of course, the problems faced in evaluating dialogue and system models are found in other domains of language engineering, even for non-interactive processes such as part-of-speech tagging, parsing, semantic disambiguation, information extration, speech transcription, and audio document indexing. So the issue of evaluation can be viewed at a more generic level, raising fundamental, theoretical questions such as: o What are the interest and benefits of evaluation for language engineering? o Do we really need these specific methodologies, since a form of evaluation sould always be present in any scientific investigation? o If evaluation is needed in language engineering, is it the case for all domains? o What form should it take? Technology evaluation (task-oriented in laboratory environment) or field/user Evaluation (complete systems in real-life conditions)? We have seen before that the the evaluation of dialogue models is still unsolved, but for domains where metrics already exists, are they satisfactory and sufficient? How can we take into account or abstract from the subjective factor introduced by human operators in the process? Do similarity measures and standards offer appropriate answers to this problem? Most of the efforts focus on evaluating process, but what about the issue of language resources evaluation? For its second day of work, the workshop organizers solicit papers on these issues, with the intent to address the problem of evaluation both from a broader perspective (including novel applications domains for evaluation, new metrics for known tasks and resource evaluation) and a more theoretical point of view (including formal theory of evaluation and infrastructural needs of language engineering). NOTE: People who would like to submit a paper on lexical semantic disambiguation evaluation should consider the parallel workshop, on July 5-6, for the closure of the SENSEVAL-2 evaluation campaign. - ----------------------------------------------------------- WORKSHOP ORGANIZATION The organization of each of the two days of the workshop will reflect the workshop's two main themes. Each day will begin with a session of presentations of selected papers and follow with panel discussions to synthesize and develop possible methodologies from additional selected workshop papers. WORKSHOP PARTICIPATION The workshop seeks participation from people involved or interested in the problem of evaluation in language processing and the research and industrial communities that study and implement dialogue models for natural-language interaction systems. The first part of the workshop will specifically draw on the natural-language interaction community, for instance like the one developing at the confluence of SIGdial and SIGCHI, which will find in this workshop an atmosphere more flavored by computational-linguistics related issues (see, for example, the First SIGdialWorkshop on Discourse and Dialogue). The second part of the workshop is intended to provide a forum for a broader audience more in the spirit of the one that attended the LREC'2000 Satellite Workshop on Evaluation (see http://www.limsi.fr/TLP/CLASS), in particular offering an opportunity to people involved in language engineering evaluation (e.g ., the CLASS audience) in the context of national or transnational projects or programs, both in Europe and abroad. - ----------------------------------------------------------- SUBMISSION DETAILS Paper submissions should follow the two-column format of ACL proceedings and should not exceed eight (8) pages, including references. We strongly recommend the use of ACL LaTeX style files or Microsoft Word Style files tailored for this year's conference. They are available from the ACL-2001 program committee Web site at http://acl2001.dfki.de/style/. Papers should be submitted electronically, as either a LaTeX, Word or PDF file to either: Patrick Paroubek, papMail to author|Respond to list|Read more issues|LINGUIST home page|Top of issue