linguistlist.org>
Editor for this issue: Jody Huellmantel <jodylinguistlist.org>
cs.rutgers.edu>________________________________________________________________ CALL FOR PARTICIPATION Workshop on Machine Translation Evaluation in conjunction with NAACL-2001 WORKSHOP ON MT EVALUATION: Hands-On Evaluation 3 June, 2001 Pittsburgh, PA United States MOTIVATION Evaluation of language tools, particularly tools that generate language, remains an interesting and general problem. Machine Translation (MT) is a prime example. Approaches to evaluating MT are even more plentiful than approaches to MT itself; the number of evaluations and range of variants is confusing to anyone considering an evaluation. In an effort to systematize MT evaluation, the NSF-funded ISLE project has created a taxonomy of evaluation-related features and measures. Unfortunately, however, many prior evaluations do not include an adequate specification of important aspects such as evaluation process complexity, cost, variance of score, etc. In an effort to drive MT evaluation to the next level, this workshop will focus on exercising with methods of acquiring such information for several important MT evaluation measures. The workshop thus embodies the challenge of Hands-On Evaluation, within the context of the framework being developed by the ISLE MT Evaluation effort. The workshop follows a workshop on MT Evaluation held at the AMTA Conference in Cuernavaca, Mexico, in October 2000, and a subsequent workshop being planned for April 2001 in Geneva. STRUCTURE OF THE WORKSHOP The first part of the workshop will introduce the ISLE MT Evaluation effort, funded by NSF and the EU, to create a general framework of characteristics in terms of which MT evaluations, past and future, can be described and classified. The framework, whose antecedents are the JEIDA and EAGLES reports, consists of taxonomies of increasingly specific features, with associated measures and pointers to systems. The discussion will review the current state of the classification effort as well as review the MT evaluation history from which it was drawn. The second, principal, part of the workshop will focus on real-world evaluation. In an effort to facilitate common ground for discussion, participants will be given specific evaluation exercises, defined by the taxonomy and recent MT evaluation trends. In addition, they will be given a set of texts generated by MT systems and human reference translations. They will be asked, during the workshop, to perform given evaluation exercises with the given data. This common framework will give insights into the evaluation process and useful metrics for driving the development process. The results of the exercises will then be presented by the participants, synthesized into a uniform description of each evaluation, and added to the ISLE taxonomy, which has been made available on the web for future analysis in MT evaluation. The results of the workshop will also be incorporated into a publicly available resource and the workbook from the workshop will be able to be used by teachers of evaluation and MT. QUESTIONS AND ISSUES Since this is a hands-on workshop, participants will be asked to submit an intent to participate. At that time, they will be able to download the relevant data for review. During the workshop, they will be given a series of exercises and split into teams for working these exercises. The result of the workshop will be at least one paper which addresses the following threads of investigation within the framework: * What is the variance inherent in an evaluation measure? * How complex is it to employ a measure? * What task(s) is the evaluation measure suited to? * What kinds of tools automate the evaluation process? * What kind of metrics are useful for users versus system developers? * How can we use the evaluation process to speed up or improve the MT development process? * What kind of impacts does real-world data have? * How can we evaluate MT when MT is a small part of the data flow? * How independent is MT of the subsequent processing? That is, cleaning up the data improves performance, but does it improve it enough? How do we quantify that? TO REGISTER Since this is a hands-on workshop, no papers are being solicited. Participants will be expected to take part in the exercises and report their conclusions. They will additionally be encouraged to contribute to a summary paper of the workshop proceedings. The data will be sent to participants in advance of the workshop, with instructions on what to do and what to prepare. The amount of work required should not exceed 4 hours (much less than paper preparation). To register an intent to participate, please send a paragraph outlining your interest in MT, experience with MT evaluation, knowledge of either Spanish or Arabic, and the following contact information to Flo Reeder (contact info below): * name * address * e-mail address * knowledge of other foreign languages * translation domain specialization Participants will need to register for the workshop as part of their NAACL registration. IMPORTANT DATES Intent to Participate: April 16, 2001 Release of Data: April 23, 2001 Workshop date: June 3, 2001 CONTACT POINTS Florence Reeder MITRE Corporation 1820 Dolley Madison Blvd. McLean, VA 22102-3481 TEL: 703-883-7156 FAX: 703-883-1379 EMAIL: freederMail to author|Respond to list|Read more issues|LINGUIST home page|Top of issue
illigal.ge.uiuc.edu>2nd Call for participation ROPNET-2001 REPRESENTAITONS AND OPERATORS FOR NETWORK PROBLEMS Bird-of-a-feather Workshop at the Genetic and Evolutionary Computation Conference 2001 ( GECCO-2001 ) San Francisco, California, July 7 - 11, 2001 (Saturday - Wednesday) A recombination of the Sixth Annual Genetic Programming Conference (GP-2001) and the International Conference on Genetic Algorithms (ICGA-2001) organized by Franz Rothlauf to be held on Saturday, July 7, 2001 WORKSHOP SUMMARY Finding good solutions for network design problems is important in many fields such as telecommunications, computer, backbone access, transportation and distribution networks. Over the last years genetic algorithms have been applied with success to a wide variety of these different problems. One of the major design issues is how the network could be represented as an artificial chromosome and what kind of operators could be defined on the chromosome. The workshop is intended to give an overview over the existing approaches and to discuss various representations and operators in the context of genetic and evolutionary computation. It should compare theoretical properties and empirical performance characteristics of different representations and operators and try to find explanations for performance differences of a genetic algorithm. The workshop will be focused on representations and operators for network problems, but it welcomes interesting contributions to encoding issues that are meaningful for network representations. PARTICIPATION Presentations will be selected according to the submitted 10-page papers which will be reviewed by at least two members of the international program committee. Accepted papers will be available in electronic form before the workshop. Abbreviated versions of the papers will be later published in the workshop proceedings. The length of each paper will be determined by the number of accepted papers. For more detailed submission guidelines and recent updates, see the workshop pages at http://btw6x2.oec.uni-bayreuth.de/ropnet/ IMPORTANT DATES (subject to change) Paper submission deadline: March 1, 2001 Decisions will be mailed by: April 1, 2001 Submissions of camera-ready papers: April 20, 2001 WORKSHOP CHAIR Franz Rothlauf Department of Information Systems University of Bayreuth 95445 Bayreuthn Germany rothlaufMail to author|Respond to list|Read more issues|LINGUIST home page|Top of issue