* * * * * * * * * * * * * * * * * * * * * * * *
LINGUIST List logo Eastern Michigan University Wayne State University *
* People & Organizations * Jobs * Calls & Conferences * Publications * Language Resources * Text & Computer Tools * Teaching & Learning * Mailing Lists * Search *
* *

LINGUIST List 25.1494

Sat Mar 29 2014

Calls: Computational Linguistics/Ireland

Editor for this issue: Bryn Hauk <brynlinguistlist.org>

Date: 27-Mar-2014
From: Manfred Stede <stedeuni-potsdam.de>
Subject: 8th Linguistic Annotation Workshop
E-mail this message to a friend

Full Title: 8th Linguistic Annotation Workshop
Short Title: LAW VIII 2014

Date: 23-Aug-2014 - 24-Aug-2014
Location: Dublin, Ireland
Contact Person: Manfred Stede
Meeting Email: < click here to access email >
Web Site: http://www.ling.uni-potsdam.de/acl-lab/law2014

Linguistic Field(s): Computational Linguistics

Call Deadline: 02-May-2014

Meeting Description:

Linguistic annotation of natural language corpora is the backbone of supervised methods for statistical natural language processing. It also provides valuable data for evaluation of both rule-based and supervised systems and can help formalize and study linguistic phenomena.

The LAW workshop, this year co-located with the COLING conference in Dublin, provides a forum for presentation and discussion of innovative research on all aspects of linguistic annotation, including creation/evaluation of annotation schemes, methods for automatic and manual annotation, use and evaluation of annotation software and frameworks, representation of linguistic data and annotations, etc.

Call for Papers:

We welcome submissions of long (8 pages) and short (4 pages) papers, posters, and demonstrations, relating to any aspect of linguistic annotation, including:

(a) Annotation procedures:
- Innovative automated and manual strategies for annotation
- Machine learning and knowledge-based methods for automation of corpus annotation
- Creation, maintenance, and interactive exploration of annotation structures and annotated data

(b) Annotation evaluation:
- Inter-annotator agreement and other evaluation metrics and strategies
- Qualitative evaluation of linguistic representation

(c) Annotation access and use:
- Representation formats/structures for merged annotations of different phenomena, and means to explore/manipulate them
- Linguistic considerations for merging annotations of distinct phenomena

(d) Annotation guidelines and standards:
- Best practices for annotation procedures and/or development and documentation of annotation schemes
- Interoperability of annotation formats and/or frameworks among different systems as well as different tasks, frameworks, modalities, and languages

(e) Annotation software and frameworks:
- Development, evaluation and/or innovative use of annotation software frameworks

(f) Annotation schemes:
- New and innovative annotation schemes
- Comparison of annotation schemes

Workshop Theme:

This year, we in particular welcome contributions that address the workshop theme: The good, the bad, and the perfect: How good does annotation need to be?

It has been said that the perfect is the enemy of the good. This may be true for some machine learning applications where a small amount of rough annotation gives good results, but it also may be used to justify low quality annotation or give higher priority or higher amounts of funding to machine learning than to human annotation.

We solicit evidence for and against 'The perfect is the enemy of the good'.

In favor of high quality annotation, Manning (2011) suggests that the largest opportunity for improvement in part-of-speech tagging lies in improving the tag set and the accuracy of annotation. (But he also suggests that perfect annotation of words into discrete lexical categories is not possible because some words do not fall cleanly into one category.) Reidsma and Carletta (2008) advocate caution in deciding how good annotation needs to be. They show that low agreement among annotators may not be harmful to machine learning as long as the disagreements are random, whereas disagreements that follow patterns can lead machine learning astray even when agreement among annotators is high. In a related vein, Min and Grishman (2012) show that it can be more cost-effective for machine learning to have lots of single-pass less-accurate annotation, than a smaller amount of more-accurate adjudicated annotation. Finally, one recent trend focuses on coarse-grained annotation schemes (McDonald et al., 2013; Petrov et al., 2013; Schneider et al., 2013) to speed up annotation and/or benefit cross-lingual training. Coarse-grained annotation schemes are attractive because they are easy to learn, but are they suitable for all applications?

(References: see worskhop website)

Read more issues|LINGUIST home page|Top of issue

Page Updated: 29-Mar-2014

Supported in part by the National Science Foundation       About LINGUIST    |   Contact Us       ILIT Logo
While the LINGUIST List makes every effort to ensure the linguistic relevance of sites listed on its pages, it cannot vouch for their contents.