LINGUIST List 28.727
Tue Feb 07 2017
FYI: Call for Task Proposals - SemEval-2018
Editor for this issue: Yue Chen <yuelinguistlist.org>
Saif Mohammad <yue
Call for Task Proposals - SemEval-2018 E-mail this message to a friend
SemEval-2018: International Workshop on Semantic Evaluations
Call for Task Proposals:
We invite proposals for tasks to be run as part of SemEval-2018. SemEval (Semantic Evaluation) is an ongoing series of evaluations of computational semantic analysis systems, organized under the umbrella of SIGLEX, the Special Interest Group on the Lexicon of the Association for Computational Linguistics.
The SemEval evaluations explore the nature of meaning in natural languages in practical terms, by providing an emergent mechanism to identify the problems (e.g., how to characterize meaning and what is necessary to compute it) and to explore the strengths of possible solutions by means of standardized evaluation on shared datasets. SemEval evaluations initially focused on identifying word senses computationally, but have later grown to investigate the interrelationships among the elements in a sentence (e.g., semantic relations, semantic parsing, semantic role labeling), relations between sentences (e.g., coreference), and author attitudes (e.g., sentiment analysis), among other research directions.
For SemEval-2018, we welcome any task that can test an automatic system for semantic analysis of text, be it application-dependent or application-independent. We especially welcome tasks for different languages, cross-lingual tasks, tasks requiring semantic interpretation, and tasks with both intrinsic and application-based evaluation. See the websites of previous editions of SemEval to get an idea about the range of tasks explored, e.g., for SemEval-2017: http://alt.qcri.org/semeval2017/
We strongly encourage proposals based on pilot studies that have already generated initial data which can provide concrete examples and discuss the challenges of preparing the full task. In the event of receiving many proposals, preference will be given to tasks that have already run a pilot study for the proposed task.
We encourage the following aspects in task design:
- Application-oriented tasks:
We welcome tasks that are devoted to developing novel applications of computational semantics. We will encourage tasks that have a clearly defined end-user application showcasing and enhancing our understanding of computational semantics, as well as extending the current state-of-the-art.
- Umbrella tasks
In order to reduce fragmentation of similar tasks and increase community effort towards solving the underlying research problems, we encourage task organizers to propose larger tasks that include several related subtasks. For example, a Semantic Similarity umbrella task might include subtasks for different kinds of similarity and different languages. Similarly, a Sentiment Analysis umbrella task might include subtasks for Twitter, Product Reviews, and Service Reviews. We also welcome task proposals for umbrella tasks focusing on different aspects of the same phenomena. For example, an Attitude Inference task might have subtasks for detecting an author’s emotional state, the sentiment of their writing, and the writing’s objectivity. In addition, the program committee will actively encourage task organizers proposing similar tasks to combine their efforts into larger umbrella tasks.
Task proposals will be reviewed by experts, and the reviews will serve as the basis for acceptance decisions. In case of conflict, more innovative new tasks will be given preference over task re-runs. Task proposals will be evaluated on:
- Interest: Is the proposed task likely to attract a sufficient number of participants?
- Data: Are the plans for collecting data convincing? Will the resulting data be high quality? Will the data annotation be ready on time?
- Evaluation: Is the methodology for evaluation sound? Is the necessary infrastructure available or can it be built in time for the shared task?
- Impact: What is the expected impact of the data in this task on future research beyond the SemEval Workshop?
Task organizers are expected to provide to task participants format checkers and standard scorers. Moreover, in order to lower the obstacles to participation, we encourage task organizers to provide baseline systems that participants can use as a starting point. A baseline system typically contains code that reads the data, creates a baseline response (e.g., random guessing, majority class prediction, etc.), and outputs the evaluation results. Whenever possible, baseline systems should be written in widely used programming languages and/or should be implemented as a component for standard NLP pipelines such as UIMA or GATE.
New Tasks vs. Task Reruns
We welcome both new tasks and task reruns. For a new task, a major concern to be addressed in the proposal is whether it would be able to attract participants. For task reruns, the organizers should in their proposal defend the need for another iteration of their task, explain, for example, why there is need for a new form of evaluation (e.g., a new metric to test new phenomena, a new application-oriented scenario, etc.) or need to test on new types of data (e.g., social media, domain-specific corpora), whether there is significant expansion in scale over a previous trial run of the task, etc.
In the case of a rerun, we further discourage carrying over the same tasks year after year and just adding new subtasks as this can lead to the accumulation of too many subtasks. Evaluating on a different dataset with the same task formulation typically should not be considered a separate subtask.
Tasks that have already run for three years will not be accepted for SemEval-2018. If however the organizers estimate there is need for another iteration of their task, they are welcome to submit a task rerun proposal for SemEval-2019 (the calendar for submissions will be announced in Feb, 2018). Solid justification for the re-run will be needed highlighting its novel aspects compared to previous editions, in respect to the criteria discussed above.
1st Call for task proposals February 10, 2017
Task proposals due April 3, 2017
Task selection notification May 22, 2017
Tasks merged June 12, 2017
Trial data ready July 31, 2017
Training data ready September 4, 2017
Test data ready December 1, 2017
Evaluation start January 10, 2018
Evaluation end January 31, 2018
Paper submission due February 28, 2018
Paper reviews due March 31, 2018
Camera ready due April 30, 2018
SemEval workshop Summer 2018
The SemEval-2018 Workshop will be co-located with a major NLP conference in 2018.
The task proposals should be a self-contained document of roughly 4-8 pages. Each proposal should contain the following:
-- A summary of the task in general
-- Motivation, why this task is needed and which communities would be interested in participating
--What the expected impact of the task will be
- Data & Resources
-- How the training/testing data will be built and/or procured
-- What source texts/corpora are going to be used? Please discuss whether existing corpora have been re-used or not.
-- How much data is going to be produced
-- How will quality of the data be ensured and evaluated
-- An example of how a data instance would look like
-- The anticipated availability of the necessary resources to the participants (copyright, etc.)
-- The resources required to prepare the task (computation and annotation time, costs of annotations, etc.) and their availability
- Pilot Task
-- Details of the pilot task, if any
-- What lessons were learned and how these will impact the future task design
-- The evaluation methodology to be used, including clear evaluation criteria
- For Task Reruns
-- Justification for why a new iteration of the task is needed, using the criteria discussed above
-- What will differ from the previous instance
-- The expected impact of the re-run compared with the previous instance
- Task organizers
-- Names, affiliations, brief description of research interests and relevant experience, contact information (email).
Proposals will be reviewed by an independent group of area experts who may not have familiarity with recent SemEval tasks and therefore, all proposals should be written in a self-explanatory manner and contain sufficient examples.
Please submit proposals by mail in PDF format to the SemEval email address: semeval-organizers
In case you are not sure whether a task is suitable for SemEval, please feel free to get in touch to discuss your idea.
Marianna Apidianaki, LIMSI-CNRS & University of Pennsylvania
Saif M. Mohammad, National Research Council Canada
Steven Bethard, University of Alabama at Birmingham
Marine Carpuat, University of Maryland
The SemEval discussion group:
Please join our discussion group at semeval3
googlegroups.com in order to receive announcements and participate in discussions.
The SemEval-2018 Website: http://alt.qcri.org/semeval2018/
Linguistic Field(s): Computational Linguistics
Page Updated: 07-Feb-2017