LINGUIST List 30.4406

Tue Nov 19 2019

FYI: Call for Participation: SemEval-2020 Shared Task 5

Editor for this issue: Everett Green <>

Date: 18-Nov-2019
From: Xiaoyu Yang <>
Subject: Call for Participation: SemEval-2020 Shared Task 5
E-mail this message to a friend

Dear all,

Here is a call to participant the SemEval 2020 Task 5: Modelling Causal Reasoning in Language: Detecting Counterfactuals. We invite everyone from interested parties to participant in the shared task.

We just released our training dataset and the submission deadline will be Friday, 31 January 2020. Please note that the test set will be available when the evaluation starts, on 10 January 2020.

Please register and find more details on our CodaLab competition webpage:
We released our dataset here and will evaluate all submissions on CodaLab.

If you have any questions during the competition, please do not hesitate to contact us by email: Or you could also contact all the organizers by emailing to

[ Task Description ]

To model counterfactual semantics and reasoning in natural language, our shared task aims to provide a benchmark for two basic problems.

Subtask1: Detecting counterfactual statements:

In this task, you are asked to determine whether a given statement is counterfactual or not. Counterfactual statements describe events that did not actually happen or cannot happen, as well as the possible consequence if the events have had happened. More specifically, counterfactuals describe events counter to facts and hence naturally involve common sense, knowledge, and reasoning. Tackling this problem is the basis for all down-stream counterfactual related causal inference analysis in natural language. For example, the following statements are counterfactuals that need to be detected: Her post-traumatic stress could have been avoided if a combination of paroxetine and exposure therapy had been prescribed two months earlier.

Subtask2: Detecting antecedent and consequence:

Indicating causal insight is an inherent characteristic of counterfactual. To further detect the causal knowledge conveyed in counterfactual statements, subtask 2 aims to locate antecedent and consequent in counterfactuals. Consider the “post-traumatic stress” example discussed above, the antecedent part is 'if a combination of paroxetine and exposure therapy had been prescribed two months earlier', and the consequent part is 'Her post-traumatic stress could have been avoided'. Such causal relations indicated by counterfactuals can be not only used for analyzing the specific statement but also be accumulated across corpora to develop domain causal knowledge.

[ Important Dates ]
Official website:
Evaluation start*: Jan 10, 2020
Evaluation end*: Jan 31, 2020
Results posted: Feb 5, 2020
System and Task description paper submissions due: Feb 23, 2020
Author notifications: March 29, 2020
Camera-ready submissions due: Apr 6, 2020:
SemEval 2020: Summer 2020

[ Task Organizers ]
Xiaodan Zhu, Queen's University
Xiaoyu Yang, Queen's University
Huasha Zhao, Alibaba Group
Qiong Zhang, Alibaba Group
Stan Matwin, Dalhousie University

Linguistic Field(s): Computational Linguistics

Page Updated: 19-Nov-2019