Publishing Partner: Cambridge University Press CUP Extra Publisher Login

The LINGUIST List is dedicated to providing information on language and language analysis, and to providing the discipline of linguistics with the infrastructure necessary to function in the digital world. LINGUIST is a free resource, run by linguistics students and faculty, and supported primarily by your donations. Please support LINGUIST List during the 2016 Fund Drive.

FYI: Benchmark for Open Relation Extraction


Author: Filipe Mesquita

Linguistic Field(s): Computational Linguistics
Semantics

FYI Body: WE WISH TO ANNOUNCE THE PUBLIC RELEASE OF THE OPEN RELATION EXTRACTION (ORE) BENCHMARK USED FOR THE EXPERIMENTS REPORTED IN THE PAPER: 

EFFECTIVENESS AND EFFICIENCY OF OPEN RELATION EXTRACTION, BY FILIPE MESQUITA, JORDAN SCHMIDEK AND DENILSON BARBOSA, APPEARING AT THE 2013 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING (EMNLP).

ORE IS THE TASK OF RECOGNIZING RELATIONSHIPS BETWEEN TWO OR MORE ENTITIES IN TEXT WITHOUT REQUIRING ANY RELATION-SPECIFIC TRAINING DATA. ORE HAS BECOME PREVALENT OVER TRADITIONAL RELATION EXTRACTION METHODS, ESPECIALLY ON THE WEB, BECAUSE OF THE INTRINSIC DIFFICULTY IN TRAINING INDIVIDUAL EXTRACTORS FOR EVERY SINGLE RELATION.

TO THE BEST OF OUR KNOWLEDGE, OUR BENCHMARK IS THE FIRST OF ITS KIND TO PROVIDE REUSABLE GOLD STANDARD ANNOTATIONS. INCLUDED IN THE BENCHMARK ARE OVER 15,000 ANNOTATIONS (OF WHICH 13,000 WERE DONE AUTOMATICALLY BY MATCHING FACTS IN A KNOWLEDGE BASE), INCLUDING BINARY RELATIONS AND N-ARY RELATIONS. WE ALSO PROVIDE EXTRACTIONS FROM 8 STATE-OF-THE-ART ORE METHODS AND EVALUATION SCRIPTS THAT COMPUTE PRECISION AND RECALL OF A GIVEN SET OF EXTRACTIONS.

FOR MORE INFORMATION AND ACCESS TO THE BENCHMARK ITSELF, PLEASE VISIT THE FOLLOWING URL:

HTTPS://SITES.GOOGLE.COM/A/UALBERTA.CA/SONEX/

--
FILIPE MESQUITA
PH.D. STUDENT
COMPUTING SCIENCE DEPARTMENT
UNIVERSITY OF ALBERTA
HTTP://WEBDOCS.CS.UALBERTA.CA/~MESQUITA/