* * * * * * * * * * * * * * * * * * * * * * * *
LINGUIST List logo Eastern Michigan University Wayne State University *
* People & Organizations * Jobs * Calls & Conferences * Publications * Language Resources * Text & Computer Tools * Teaching & Learning * Mailing Lists * Search *
* *
LINGUIST List 22.4

Tue Jan 04 2011

Diss: Comp Ling/Psycholing: McCrae: 'A Computational Model for the ...'

Editor for this issue: Mfon Udoinyang <mfonlinguistlist.org>

To post to LINGUIST, use our convenient web form at http://linguistlist.org/LL/posttolinguist.cfm.
        1.     Patrick McCrae , A Computational Model for the Influence of Cross-Modal Context upon Syntactic Parsing

Message 1: A Computational Model for the Influence of Cross-Modal Context upon Syntactic Parsing
Date: 29-Dec-2010
From: Patrick McCrae <patrick.mccraeinformatik.uni-hamburg.de>
Subject: A Computational Model for the Influence of Cross-Modal Context upon Syntactic Parsing
E-mail this message to a friend

Institution: Hamburg University
Program: Department of Informatics
Dissertation Status: Completed
Degree Date: 2010

Author: Patrick McCrae

Dissertation Title: A Computational Model for the Influence of Cross-Modal
Context upon Syntactic Parsing

Dissertation URL: http://www.sub.uni-hamburg.de/opus/volltexte/2010/4800/index.html

Linguistic Field(s): Computational Linguistics

Dissertation Director(s):
Wolfgang Menzel
Maosong Sun
Christopher Habel

Dissertation Abstract:

The focus of this thesis is to motivate, specify and validate a computational
model for the cross-modal influence of visual scene context upon natural
language understanding and the process of syntactic parsing, in particular. We
argue for a computational model that establishes cross-modal referential links
between words in the linguistic input and entities in a visual scene context.
Cross-modal referential links are assigned on the basis of conceptual
compatibility between the concepts activated in the linguistic modality and the
concepts instantiated in visual context. The proposed model utilises the
thematic relations in the visual scene context to modulate attachments in the
linguistic analysis.

We base our model architecture on the cognitive framework of Conceptual
Semantics by Ray Jackendoff. In our model, we adopt the central tenet of
Conceptual Semantics that all cross-modal interactions of non-linguistic
modalities with language are mediated by Conceptual Structure, a single, uniform
representation of linguistic and non-linguistic semantics. Conceptual Structure
propagates the influence of the non-linguistic modalities into syntactic
representation via a syntax-semantics interface. The purpose of this interface
is to map between the syntactic and the semantic representation by means of
representational correspondence rules.

Our model implements central aspects of the cognitive architecture in Conceptual
Semantics. We encode the semantic information for all entities, be they
linguistic or non-linguistic in nature, on a single level of semantic
representation. In particular, the semantic part of linguistic analysis and
visual scene information are included in this representation. The semantic
preferences arising from visual context constrain the semantic part of
linguistic analysis. The semantic part of linguistic analysis, in turn,
constrains syntactic analysis via the syntax-semantics interface. In this way,
our model achieves a semantically mediated propagation of non-linguistic visual
scene information into syntactic representation.

In Part III of the thesis we validate our model's context integration behaviour
under a range of experimental conditions. The integration of visual scene
context as a hard constraint on linguistic analysis enforces an absolute
dominance of visual context information over linguistic analysis. As a result,
hard integration may afford a contextualised linguistic analysis that violates
linguistic well-formedness preferences in order to be semantically compatible
with the modelled visual context. Integrating visual context information as a
soft constraint on linguistic analysis permits a balance between conflicting
linguistic and contextual references based on the strength of the individual
preferences. Under soft integration, our model also diagnoses which aspects of
linguistic analysis are in conflict with visual context information. Diagnosis
constitutes an important cognitive capability of natural systems in the situated

We further demonstrate our model's robustness to conceptual underspecification
in the contextual representation. Our experiments show that the integration of
conceptually underspecified context representations provides sufficient
information to support the process of syntactic disambiguation. The capability
of processing conceptually underspecified semantic information is a relevant
capability of natural systems when dealing with perceptual uncertainty and
perceptual ambiguity.
Read more issues|LINGUIST home page|Top of issue

Page Updated: 04-Jan-2011

Supported in part by the National Science Foundation       About LINGUIST    |   Contact Us       ILIT Logo
While the LINGUIST List makes every effort to ensure the linguistic relevance of sites listed on its pages, it cannot vouch for their contents.