LINGUIST List 25.2394

Mon Jun 02 2014

Diss: Text/Corpus Ling: Hiippala: 'Modelling the Structure of a Multimodal Artefact'

Editor for this issue: Danuta Allen <>

Date: 02-Jun-2014
From: Tuomo Hiippala <>
Subject: Modelling the Structure of a Multimodal Artefact
E-mail this message to a friend

Institution: University of Helsinki
Program: Department of Modern Languages
Dissertation Status: Completed
Degree Date: 2014

Author: Tuomo Hiippala

Dissertation Title: Modelling the Structure of a Multimodal Artefact

Dissertation URL:

Linguistic Field(s): Text/Corpus Linguistics

Dissertation Director:
Eija Ventola

Dissertation Abstract:

This dissertation studied the structure of multimodal artefacts, or how language, image and other semiotic modes combine and interact in documents. This places the study within the emerging field of multimodal research, which uses linguistic methods to study the interaction of multiple semiotic modes. Despite the growing amount of multimodal research, the structure of multimodal artefacts has not received the attention it warrants. Previous studies have been either very detailed or exceedingly abstract, leaving a significant gap between data and theory, which this dissertation attempted to bridge. To do so, the dissertation adopted a data-driven approach to multimodal analysis, addressing the structure of multimodal artefacts, the factors that shape the artefact structure, and the role of structure in the recognition and interpretation of the artifacts. The data consisted of tourist brochures produced by the city of Helsinki between 1967 and 2008, which allowed a longitudinal perspective to their multimodal structure. A total of 58 double-pages were annotated for their content, visual appearance, layout and rhetorical organisation, and compiled into an XML-based multimodal corpus. To study the corpus, the dissertation developed visualization methods that combined information from multiple analytical layers of the corpus to represent the multimodal structures in the data. The study revealed the functional motivation behind the structure of the tourist brochures, identifying patterns in their hierarchical and rhetorical organisation, which were used to fulfil specific communicative tasks. The configuration of these patterns, in turn, signalled how the brochure was to be interpreted. The results also showed that after the year 1985, which marked the introduction of desktop publishing software, the organising principles of the tourist brochures have shifted towards a more fragmented and non-linear structure.

Page Updated: 02-Jun-2014