Review of Perspectives on Multimodality
|
|
|
|
|
Review:
|
Date: Thu, 19 May 2005 22:24:30 +1000
From: Judie Cross <jrkjcross@bigpond.com>
Subject: Perspectives on Multimodality
EDITORS: Ventola, Eija; Charles, Cassily; Kaltenbacher, Martin
TITLE: Perspectives on Multimodality
SERIES: Document Design Companion Series
PUBLISHER: John Benjamins Publishing Company
YEAR: 2004
Judie Cross, Department of Languages, Randwick TAFENSW, Sydney
Institute
SUMMARY
This volume addresses the current need for innovative and scholarly
analyses of multimodal discourse characterising both new and old
media. The collection of papers is a welcome contribution for those in
the research community with an interest in examining how
communication is affected by the interaction of semiotic resources in
texts. The papers are organised into chapters forming two main
sections: the first part explores problematic theoretical issues; the
second part describes a range of increasingly specific applications.
SYNOPSIS
In Part 1, "Multimodal Issues", the first chapter of the book, "In
between modes: Language and image in printed media", Hartmut
Stöckl focuses on modes. He addresses the suggestion by Kress and
van Leeuwen (2001), that all modes are characterised by semiotic
properties, cognitive orientation and semantic potential, but
distinguished according to the way in which they are perceived. He
attempts to identify a common set of semiotic principles, which could
be operating across all modes constituting a hierarchical network.
Chapter 2, "Modelling multiple semiotic systems: The case of gesture
and speech" explores mode as the primary organiser of semiotic
systems and suggests viewing these systems along a continuum
ranging from a proto-language to a fully-fledged one. Peter Muntigl
views all modes (music, dance, gesture) as being modelled on
language and structured by a grammar, which is amenable to analysis
according to Halliday's metafunctions (1994).
Victor Lim Fei in Chapter 3, "Problematising 'semiotic resource'", also
builds on Systemic-Functional Linguistics (SFL) theory and
problematises modes, but concentrates on images, positing icons as
the vocabulary of visual language.
In the last chapter of Part 1, Chapter 4, "Multimodality and empiricism:
Preparing for a corpus-based approach to the study of multimodal
meaning-making", John Bateman, Judy Delin and Renate Henschel
present their innovative multimodal annotated corpus GeM (Genre
and Modality), the aim of which is to investigate and validate the
appropriateness of a multimodal view of genre. They argue that since
most multimodal discourse analysis is informal and interpretative, it is
necessary to gather evidence for the theories proposed. This they
are attempting to do by developing a corpus of annotations. They
outline their procedure for this, providing examples of how tags are
constructed for the variety of visual layers (photos, captions, headers)
in documents.
Part II, "Analyses and Applications", covers a range of multimodal
critiques from different disciplines and contexts. In the first chapter of
this section, "On the effectiveness of mathematics", Kay O'Halloran
demonstrates that visual display in mathematics is crucial as regards
meaning and problem solving. She traces the presentation of
mathematical problems since the early Renaissance and argues that
mathematics has always been a form of multimodal discourse,
although its representation appears to have been less abstract in the
past than it is today. She frames mathematics as a multisemiotic
construction of language, mathematical symbolism and visual display,
which function inter-semiotically as well as intra-semiotically. Her
conception is offered as a way of facilitating the examination of
meaning potential of system choices within and across resources.
In Chapter 6 "Multimodality in language teaching in CD-ROMs", Martin
Kaltenbacher presents evidence demonstrating that the visuals used
in many CDs created for English language teaching are too general or
ambiguous to assist language learning. He analyses a variety of still
and moving images including the feedback provided by the integration
of sound waves, text-picture combinations and the use of short video
clips. He makes the point that too often these visualisations are not
easily interpretable, do not help disambiguate meanings to be learnt
and are not exact representations. His proposal is to replace complex
visuals with icons.
Markus Rheindorf next investigates the relations between modes in
the film "Dirty Dancing". His approach to film is a transdisciplinary
one, where he analyses an ensemble of modal patterns, which he
argues, characterise the film genre and, if recognised, can thereby
deepen our insights.
In Chapter 8, "Multimodal text analysis and subtitling", Christopher
Taylor outlines a methodology for obtaining multimodal transcripts for
film. He demonstrates how meaning created in one modality (such as
word play in text) may be best translated in another (such as visual)
and therefore advocates intersemiotic translation techniques as a
basis for the selection of verbal elements for subtitling.
In the next chapter, "Multimodality in the translation of humour in
comics", Kaindl stresses how the humour in comics is not only created
verbally, but also via the visual or non-verbal dimension of texts.
Kaindl argues that pictorial elements have, however, generally been
disregarded by translators, and that this should no longer continue.
He advocates translators develop methods of analysis, which take into
account the holistic nature of these semiotically complex texts.
In "Multimodality in operation", Chapter 10 by Andrea Hofinger and
Eija Ventola, the design of museum exhibits are described as semiotic
spaces, which not only display objects for viewing, but also involve
dynamic processes of interpretation. A multimodal description, based
on the systemic-functional linguistic model of Halliday (1978), is
applied to a room in the Mozart-Wohnhaus Museum in Salzburg,
focusing specifically on how meaning is constructed by choice of
objects and their display arrangement. The authors demonstrate how
selection and arrangement of meaning resources (such as a family
portrait and an audio-taped recording) interrelate and contribute to or
detract from the overall impact of an exhibit.
"Drawing on theories of inter-semiotic layering to analyse multimodality
in medical self-counselling texts and hypertexts" is the title of Chapter
11. In this chapter Eva Martha Eckkrammer adopts an integrative
approach to text and hypertext, arguing that pictorial and verbal
elements always develop meaning through a process of inter-semiotic
layering. Transposition, juxtaposition, combination and fusion are the
four most frequent forms of layering she identifies via a practical focus
on medical self-counselling texts, raising questions for the
development of multimodal research and text intelligibility.
The final chapter in this collection is "On the multimodality of
interpreting in medical briefings for informed consent". In this chapter
Kristin Bührig analyses a particular type of hospital interaction whose
purpose is to provide a briefing for informed consent. The aim of
using an interpreter in such situations is to facilitate understanding
between a doctor and patient. In the particular briefing investigated,
Bührig demonstrates how the doctor and the untrained interpreter (as
is commonly the situation) used different "linguistic action patterns":
the doctor used a diagram systematically to build on the patient's
knowledge; the daughter-interpreter only used the diagram as a
reference for linking new knowledge. Without skills or preparation in
building-up knowledge by integrating language with other modes, it
seems language barriers can only be partially overcome.
EVALUATION
As stated at the end of the book's introduction, its impetus was
provided by the discussions that occurred during the First
International Symposium on Multimodal Discourse at the University of
Salzburg in 2002. In this respect the volume contributes a wealth of
exploratory, innovative and challenging perspectives to the
development of the theory and practice of multimodal text analysis.
Further, the sequencing of the papers in this volume are organised in
a manner that allows for the gradual cumulation of relevant
knowledge. In spite of the title "Multimodal Perspectives", however, it
is still the case that a systemic-functional linguistic analysis of the
interaction of written text and images predominates, while other
frameworks and modes receive less attention.
Probably the most valuable and challenging aspect of the collection is
the final chapter of the first part, which outlines the enormous task
begun by Bateman et al to build up a corpus of annotations
appropriately tagged so that a body of evidence can be accessed for
supporting interpretations of multimodal texts. While recognising the
value of current multimodal analyses, the generally impressionistic
nature of such analyses is understandably of concern to the authors
of this chapter. In response they are designing multimodal corpora,
drawing on state of the art methods in their endeavour to establish
empirically whether a systematic and regular relationship exists
between different document genres and their potential realisation in
various combinations: verbal text, layout, graphics, pictures and
diagrams. The method by which the GeM model of layering for
classification is outlined and exemplified in considerable detail and it
appears that, in endeavouring to address their concern, the designers
have set themselves long-term and demanding work, not only as
regards scale and detail, but also as regards whether this analytical
approach is appropriate for the kinds of meaning-making which is their
purpose. Nevertheless, this approach is, as acknowledged, simply a
necessary one to be attempted and is, therefore, impressive, of great
interest to the research community and potentially easily accessible.
The three other chapters in this first theoretical section of the book
are more exploratory in nature, further problematising the interaction
of semiotic resources from a Systemic Functional Linguistics (SFL)
perspective, especially with reference to the ground-breaking work of
Kress and van Leeuwen (1996). Muntigl's approach is refreshing as
he attends not only to visual systems, but also to gesture and sign
languages. Acknowledging Kress and van Leeuwen (1996) as well as
the solid work of Kendon (2004), Muntigl's suggestion that semiotic
systems can best be perceived along a continuum is illuminating. Lim
Fei's SFL perspective, however, is slightly more radical in its proposal
that the building blocks for visual images be icons, just as lexical items
are the building-blocks for language. As a means for stimulating
debate, this is an interesting suggestion although, as Lim Fei
acknowledges, there would be considerable scepticism regarding
what constitutes the boundaries of an icon, its recognition,
arrangement and internal grammar.
Kay O'Halloran's chapter on the history and construction of
mathematics, situated at the beginning of section two, is an
outstanding contribution to this book. For those with a minimal
understanding of mathematics, however, this is not an easy read.
Despite this, its value lies in how it still manages to demonstrate
convincingly the means by which mathematics, a multisemiotic
discourse which makes meaning through an interlocking of systems
(symbolism, visual display and language), has been and will continue
to be an effective and valuable resource for describing our physical
experience of the world.
Kaltenbacher's critique of language teaching CD-ROMs makes some
thought-provoking comments as regards the ambiguous nature of
many visuals in these resources, although the sample investigated
may not be representative of what is becoming available globally.
Rheindorf exemplifies a valuable transdisciplinary approach to film, but
stops short of specifying how one's insights can be deepened. The
next two chapters on subtitling and translation, however, exemplify the
value and expansion of meaning created by the extension of linguistic
analysis to encompass other modes. The disciplines of translating and
interpreting can only benefit from adoption of a multimodal approach.
Further, Hofinger and Ventola's analysis of intersemiosis in operation
provides an illustrative application and adaptation of the value of an
SFL approach to multimodal constructs as well as providing a model
for creating dynamic multimodal displays.
Finally, valuable examples of how important understanding of and skill
in the process meaning-making, created by the interaction of semiotic
resources, is found in the medical focus of the last two chapters.
Bührig demonstrates convincingly, by way of contrasting two different
communication styles making use of a diagram, the significance of an
approach, which has a sound systematic basis. This final chapter, like
many of the others, exemplifies the need for continuing investigation
into the ways by which language and other modes, such as the visual
and gestural, involve the development of awareness, fresh
orientations and specialised skills for effective communication.
REFERENCES
Halliday, M. A. K. 1994. An introduction to functional grammar (2 ed.).
London: Edward Arnold.
Kendon, Adam. 2004. Gesture: Visible Action as Utterance.
Cambridge: Cambridge University Press.
Kress, Gunther & van Leeuwen, Theo. 2001. Multimodal Discourse:
The Modes and Media of contemporary Communication. London:
Arnold.
Kress, Gunther & van Leeuwen, Theo. 1996. Reading Images: The
Grammar of Visual Design. London Great Britain: Routledge .
Lemke, J. 1998. Multiplying meaning: Visual and verbal semiotics in
scientific text. In J. R. Martin & R. Veel (Eds.), Reading science:
Critical and functional perspectives on discourses of science (pp. 87-
113). London: Routledge.
O'Toole, Michael. 1994. The Language of Displayed Art. London:
Leicester University Press.
|
| |
ABOUT THE REVIEWER:
ABOUT THE REVIEWER
Judith Leah Cross was awarded her PhD by Macquarie University in
1999 for her thesis 'Textual Realisations', which built on the theories
of Kress and van Leeuwen (1996) as well as O'Toole (1994) in order
to examine how meaning-making is affected when printed children's
image texts are adapted into film, comics, or electronic formats.
Multimodality was the focus for her thesis and continues to be of
increasing relevance to her present work in curriculum design and
blended delivery of English for Academic Purposes.
|
|
|
|
|
|
