Date: 28-Sep-2006
From: Betty Pun <bpunokyahoo.com>
Subject: Multimodal Discourse Analysis
Announced at http://linguistlist.org/issues/17/17-1139.html
EDITOR: O'Halloran, Kay L.TITLE: Multimodal Discourse AnalysisSUBTITLE: Systemic Functional PerspectivesSERIES: Open Linguistics SeriesPUBLISHER: ContinuumYEAR: 2004
Betty Pun, School of Modern Language Studies, Faculty of Arts and SocialSciences, University of New South Wales
SUMMARY
Since the publication of Kress and van Leeuwen's (1990, 1996) explorationof the 'grammar' of visual images, which takes Halliday's (1973, 1978,2004) systemic-functional theory for language as their underlyinganalytical approach, as well as O'Toole's (1994) systemic-functionalinvestigation of paintings, sculpture and architecture, there is a growinginterest in the field of multimodal studies and the applicationsystemic-functional theory for exploring multimodal discourse. A keyemphasis in the field of multimodality is the equal importance which bothlinguistic and non-linguistic semiotic resources (such as visual images,sound, and others) contribute to communicative purposes. MultimodalDiscourse Analysis represents one of the many recent research efforts infurther developing the 'meta-language' for multimodal studies.
It is indicated in the introductory chapter of Multimodal DiscourseAnalysis that the main purpose of the book is to develop the theory andpractices for investigating multimodal texts (p. 1). The book consists ofresearch papers contributed by established scholars in the field ofmultimodal studies, such as Michael O'Toole, Anthony Baldry, and KayO'Halloran, and researchers from the Semiotics Research Group (SRG) at theNational University of Singapore. The theoretical approach of the papers isinformed by Hallidayan metafunctional theory to language; and the types ofmultimodal discourse that are examined range from material objects inthree-dimensional space, to electronic media and film, to print-basedtexts. As some of the examined texts are based in Singaporean context, thisedited volume can to a certain extent be seen as a collection of criticalinvestigations on everyday discourses in Singapore (p. 2). In terms ofoverall organisation, the book is made up of three parts, with each partfocussing on one of the three types of multimodal discourse.
Part I focuses on the semiotics of three-dimensional space; and it beginswith a metafunctional investigation of the Sydney Opera House by MichaelO'Toole. The paper illustrates the various semiotic choices that allow thebuilding to realise its practical or pragmatic function (for example, as asite for theatre and concert performances), and the social relationsbetween the building and its users. It also discusses both the aestheticand compositional qualities of the Sydney Opera House. In exploring theSydney Opera House from a metafunctional perspective, the investigationseeks to demonstrate the potential of systemic-functional theory inextending the idea of 'functionalism' as understood and defined within thefield of architecture. That is, the qualities of architectural buildingscan be conceived in relation to their experiential (i.e. practical orpragmatic), interpersonal (i.e. aesthetic) and textual (i.e. compositional)functions.
Chapter two is a discussion by Alfred Pang; it presents a social semioticanalysis of a museum exhibition at the Singapore History Museum. Drawing onthe concept of metafunctions, Martin's (2000) Appraisal Theory (forexamining the interpersonal stance inscribed in linguistic texts) andO'Toole's rank-scale approach to visual arts, the analysis aims toillustrate the co-contextualising relations of multiple meaning-makingresources (such as linguistic texts, photographs, the way in which thedisplays are structured, and others) in the history exhibition, and howsuch relations contribute to the communicative complexities as well as theconstruction of (dominant) ideologies in the exhibition.
In chapter three, Safeyaton Alias explores the semiotic organisation ofspace of Singapore's Orchard Road (a key shopping and entertainmentprecinct) and Marriott Hotel. Similar to Pang, a key motive of Alias'investigation is to examine the way in which prevailing ideologies withinthe Singaporean society are manifested. As such, Singapore is treated as atext that is both 'three-dimensional and multi-semiotic' (p. 60) in thediscussion. Like Pang, the underlying theoretical influence of Alias'discussion is systemic-functional theory's metafunctional perspective ofcommunication and O'Toole's (1994) rank-scale approach. In examining theorganisation of spaces in and around Orchard Road and Marriott Hotel, forinstance, the way a prayer hall for Muslims and the SingaporeanPresidential Palace are located and presented to the public, the manner inwhich foreign cultural influences (such as cafes which serve coffee andburgers with outdoor seating) are foregrounded whilst local cultures arebackgrounded (e.g. hawker food stalls are confined at the basements ofbuildings), the discussion suggests that both the political ideologies ofthe Singaporean authorities, as well as the socio-economic demands of theauthorities (especially to present an image of sophistication toSingapore), are reflected.
Part II of the book moves its attention to electronic media and film; andit opens with Anthony Baldry's investigation on the role which computertechnology may play in exploring film texts. Taking the concepts of phaseand transition in phasal analysis (Gregory 1995, 2002; Gregory and Malcolm1981) and the notions of type and instance (Baldry and Thibault 2001) aspoints of departure, Baldry suggests that the incorporation of computertechnology, such as the Multimodal Corpus Authoring (MCA) System – aconcordancing tool which has been designed by Baldry for analysing filmtexts, offers new potentials for better capturing and illuminating thecomplex interplays of multiple semiotic resources in time-based multimodaltexts (such as film). This, in turn, also provides new possibilities indiscussing and understanding multimodal discourse in pedagogic context.
Kay O'Halloran further examines the possibilities which computertechnology, for instance, video-editing tool, multimodal concordancingtool, offers in analysis of film texts in chapter five. The discussionconcentrates on the temporal and spatial dynamics in the visual semiosis infilm texts; and the notions of transition and rank underlie O'Halloran'sproposed framework for analysis. Like Baldry, a primary concern ofO'Halloran with regard to multimodal analysis of film is the difficultiesin capturing the dynamic interactions of various semiotic modalities in thetexts. Given the complex interplays of various visual resources in filmtexts, O'Halloran notes that the points of transition (between differentphases) in such time-based multimodal texts are fluid rather than static.O'Halloran further suggests a more dynamic understanding of the idea ofrank is needed when exploring film texts.
Chapter six focuses on electronic media. It presents an examination byArthur Kok on the way meanings are created and produced through processesof intersemiosis, or multisemiotic mediation, in hypertexts. Drawinginspiration from both Halliday's (2004) idea of rank in linguistic analysisand O'Toole's (1994) rank-scale approach for discussing visual arts, Kokproposes four levels of abstraction, namely, Item, Lexia, Cluster, and Web,for analysing hypertexts; and three levels of abstraction of intersemiosis:Relation, Intersection, and Manifestation. The discussion suggests the twolower orders of abstraction (Item and Lexia) are sites where processes ofintersemiosis take place.
The third and final section (Part III) of the edited volume centres on theprint media. It opens with an investigation by Cheong Yin Yuen on therealisation of ideational meaning in print advertisements. Taking itsanalytical departure from Hasan's (1996) idea of generic structurepotential in relation to linguistic texts, Cheong begins by examining thegeneric structure of five selected advertisements. Cheong further proposesfive strategies (the Bidirectional Investment of meaning; ContextualPropensity; Interpretative Space; Semantic Effervescence; and VisualMetaphor) from which the ideational meaning potential of theadvertisements, as well as the semiotic intricacies and nuances of both thelinguistic and visual modalities, can be illuminated.
In chapter eight, Libo Guo explores the notion of multimodality inscientific discourses within educational context. In deconstructing itsselected texts from a metafunctional perspective as well as a rank-scaleapproach, the discussion – which also draws on theoretical resources likethe concept of semiotic metaphor (O'Halloran 1999a, 1999b, 2003, 2005), thenotion of reading path (Kress and van Leeuwen 1990, 1996; Kress 2003) inits analysis – suggests the potential which the metafunctional approach mayhave in helping students from non-native English speaking backgrounds tobetter understand university-level scientific discourses.
The last chapter of the book is concerned with the IntegrativeMulti-Semiotic Model (IMM), an integrative framework which Victor Lim Feiproposes for analysing texts that involve the co-deployment of language andvisual images (e.g. picture books). The main project of the proposed modelis to develop a 'meta-language' for examining the various semiotic choicesin multimodal texts, and for accounting the complexities and nuances in thenegotiation and mediation of meanings between the visual and verbalsemiotic modalities in the texts. In developing the framework, Lim bringsin and integrates theoretical resources from systemic-functional theory andideas from analytical models of multimodal studies. One of the aspects ofthe proposed model which Lim gives particular attention to is the Space ofIntegration (SoI), 'the theoretical platform' for discussing the semioticinteractions of visual-verbal modalities in multimodal texts (p. 238-239).Ideas that are drawn on for illuminating such interactions include semioticmetaphor, Cheong's (this volume) concepts of bidirectional investment ofmeaning and contextualisation propensity, and others.
EVALUATION
This edited volume presents a collection of research papers that is bothdiverse and stimulating. The organisation of topics into three types ofmultimodal discourse allows readers to identify some of the key concernsand questions that are specific to particular types of multimodaldiscourse. For instance, one of the common concerns which researchers facein exploring electronic media and film is the impact of technology. Thiscan influence the way particular multimodal text is defined (e.g.hypertexts); it can also relate to the potentials which technology mayoffer to analysis of multimodal discourse (e.g. see papers in Part II).Given the field of multimodality is still in an exploratory stage and theconstraints in scope of the papers, some of the issues that are raised inthe discussions are yet to be resolved, and remain to be furtherinvestigated. Nevertheless, the discussions in this edited volume providesome invaluable and interesting insights of new possibilities andchallenges in multimodal studies.
Another aspect that is of interest in this book is the application ofsystemic-functional theory in the discussions, particularly given thediverse range of questions which the research papers seek to explore. Thediscussions demonstrate the theoretical potential which the underlyingprinciples of systemic-functional theory, for instance, the notion ofchoice, the concept of metafuction, can provide for investigations ofnon-linguistic texts. In this regard, the book represents a keycontribution in extending systemic-functional theory and practice to thefield of multimodal studies.
Nevertheless, some thorny issues also emerged from the discussions in thisbook. One of them is the need to introduce and discuss some of the keyunderlying assumptions of systemic-functional theory, as well astheoretical issues which have been discussed by researchers in the field ofmultimodal studies (e.g. Baldry 2000; Lemke 1998; Thibault 2000) in theintroductory chapter. The book would also benefit from a brief explanationof both Kress and van Leeuwen's and O'Toole's analytical frameworks formultimodal texts. This would not only provide readers who are alreadyfamiliar with the basic tenets of systemic-functional theory and areinterested in multimodal studies an avenue that facilities furtherengagement with the discussions in the book. It would also help to attractand address new and larger audience.
The use of terminology, for instance, terms such as 'multimodal','multisemiotic', 'multisemiosis', 'intersemiosis', and how these terms areunderstood within the context of multimodal studies is another issue whichneeds to be further explored. For instance, while some researchers useeither multimodal or multisemiotic to describe texts that deploy multiplesemiotic resources, others appear to use the two terms interchangeably. Inorder to further develop the theory of multimodal studies, it is necessaryto clarify the scope and boundaries in the way the terms are defined.Having said that, clarifying terminology is a project in progress in manyfields of study; and is a phase which many fields of research would haveexperienced as they continue to further develop the theory. In this regard,despite the problems in the use of terminology, this edited collectionnevertheless highlights an aspect in the theory of multimodal studies thatneeds further attention; and is further developed in recent investigations,such as O'Halloran's (2005: 19-21) clarification of terms within herdiscussion of mathematical discourse.
A key emphasis that is stressed in the discussions in this collected volumeis that meanings created in multimodal texts are due to the complex andintricate interactions of semiosis across multiple meaning-making resourcesin the texts. And, this is an aspect which the investigations aim todemonstrate. However, some illustrations of such inter-semioticinteractions do not appear to be illuminating. One suspects, perhaps, thiscould be because language may not be the best semiotic modality to captureand illustrate the nuances and intricacies of intersemiosis – i.e. theco-ordination of meanings across different semiotic systems (Ravelli 2000)– in multimodal texts. Another possible explanation to this problem couldbe the rigid application of rank-scale approach in the articles. Although aschema of different ranks as well as the semiotic choices in each rank inrelation to the texts that are being examined are provided, the papers tendnot to explain the rationale and motivation which underpin the use of arank-scale approach for analysing the selected texts, and illustrate howthis approach can help highlight the complex inter-semiotic interactionswhich the discussions seek to explore. Without illuminating these issues,questions, such as what is the potential of rank-scale approach inillustrating the semiotic complexities of multimodal discourse, remain.This becomes a potential aspect that can frustrate readers who wish toengage with the discussions.
In spite of such limitations, the book nevertheless present somechallenging questions in relation to multimodal studies. It providesalternative ways of looking at and thinking about different forms ofcommunication. Although the book may not provide answers to the questionsand issues that are raised in the discussions, it offers theoreticalresources that allow us to talk and think about different types ofcommunicative texts. In this regard, Multimodal Discourse Analysis is athought-provoking book for audience who are interested in the field ofmultimodal studies.
REFERENCES
Baldry, Anthony P. (2000) Introduction. In Baldry, Anthony P. (ed.)Multimodality and Multimediality in the Distance Learning Age. Italy:Palladino Editore, pp. 11-39.
Baldry, Anthony P. and Paul J. Thibault (2001) Towards multimodal corpora.In G. Aston and L. Burnard (eds.) Corpora in the Description and Teachingof English. Bologna: CLUEB, pp. 87-102.
Gregory, Michael (1995) Generic expectancies and discoursal surprises: JohnDonne's 'The Good Morrow'. In Peter Fries and Michael Gregory (eds.)Discourse in Society: Systemic-Functional Perspectives, Meaning and Choicein Language: Studies for Michael Halliday. Norwood, NJ: Ablex, pp. 67-84.
Gregory, Michael (2002) Phasal analysis within communication linguistics:two contrastive discourses. In P. Fries, M. Cummings, D. Lockwood and W.Sprueill (eds.) Relations and Functions within and around Language. London:Continuum, pp. 316-345.
Gregory, Michael Karen Malcolm (1981) Generic situation and discoursephase: an approach to the analysis of children's talk. Mimeo, AppliedLinguistics Research Working Group. Glendon College, York University, Toronto.
Halliday, M.A.K. (1973) Explorations in the Functions of Language. London:Arnold.
Halliday, M.A.K. (1978) Language as a Social Semiotic. London: Edward Arnold.
Halliday, M.A.K. (2004) An Introduction to Functional Grammar. Revised byChristian M.I.M Matthiessen. 3rd edition. London: Arnold.
Hasan, Raquiya (1996) What's going on: a dynamic view of context inlanguage. In C. Cloran, D. Butt and G. Williams (eds.) Ways of Seeing, Waysof Meaning. London: Cassell, pp. 37-50.
Kress, Gunther (2003) Literacy in the New Media Age. London: Routledge.
Kress, Gunther and Theo van Leeuwen (1990) Reading Images. Geelong: DeakinUniversity Press.
Kress, Gunther and Theo van Leeuwen (1996) Reading Images: The Grammar ofVisual Design. London & New York: Routledge.
Lemke, Jay L. (1998) Multiplying meaning: visual and verbal semiotics inscientific text. In J.R. Martin and R. Veel (eds.) Reading Science:Critical and Functional Perspectives on Discourses of Science. London:Routledge, pp. 87-113.
Martin, J.R. (2000) Beyond exchange: appraisal systems in English. In S.Hunston and G. Thompson (eds.) Evaluation in Text: Authorial Stance and theConstruction of Discourse. Oxford: Oxford University Press, pp. 142-175.
O'Halloran, Kay L. (1999a) Interdependence, interaction and metaphor inmulti-semiotic texts. Social Semiotics, 9(3): 317-354.
O'Halloran, Kay L. (1999b) Towards a systemic-functional analysis ofmulti-semiotic analysis. Semiotica, 124(1/2): 1-29.
O'Halloran, Kay L. (2003) Intersemiosis in mathematics and science:grammatical metaphor and semiotic metaphor. In A.M. Simon-Vandenbergen, M.Taverniers and L. Ravelli (eds.) Grammatical Metaphor: Views from SystemicFunctional Linguistics. Amsterdam: John Benjamins, pp. 337-365.
O'Halloran, Kay L. (2005) Mathematical Discourse: Language, Symbolism andVisual Images. London: Continuum.
O'Toole, Michael (1994) The Language of Displayed Art. London: LeicesterUniversity Press.
Ravelli, Louise J. (2000) Beyond shopping: constructing the Sydney Olympicsin three-dimensional text. Text, 20(4): 489-515.
Thibault, Paul J. (2000) The multimodal transcription of a televisionadvertisement: theory and practice. In Anthony P. Baldry (ed.)Multimodality and Multimediality in the Distance Learning Age. Italy:Palladino Editore, pp. 311-385.
ABOUT THE REVIEWER:
Betty Pun received her Ph.D. in Chinese Studies and Linguistics from theUniversity of New South Wales, Sydney, Australia in 2006. Her researchinterests include social semiotic analysis of visual communication andChinese cinema.
|