"Kissine offers a new theory of speech acts which is philosophically sophisticated and builds on work in cognitive science, formal semantics, and linguistic typology. This highly readable, brilliant essay is a major contribution to the field."
EDITOR: O'Halloran, Kay L. TITLE: Multimodal Discourse Analysis SUBTITLE: Systemic Functional Perspectives SERIES: Open Linguistics Series PUBLISHER: Continuum YEAR: 2004
Betty Pun, School of Modern Language Studies, Faculty of Arts and Social Sciences, University of New South Wales
SUMMARY
Since the publication of Kress and van Leeuwen's (1990, 1996) exploration of the 'grammar' of visual images, which takes Halliday's (1973, 1978, 2004) systemic-functional theory for language as their underlying analytical approach, as well as O'Toole's (1994) systemic-functional investigation of paintings, sculpture and architecture, there is a growing interest in the field of multimodal studies and the application systemic-functional theory for exploring multimodal discourse. A key emphasis in the field of multimodality is the equal importance which both linguistic and non-linguistic semiotic resources (such as visual images, sound, and others) contribute to communicative purposes. Multimodal Discourse Analysis represents one of the many recent research efforts in further developing the 'meta-language' for multimodal studies.
It is indicated in the introductory chapter of Multimodal Discourse Analysis that the main purpose of the book is to develop the theory and practices for investigating multimodal texts (p. 1). The book consists of research papers contributed by established scholars in the field of multimodal studies, such as Michael O'Toole, Anthony Baldry, and Kay O'Halloran, and researchers from the Semiotics Research Group (SRG) at the National University of Singapore. The theoretical approach of the papers is informed by Hallidayan metafunctional theory to language; and the types of multimodal discourse that are examined range from material objects in three-dimensional space, to electronic media and film, to print-based texts. As some of the examined texts are based in Singaporean context, this edited volume can to a certain extent be seen as a collection of critical investigations on everyday discourses in Singapore (p. 2). In terms of overall organisation, the book is made up of three parts, with each part focussing on one of the three types of multimodal discourse.
Part I focuses on the semiotics of three-dimensional space; and it begins with a metafunctional investigation of the Sydney Opera House by Michael O'Toole. The paper illustrates the various semiotic choices that allow the building to realise its practical or pragmatic function (for example, as a site for theatre and concert performances), and the social relations between the building and its users. It also discusses both the aesthetic and compositional qualities of the Sydney Opera House. In exploring the Sydney Opera House from a metafunctional perspective, the investigation seeks to demonstrate the potential of systemic-functional theory in extending the idea of 'functionalism' as understood and defined within the field of architecture. That is, the qualities of architectural buildings can be conceived in relation to their experiential (i.e. practical or pragmatic), interpersonal (i.e. aesthetic) and textual (i.e. compositional) functions.
Chapter two is a discussion by Alfred Pang; it presents a social semiotic analysis of a museum exhibition at the Singapore History Museum. Drawing on the concept of metafunctions, Martin's (2000) Appraisal Theory (for examining the interpersonal stance inscribed in linguistic texts) and O'Toole's rank-scale approach to visual arts, the analysis aims to illustrate the co-contextualising relations of multiple meaning-making resources (such as linguistic texts, photographs, the way in which the displays are structured, and others) in the history exhibition, and how such relations contribute to the communicative complexities as well as the construction of (dominant) ideologies in the exhibition.
In chapter three, Safeyaton Alias explores the semiotic organisation of space of Singapore's Orchard Road (a key shopping and entertainment precinct) and Marriott Hotel. Similar to Pang, a key motive of Alias' investigation is to examine the way in which prevailing ideologies within the Singaporean society are manifested. As such, Singapore is treated as a text that is both 'three-dimensional and multi-semiotic' (p. 60) in the discussion. Like Pang, the underlying theoretical influence of Alias' discussion is systemic-functional theory's metafunctional perspective of communication and O'Toole's (1994) rank-scale approach. In examining the organisation of spaces in and around Orchard Road and Marriott Hotel, for instance, the way a prayer hall for Muslims and the Singaporean Presidential Palace are located and presented to the public, the manner in which foreign cultural influences (such as cafes which serve coffee and burgers with outdoor seating) are foregrounded whilst local cultures are backgrounded (e.g. hawker food stalls are confined at the basements of buildings), the discussion suggests that both the political ideologies of the Singaporean authorities, as well as the socio-economic demands of the authorities (especially to present an image of sophistication to Singapore), are reflected.
Part II of the book moves its attention to electronic media and film; and it opens with Anthony Baldry's investigation on the role which computer technology may play in exploring film texts. Taking the concepts of phase and transition in phasal analysis (Gregory 1995, 2002; Gregory and Malcolm 1981) and the notions of type and instance (Baldry and Thibault 2001) as points of departure, Baldry suggests that the incorporation of computer technology, such as the Multimodal Corpus Authoring (MCA) System – a concordancing tool which has been designed by Baldry for analysing film texts, offers new potentials for better capturing and illuminating the complex interplays of multiple semiotic resources in time-based multimodal texts (such as film). This, in turn, also provides new possibilities in discussing and understanding multimodal discourse in pedagogic context.
Kay O'Halloran further examines the possibilities which computer technology, for instance, video-editing tool, multimodal concordancing tool, offers in analysis of film texts in chapter five. The discussion concentrates on the temporal and spatial dynamics in the visual semiosis in film texts; and the notions of transition and rank underlie O'Halloran's proposed framework for analysis. Like Baldry, a primary concern of O'Halloran with regard to multimodal analysis of film is the difficulties in capturing the dynamic interactions of various semiotic modalities in the texts. Given the complex interplays of various visual resources in film texts, O'Halloran notes that the points of transition (between different phases) in such time-based multimodal texts are fluid rather than static. O'Halloran further suggests a more dynamic understanding of the idea of rank is needed when exploring film texts.
Chapter six focuses on electronic media. It presents an examination by Arthur Kok on the way meanings are created and produced through processes of intersemiosis, or multisemiotic mediation, in hypertexts. Drawing inspiration from both Halliday's (2004) idea of rank in linguistic analysis and O'Toole's (1994) rank-scale approach for discussing visual arts, Kok proposes four levels of abstraction, namely, Item, Lexia, Cluster, and Web, for analysing hypertexts; and three levels of abstraction of intersemiosis: Relation, Intersection, and Manifestation. The discussion suggests the two lower orders of abstraction (Item and Lexia) are sites where processes of intersemiosis take place.
The third and final section (Part III) of the edited volume centres on the print media. It opens with an investigation by Cheong Yin Yuen on the realisation of ideational meaning in print advertisements. Taking its analytical departure from Hasan's (1996) idea of generic structure potential in relation to linguistic texts, Cheong begins by examining the generic structure of five selected advertisements. Cheong further proposes five strategies (the Bidirectional Investment of meaning; Contextual Propensity; Interpretative Space; Semantic Effervescence; and Visual Metaphor) from which the ideational meaning potential of the advertisements, as well as the semiotic intricacies and nuances of both the linguistic and visual modalities, can be illuminated.
In chapter eight, Libo Guo explores the notion of multimodality in scientific discourses within educational context. In deconstructing its selected texts from a metafunctional perspective as well as a rank-scale approach, the discussion – which also draws on theoretical resources like the concept of semiotic metaphor (O'Halloran 1999a, 1999b, 2003, 2005), the notion of reading path (Kress and van Leeuwen 1990, 1996; Kress 2003) in its analysis – suggests the potential which the metafunctional approach may have in helping students from non-native English speaking backgrounds to better understand university-level scientific discourses.
The last chapter of the book is concerned with the Integrative Multi-Semiotic Model (IMM), an integrative framework which Victor Lim Fei proposes for analysing texts that involve the co-deployment of language and visual images (e.g. picture books). The main project of the proposed model is to develop a 'meta-language' for examining the various semiotic choices in multimodal texts, and for accounting the complexities and nuances in the negotiation and mediation of meanings between the visual and verbal semiotic modalities in the texts. In developing the framework, Lim brings in and integrates theoretical resources from systemic-functional theory and ideas from analytical models of multimodal studies. One of the aspects of the proposed model which Lim gives particular attention to is the Space of Integration (SoI), 'the theoretical platform' for discussing the semiotic interactions of visual-verbal modalities in multimodal texts (p. 238-239). Ideas that are drawn on for illuminating such interactions include semiotic metaphor, Cheong's (this volume) concepts of bidirectional investment of meaning and contextualisation propensity, and others.
EVALUATION
This edited volume presents a collection of research papers that is both diverse and stimulating. The organisation of topics into three types of multimodal discourse allows readers to identify some of the key concerns and questions that are specific to particular types of multimodal discourse. For instance, one of the common concerns which researchers face in exploring electronic media and film is the impact of technology. This can influence the way particular multimodal text is defined (e.g. hypertexts); it can also relate to the potentials which technology may offer to analysis of multimodal discourse (e.g. see papers in Part II). Given the field of multimodality is still in an exploratory stage and the constraints in scope of the papers, some of the issues that are raised in the discussions are yet to be resolved, and remain to be further investigated. Nevertheless, the discussions in this edited volume provide some invaluable and interesting insights of new possibilities and challenges in multimodal studies.
Another aspect that is of interest in this book is the application of systemic-functional theory in the discussions, particularly given the diverse range of questions which the research papers seek to explore. The discussions demonstrate the theoretical potential which the underlying principles of systemic-functional theory, for instance, the notion of choice, the concept of metafuction, can provide for investigations of non-linguistic texts. In this regard, the book represents a key contribution in extending systemic-functional theory and practice to the field of multimodal studies.
Nevertheless, some thorny issues also emerged from the discussions in this book. One of them is the need to introduce and discuss some of the key underlying assumptions of systemic-functional theory, as well as theoretical issues which have been discussed by researchers in the field of multimodal studies (e.g. Baldry 2000; Lemke 1998; Thibault 2000) in the introductory chapter. The book would also benefit from a brief explanation of both Kress and van Leeuwen's and O'Toole's analytical frameworks for multimodal texts. This would not only provide readers who are already familiar with the basic tenets of systemic-functional theory and are interested in multimodal studies an avenue that facilities further engagement with the discussions in the book. It would also help to attract and address new and larger audience.
The use of terminology, for instance, terms such as 'multimodal', 'multisemiotic', 'multisemiosis', 'intersemiosis', and how these terms are understood within the context of multimodal studies is another issue which needs to be further explored. For instance, while some researchers use either multimodal or multisemiotic to describe texts that deploy multiple semiotic resources, others appear to use the two terms interchangeably. In order to further develop the theory of multimodal studies, it is necessary to clarify the scope and boundaries in the way the terms are defined. Having said that, clarifying terminology is a project in progress in many fields of study; and is a phase which many fields of research would have experienced as they continue to further develop the theory. In this regard, despite the problems in the use of terminology, this edited collection nevertheless highlights an aspect in the theory of multimodal studies that needs further attention; and is further developed in recent investigations, such as O'Halloran's (2005: 19-21) clarification of terms within her discussion of mathematical discourse.
A key emphasis that is stressed in the discussions in this collected volume is that meanings created in multimodal texts are due to the complex and intricate interactions of semiosis across multiple meaning-making resources in the texts. And, this is an aspect which the investigations aim to demonstrate. However, some illustrations of such inter-semiotic interactions do not appear to be illuminating. One suspects, perhaps, this could be because language may not be the best semiotic modality to capture and illustrate the nuances and intricacies of intersemiosis – i.e. the co-ordination of meanings across different semiotic systems (Ravelli 2000) – in multimodal texts. Another possible explanation to this problem could be the rigid application of rank-scale approach in the articles. Although a schema of different ranks as well as the semiotic choices in each rank in relation to the texts that are being examined are provided, the papers tend not to explain the rationale and motivation which underpin the use of a rank-scale approach for analysing the selected texts, and illustrate how this approach can help highlight the complex inter-semiotic interactions which the discussions seek to explore. Without illuminating these issues, questions, such as what is the potential of rank-scale approach in illustrating the semiotic complexities of multimodal discourse, remain. This becomes a potential aspect that can frustrate readers who wish to engage with the discussions.
In spite of such limitations, the book nevertheless present some challenging questions in relation to multimodal studies. It provides alternative ways of looking at and thinking about different forms of communication. Although the book may not provide answers to the questions and issues that are raised in the discussions, it offers theoretical resources that allow us to talk and think about different types of communicative texts. In this regard, Multimodal Discourse Analysis is a thought-provoking book for audience who are interested in the field of multimodal studies.
REFERENCES
Baldry, Anthony P. (2000) Introduction. In Baldry, Anthony P. (ed.) Multimodality and Multimediality in the Distance Learning Age. Italy: Palladino Editore, pp. 11-39.
Baldry, Anthony P. and Paul J. Thibault (2001) Towards multimodal corpora. In G. Aston and L. Burnard (eds.) Corpora in the Description and Teaching of English. Bologna: CLUEB, pp. 87-102.
Gregory, Michael (1995) Generic expectancies and discoursal surprises: John Donne's 'The Good Morrow'. In Peter Fries and Michael Gregory (eds.) Discourse in Society: Systemic-Functional Perspectives, Meaning and Choice in Language: Studies for Michael Halliday. Norwood, NJ: Ablex, pp. 67-84.
Gregory, Michael (2002) Phasal analysis within communication linguistics: two contrastive discourses. In P. Fries, M. Cummings, D. Lockwood and W. Sprueill (eds.) Relations and Functions within and around Language. London: Continuum, pp. 316-345.
Gregory, Michael Karen Malcolm (1981) Generic situation and discourse phase: an approach to the analysis of children's talk. Mimeo, Applied Linguistics Research Working Group. Glendon College, York University, Toronto.
Halliday, M.A.K. (1973) Explorations in the Functions of Language. London: Arnold.
Halliday, M.A.K. (1978) Language as a Social Semiotic. London: Edward Arnold.
Halliday, M.A.K. (2004) An Introduction to Functional Grammar. Revised by Christian M.I.M Matthiessen. 3rd edition. London: Arnold.
Hasan, Raquiya (1996) What's going on: a dynamic view of context in language. In C. Cloran, D. Butt and G. Williams (eds.) Ways of Seeing, Ways of Meaning. London: Cassell, pp. 37-50.
Kress, Gunther (2003) Literacy in the New Media Age. London: Routledge.
Kress, Gunther and Theo van Leeuwen (1990) Reading Images. Geelong: Deakin University Press.
Kress, Gunther and Theo van Leeuwen (1996) Reading Images: The Grammar of Visual Design. London & New York: Routledge.
Lemke, Jay L. (1998) Multiplying meaning: visual and verbal semiotics in scientific text. In J.R. Martin and R. Veel (eds.) Reading Science: Critical and Functional Perspectives on Discourses of Science. London: Routledge, pp. 87-113.
Martin, J.R. (2000) Beyond exchange: appraisal systems in English. In S. Hunston and G. Thompson (eds.) Evaluation in Text: Authorial Stance and the Construction of Discourse. Oxford: Oxford University Press, pp. 142-175.
O'Halloran, Kay L. (1999a) Interdependence, interaction and metaphor in multi-semiotic texts. Social Semiotics, 9(3): 317-354.
O'Halloran, Kay L. (1999b) Towards a systemic-functional analysis of multi-semiotic analysis. Semiotica, 124(1/2): 1-29.
O'Halloran, Kay L. (2003) Intersemiosis in mathematics and science: grammatical metaphor and semiotic metaphor. In A.M. Simon-Vandenbergen, M. Taverniers and L. Ravelli (eds.) Grammatical Metaphor: Views from Systemic Functional Linguistics. Amsterdam: John Benjamins, pp. 337-365.
O'Halloran, Kay L. (2005) Mathematical Discourse: Language, Symbolism and Visual Images. London: Continuum.
O'Toole, Michael (1994) The Language of Displayed Art. London: Leicester University Press.
Ravelli, Louise J. (2000) Beyond shopping: constructing the Sydney Olympics in three-dimensional text. Text, 20(4): 489-515.
Thibault, Paul J. (2000) The multimodal transcription of a television advertisement: theory and practice. In Anthony P. Baldry (ed.) Multimodality and Multimediality in the Distance Learning Age. Italy: Palladino Editore, pp. 311-385.
ABOUT THE REVIEWER:
ABOUT THE REVIEWER:
Betty Pun received her Ph.D. in Chinese Studies and Linguistics from the
University of New South Wales, Sydney, Australia in 2006. Her research
interests include social semiotic analysis of visual communication and
Chinese cinema.
