Frank Serafini - Beyond the Visual: An Introduction to Researching Multimodal Phenomena

Theo van Leeuwen

Lindsey Moses

Stephanie F. Reid, Danielle Kachorsky, and Kathryn P. Chapman

Peggy Albers

Frank Serafini

Evelyn Arizpe and Julie E. McAdam

Emilia Djonov

Jeffrey B. Holmes, Earl Aguilera, and Kelly M. Tran

Len Unsworth

Jennifer Rowsell

Marva Cappello

Katie Bernstein

Fei Victor Lim

Elisabetta Adami

Kathy A. Mills and Lesley Friend

Kate Cowan and John Potter

Louise Ravelli

Multimodality is a key feature of contemporary communication, so it is not surprising that it has moved center stage in many fields. Just as language is not only of interest to linguists, but also to psychologists, sociologists, literary scholars, philosophers, and more, so multimodality is not an exclusive specialty, but of interest to linguists, discourse analysts, rhetoricians, scholars of media and cultural studies, semioticians, and many others. This makes it a complex but also interesting and rich field that offers multiple vantage points that can all contribute to our understanding of multimodal communication.

In this book Serafini sets out to map that complex terrain, surveying a wide range of forms of multimodal inquiry and a vast array of analytical frameworks and methodological choicesmultimodal framing analysis, multimodal genre analysis, multimodal cultural analysis, digitally based multimodal analysis, and critical multimodal analysis, to name just a few of the approaches he reviews. A series of vignettes then shows how well-known researchers use these approaches to investigate multimodal texts, artefacts, spaces, and events, including wine labels, childrens picture books, animation software, video games, and much more.

But Serafini also clearly brings out what all these diverse approaches have in commonthe quest to understand how these different texts and artefacts and events and spaces use semiotic resources to make meaning, and how multimodal analysis, if well contextualized, can yield understandings of social life in a wide range of areas, including education, health care, entertainment, and more. And he also shows that these different approaches, when looked at as practices, as the things multimodal researchers do , have much in common.

There is of course a risk in focusing on multimodal analysis as a practice in this way, the risk of foregrounding analytical procedure over content and rigor over insight, as if all we need to do research is choose an object of interest and a toolbox of choice. This is a risk, because the more explicit and detailed our procedures, the more they can be automatized, and the more they are automatized, the more likely they are to stagnate.

But toward the end of his journey through the vast terrain of multimodal research, Serafini comes to a number of conclusions that counter this risk. Developing a multimodal research project is not a matter of choosing a particular framework from a menu of choices. The choices that exist should, instead, be treated as resources that can be combined and adapted to suit the needs of particular projects in ways that are always to some degree new. Conducting high-quality research requires making connections and alignments among theories, research designs, and analytical procedures (p. 209), he says and we have to rely less on allegiances to established ways of thinking grounded in traditional academic domains and embrace new ways of approaching, conceptualizing, and analyzing contemporary forms, as well as firmly ground our analytical approaches upon a more substantial and expanded theoretical terrain (p. 209).

Closely related to that, he offers another very important piece of advice: Analytical frameworks are also always frames that illuminate certain aspects of a phenomenon and send other aspects to the periphery (p. 210). It is therefore important to begin with an open-minded form of observation that is not yet filtered through a specific analytical framework, to begin with some literal description of the contents being examined before advancing into metaphorical or connotative descriptions and assumptions (p. 113).

All this makes the book more than a taxonomy of fields and subfields, more than a how to textbook providing ready-made recipes. Instead, it opens up the field and shows the next generation of multimodal researchers how much there already is, but also how much there is still to discover, and how much scope for innovation and creativity the field still offers.