Multimodal and Digital Reading

The excerpts included below come from the First Monday article, “Digital reading spaces: How expert readers handle books, the Web and electronic paper” by Terje Hillesund.

I think that these quotations apply equally to “graphic novels,” graphic narrative/sequential art and Web comics,  although the article makes no reference to any of these terms–couched as it is in formal academic discourse.

Among researchers studying current changes in reading, semioticians are particularly preoccupied with the materiality of semiotic resources, along a range of media. Since the 1970s and 1980s, desktop publishing and offset printing have dominated composition and printing, making the use of photo and graphic illustrations far less complicated. As a result, today’s newspapers, magazines, textbooks and trade books are often sophisticated publications in which much of the information is given pictorially and by other visual means. Researchers such as Günter Kress (2003) and Theo van Leeuwen (2001; 2006) have described the visual grammar of multimodal texts, suggesting that multimodal reading is not primarily a continuous or discontinuous reading of verbal text, but rather composite reading in which attention jumps back and forth between illustrations and text. Researchers encounter great challenges in trying to explain how meaning is construed in the many kinds of multimodal reading that are emerging, both in print and on screen.

Multimodality, hypertext and the urge to click

Multimodality is not a new phenomenon. Illuminated manuscripts and illustrated books have a long history, as Kress and van Leeuwen (2001; 2006) point out. The use of graphs, diagrams, maps, models, drawings and photographs often increases the informational and aesthetic value of print publications. In addition, a heavily illustrated magazine or textbook offers the user several choices. The reader can look at pictures and the accompanying captions and titles and form a good idea of what the article is about. Parallel to this, the background information and explanations of the main text can be read to get the full story. Either way, due to the salience of pictures and inclinations in our perception, the eyes will jump back and forth between text and illustrations. Direct visual perceptions will complement or replace the mental images usually produced during reading. In a spatial sense, strictly verbal reading will thus be discontinuous. Multimodal reading, on the other hand, will in a temporal meaning go on uninterrupted; the reader will construe visual–verbal meaning units not reducible to any of the two modalities. However, as the use of illustrations increases, a visual logic will eventually take precedence and dominate, as is the case with many modern magazines and text books. In publications of this kind, verbal text plays an auxiliary or reciprocal role, anchoring and contextualising pictures. For readers, the meaning is derived from self–sufficient visual–verbal entities dominated by images, and the process of reading inevitably changes as the reader starts looking and flicking.

Hillesund, Terje. “Digital reading spaces: How expert readers handle books, the Web and electronic paper” First Monday [Online], Volume 15 Number 4 (11 April 2010).

Reproduced under the Creative Commons 3.0 Attribution License.

Advertisements

0 Responses to “Multimodal and Digital Reading”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 52 other followers


%d bloggers like this: