The Encounter among Multimodality, Metaphor, and Interpretation: Evidence from Political Cartoons and Poetry Music
Date Issued
2015
Date
2015
Author(s)
Lin, Tiffany Ying-Yu
Abstract
In the past few decades, studies on how verbal metaphors shape our thoughts have been widely discussed in the filed of cognitive science, cognitive linguistics, psychology, mathematics, art, and philosophy, etc. Under this view, “non-verbal” metaphors have also received considerable attention, while previous studies mainly focused on the mental processing and one-on-one correspondence between language and other mode, especially the pictorial mode. However, this study believes the relations between multimodalities can be viewed as a complex continuum of various kinds of combinations. Therefore, this study proposes an alternative interactive interpretation model, the Multimodal Fusion Model, to account for the one-on-many relationship between language and multimodality (visual and aural modes) and to explore how certain topic/issue has been represented in various forms in distinct genre. Based on the evidence from the multimodal genre of political cartoons and poetry music, we examined (1) how the hotly debated topic, the U.S. beef import issue, has been represented in various political cartoons in different newspapers and (2) how a classic poem (i.e. Serendipity) has been conceptualized in four versions of songs. We further incorporate the audience response analysis from the interview surveys to examine how the audience actually interprets the multimodal genre. Our findings also show how prominent multimodal features and multimodal cues (visual/aural/verbal cues) help facilitate the interpretation process of multimodal metaphor in specific genre. This study constructs a multimodal corpus involving 56 political cartoons with regard to U.S. beef import issues as reported in two dominant Taiwanese newspapers, the Liberty Times and United Daily News. Our corpus shows that multimodal fusion combining the visual mode, verbal mode, and the conceptual level evolves from two metonymic-metaphoric networks, i.e., related metonymic network and diversified metaphoric network. The result demonstrates that multimodal fusion is a significant and recurrent representation technique in the genre of political cartoon and has the cognitive function of encapsulating the abstract complex political debates efficiently with irony and humorous effect. In addition, this study emphasizes the important role of metonymy and demonstrates how metonymies and metaphors are interwoven in the process of multimodal fusion, which underlies the metaphorical mappings of conceptual scenarios related to “POLITICS IS GAME” and “POLITICS IS WAR”. Although the critical messages and distinct political stances in two newspapers both emerged through multimodal fusion, they are highlighted and contrasted through prominent visual features and verbal context in political cartoons. Based on the evidence from four versions of poetry music composed for the poem “Serendipity” by Hsu Chih-Mo, this study provides a metaphor-based musical analysis under the scope of multimodal fusion model. We discovered the “chain of metaphors” as an interactive and echoing relationship between metaphors that creates a holistic representation of multimodal metaphor combining the verbal and aural mode in the genre of poetry music. Furthermore, the results show the metonymic representations of metaphor in poetry music have been realized through the “pictorial effect” of partial visualization of cloud image on the score representing CLOUD metaphor and aural simulation of wave rhythm representing SEA metaphor, etc. Through cross-comparison analysis, we propose two major aspects of metaphor, the “concrete image” and “abstract metaphorical extension”, have been highlighted differently through musical techniques and features such as dynamics, pitch, and timbre, etc. in four versions of songs. Moreover, this study provides a concrete-abstract continuum as a categorization of composing patterns and musical effects. Viewing from the embodied perspective, our findings also show that the concrete relationship of space and abstract attributes of emotion can be mapped onto the representations of prominent musical features in poetry music. This study further incorporates the quantitative and qualitative analysis from audience response survey. The results show that higher level of audience interpretation in political cartoons relies on the effective visual recognition of caricatures, comprehension of verbal and contextual information, and interactive connection of the verbal mode, visual mode, and conceptual level. On the other hand, the higher level of audience interpretation in poetry music is related to the distinction between the concrete image and abstract metaphorical extension. The results also suggest that the audience’s preference is relevant to the correspondence between the verbal and aural mode, which implicitly reflects how the audience “feels” about the musical effect and aesthetic functions created from multimodal fusion in different versions of songs. From the perspectives of researcher and audience, this study investigates how verbal mode, visual mode, aural mode, and conceptual level are interacted, integrated, and interpreted through multimodal fusion. We hope to shed light on the interdisciplinary studies on multimodality and the interface between the cognitive mechanism and pragmatic multimodal use of language, picture, music, and the aesthetic experience of the multimodality appreciation process.
Subjects
Multimodal Fusion Model
multimodal metaphor/metonymy
audience response
multimodal cues
multimodal genre
Type
thesis
File(s)![Thumbnail Image]()
Loading...
Name
ntu-104-D98142002-1.pdf
Size
23.54 KB
Format
Adobe PDF
Checksum
(MD5):1ce7da0104f01bdedeb8789f7696604d