Audio-visual Integration Modifies Emotional Judgment in Music

The conventional view that perceived emotion in music is derived mainly from auditory signals has led to neglect of the contribution of the visual image. In this study, we manipulated mode (major vs. minor) and examined the influence of a video image on the music. Melodies in either major or minor mode were controlled for tempo and rhythm and played to the participants. We found that Taiwanese participants, like Westerners, judged major melodies as expressing positive, and minor melodies negative, emotions (Experiment 1). The major or minor melodies were then paired with video images of the singers, which were either emotionally congruent or incongruent with their modes. Results showed that participants perceived stronger positive or negative emotions with congruent audio-visual stimuli (i.e., the congruency effect, Experiment 2). Compared to listening to music alone, stronger emotions were perceived when an emotionally congruent video image was added (i.e., the enhancement effect) and weaker emotions were perceived when an incongruent image was added (i.e., the attenuation effect) (Experiment 3). These results can be explained by the SAME model of Overy and Molnar-Szakacs (2009).