Situation and behavior understanding by trope detection on films
Journal
The Web Conference 2021 - Proceedings of the World Wide Web Conference, WWW 2021
Pages
3188-3198
Date Issued
2021
Author(s)
Abstract
The human ability of deep cognitive skills is crucial for the development of various real-world applications that process diverse and abundant user generated input. While recent progress of deep learning and natural language processing have enabled learning system to reach human performance on some benchmarks requiring shallow semantics, such human ability still remains challenging for even modern contextual embedding models, as pointed out by many recent studies [9, 10, 22, 24, 32]. Existing machine comprehension datasets assume sentence-level input, lack of casual or motivational inferences, or can be answered with question-answer bias. Here, we present a challenging novel task, trope detection on films, in an effort to create a situation and behavior understanding for machines. Tropes are frequently used storytelling devices for creative works. Comparing to existing movie tag prediction tasks, tropes are more sophisticated as they can vary widely, from a moral concept to a series of circumstances, and embedded with motivations and cause-and-effects. We introduce a new dataset, Tropes in Movie Synopses (TiMoS), with 5623 movie synopses and 95 different tropes collecting from a Wikipedia-style database, TVTropes. We present a multi-stream comprehension network (MulCom) leveraging multi-level attention of words, sentences, and role relations. Experimental result demonstrates that modern models including BERT contextual embedding, movie tag prediction systems, and relational networks, perform at most 37% of human performance (23.97/64.87) in terms of F1 score. Our MulCom outperforms all modern baselines, by 1.5 to 5.0 F1 score and 1.5 to 3.0 mean of average precision (mAP) score. We also provide a detailed analysis and human evaluation to pave ways for future research. ?? 2021 ACM.
Subjects
Benchmarking; Deep learning; Embeddings; Motion pictures; Natural language processing systems; Semantics; World Wide Web; Behavior understanding; Cause and effects; Human evaluation; Human performance; NAtural language processing; Prediction systems; Prediction tasks; Relational network; Learning systems
Other Subjects
Dataset; Natural language processing; Trope detection
Type
conference paper