Cold-start active learning through self-supervised language modeling
Journal
EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference
Pages
7935-7948
Date Issued
2020
Author(s)
Abstract
Active learning strives to reduce annotation costs by choosing the most critical examples to label. Typically, the active learning strategy is contingent on the classification model. For instance, uncertainty sampling depends on poorly calibrated model confidence scores. In the cold-start setting, active learning is impractical because of model instability and data scarcity. Fortunately, modern NLP provides an additional source of information: pretrained language models. The pre-training loss can find examples that surprise the model and should be labeled for efficient fine-tuning. Therefore, we treat the language modeling loss as a proxy for classification uncertainty. With BERT, we develop a simple strategy based on the masked language modeling loss that minimizes labeling costs for text classification. Compared to other baselines, our approach reaches higher accuracy within less sampling iterations and computation time. © 2020 Association for Computational Linguistics.
Other Subjects
Artificial intelligence; Classification (of information); Computational linguistics; Learning systems; Modeling languages; Natural language processing systems; Uncertainty analysis; Active Learning; Active learning strategies; Calibrated model; Classification models; Cold-start; Confidence score; Data scarcity; Language model; Sources of informations; Uncertainty samplings; Text processing
Type
conference paper