Kao W.-THUNG-YI LEE2023-06-092023-06-092021https://www.scopus.com/inward/record.uri?eid=2-s2.0-85126931361&partnerID=40&md5=2c39cfe031a8384cbc790a4dcf8fd343https://scholars.lib.ntu.edu.tw/handle/123456789/632481This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications. To verify pre-trained models' transferability, we test the pre-trained models on text classification tasks with meanings of tokens mismatches, and realworld non-text token sequence classification data, including amino acid, DNA, and music. We find that even on non-text data, the models pre-trained on text converge faster, perform better than the randomly initialized models, and only slightly worse than the models using task-specific knowledge. We also find that the representations of the text and non-text pretrained models share non-trivial similarities. © 2021 Association for Computational Linguistics.Classification (of information); Computational linguistics; Amino-acids; Cross-disciplinary; Model transferabilities; Non-trivial; Power; Real-world; Sequence classification; Specific knowledge; Text data; Token sequences; Text processingconference paper2-s2.0-85126931361