https://scholars.lib.ntu.edu.tw/handle/123456789/632062
DC 欄位 | 值 | 語言 |
---|---|---|
dc.contributor.author | Chiang T.-R | en-US |
dc.contributor.author | YUN-NUNG CHEN | en-US |
dc.creator | Chiang T.-R;Chen Y.-N. | - |
dc.date.accessioned | 2023-06-09T07:30:52Z | - |
dc.date.available | 2023-06-09T07:30:52Z | - |
dc.date.issued | 2021 | - |
dc.identifier.uri | https://www.scopus.com/inward/record.uri?eid=2-s2.0-85127262785&partnerID=40&md5=c1ed9ef534f737cab34949245c3469f7 | - |
dc.identifier.uri | https://scholars.lib.ntu.edu.tw/handle/123456789/632062 | - |
dc.description.abstract | This work focuses on relating two mysteries in neural-based text generation: exposure bias, and text degeneration. Despite the long time since exposure bias was mentioned and the numerous studies for its remedy, to our knowledge, its impact on text generation has not yet been verified. Text degeneration is a problem that the widely-used pre-trained language model GPT-2 was recently found to suffer from (Holtzman et al., 2020). Motivated by the unknown causation of the text degeneration, in this paper we attempt to relate these two mysteries. Specifically, we first qualitatively and quantitatively identify mistakes made before text degeneration occurs. Then we investigate the significance of the mistakes by inspecting the hidden states in GPT-2. Our results show that text degeneration is likely to be partly caused by exposure bias. We also study the self-reinforcing mechanism of text degeneration, explaining why the mistakes amplify. In sum, our study provides a more concrete foundation for further investigation on exposure bias and text degeneration problems. © 2021 Association for Computational Linguistics. | - |
dc.relation.ispartof | BlackboxNLP 2021 - Proceedings of the 4th BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP | - |
dc.subject.other | Concrete foundation; Hidden state; Language model; Reinforcing mechanism; Self reinforcing; Text generations | - |
dc.title | Relating Neural Text Degeneration to Exposure Bias | en_US |
dc.type | conference paper | en |
dc.identifier.scopus | 2-s2.0-85127262785 | - |
dc.relation.pages | 228-239 | - |
item.fulltext | no fulltext | - |
item.openairecristype | http://purl.org/coar/resource_type/c_5794 | - |
item.cerifentitytype | Publications | - |
item.openairetype | conference paper | - |
item.grantfulltext | none | - |
crisitem.author.dept | Networking and Multimedia | - |
crisitem.author.dept | Computer Science and Information Engineering | - |
crisitem.author.dept | FinTech Center | - |
crisitem.author.orcid | 0000-0003-1777-3942 | - |
crisitem.author.parentorg | College of Electrical Engineering and Computer Science | - |
crisitem.author.parentorg | College of Electrical Engineering and Computer Science | - |
crisitem.author.parentorg | Others: University-Level Research Centers | - |
顯示於: | 資訊工程學系 |
在 IR 系統中的文件,除了特別指名其著作權條款之外,均受到著作權保護,並且保留所有的權利。