Effective Compression of Language Models by Combining Pruning and Knowledge Distillation
Part Of
Proceedings - 2024 IEEE 48th Annual Computers, Software, and Applications Conference, COMPSAC 2024
Journal Volume
30
Start Page
429
End Page
438
ISBN (of the container)
979-835037696-8
Date Issued
2024-07-02
Author(s)
Abstract
In recent years, Transformer has become an important architecture in language models and has achieved high performance in many natural language processing tasks. However, deploying Transformer architectures efficiently faces challenges because of their large model size and high inference time. Weight pruning is a prominent model compression technique that removes some weights in a model. However, after pruning, Transformer models require repeating the whole training process, including pre-training on a large generalized dataset and fine-tuning on a small downstream dataset, to recover the accuracy. The whole training process takes a long time and requires many computation resources. To address the challenge, this work proposes a pruning method that combines with knowledge distillation to avoid a long re-training time while recovering the accuracy. We use the N:m pruning method that is extended from NVIDIA's 2:4 pruning to compress the Transformer model. After pruning, we retrain the model by leveraging knowledge distillation to make the pruned model learn from the dense model. With this method, the pruned model can achieve comparable accuracy by using only downstream datasets and take much less time than traditional retraining. The experimental results show that DistilBERT in a 1:4 structure can achieve comparable accuracy on the SQuAD v1.1 and v2.0 datasets and a speedups in inference time compared to the original dense model.
Event(s)
48th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2024
Publisher
IEEE
Type
conference paper