https://scholars.lib.ntu.edu.tw/handle/123456789/581463
標題: | Fully nested neural network for adaptive compression and quantization | 作者: | Cui Y Liu Z Yao W Li Q Chan A.B Xue C.J. TEI-WEI KUO |
公開日期: | 2020 | 卷: | 2021-January | 起(迄)頁: | 2080-2087 | 來源出版物: | IJCAI International Joint Conference on Artificial Intelligence | 摘要: | Neural network compression and quantization are important tasks for fitting state-of-the-art models into the computational, memory and power constraints of mobile devices and embedded hardware. Recent approaches to model compression/quantization are based on reinforcement learning or search methods to compress/quantize the neural network for a specific hardware platform. However, these methods require multiple runs to compress/quantize the same base neural network to different hardware setups. In this work, we propose a fully nested neural network (FN3) that runs only once to build a nested set of compressed/quantized models, which is optimal for different resource constraints. Specifically, we exploit the additive characteristic in different levels of building blocks in neural network and propose an ordered dropout (ODO) operation that ranks the building blocks. Given a trained FN3, a fast heuristic search algorithm is run offline to find the optimal removal of components to maximize the accuracy under different constraints. Compared with the related works on adaptive neural network designed only for channels or bits, the proposed approach is unified for different levels of building blocks (bits, neurons, channels, residual paths and layers). Empirical results validate strong practical performance of the proposed approach. ? 2020 Inst. Sci. inf., Univ. Defence in Belgrade. All rights reserved. |
URI: | https://www.scopus.com/inward/record.uri?eid=2-s2.0-85097350250&partnerID=40&md5=ed52e7350f663feda9a06540f4af24f5 https://scholars.lib.ntu.edu.tw/handle/123456789/581463 |
ISSN: | 10450823 | SDG/關鍵字: | Heuristic algorithms; Optimization; Reinforcement learning; Adaptive compression; Adaptive neural networks; Heuristic search algorithms; Model compression; Network compression; Power constraints; Resource Constraint; Specific hardware; Multilayer neural networks |
顯示於: | 資訊工程學系 |
在 IR 系統中的文件,除了特別指名其著作權條款之外,均受到著作權保護,並且保留所有的權利。