https://scholars.lib.ntu.edu.tw/handle/123456789/632138
標題: | Limited-memory common-directions method for large-scale optimization: convergence, parallelization, and distributed optimization | 作者: | Lee C.-P Wang P.-W CHIH-JEN LIN |
關鍵字: | First-order method; Optimal method; Second-order method; Smooth optimization | 公開日期: | 2022 | 來源出版物: | Mathematical Programming Computation | 摘要: | In this paper, we present a limited-memory common-directions method for smooth optimization that interpolates between first- and second-order methods. At each iteration, a subspace of a limited dimension size is constructed using first-order information from previous iterations, and an efficient Newton method is deployed to find an approximate minimizer within this subspace. With properly selected subspace of dimension as small as two, the proposed algorithm achieves the optimal convergence rates for first-order methods while remaining a descent method, and it also possesses fast convergence speed on nonconvex problems. Since the major operations of our method are dense matrix-matrix operations, the proposed method can be efficiently parallelized in multicore environments even for sparse problems. By wisely utilizing historical information, our method is also communication-efficient in distributed optimization that uses multiple machines as the Newton steps can be calculated with little communication. Numerical study shows that our method has superior empirical performance on real-world large-scale machine learning problems. © 2022, The Author(s). |
URI: | https://www.scopus.com/inward/record.uri?eid=2-s2.0-85127354471&doi=10.1007%2fs12532-022-00219-z&partnerID=40&md5=2c5f5c095ca5d88a77fe4cddba851dd5 https://scholars.lib.ntu.edu.tw/handle/123456789/632138 |
ISSN: | 18672949 | DOI: | 10.1007/s12532-022-00219-z | SDG/關鍵字: | Numerical methods; Distributed optimization; First-order methods; Large-scale optimization; Limited memory; Optimal methods; Optimisations; Optimization convergence; Parallelizations; Second-order methods; Smooth optimization; Newton-Raphson method |
顯示於: | 資訊工程學系 |
在 IR 系統中的文件,除了特別指名其著作權條款之外,均受到著作權保護,並且保留所有的權利。