High Dimensional Analog Range In-3D-NAND Search Accelerator for Applications of Search in Few-Shot Learning Model and Retrieval in Retrieval Augmented Generation
Part Of
Technical Digest - International Electron Devices Meeting, IEDM
Start Page
1
End Page
4
ISBN (of the container)
979-835036542-9
Date Issued
2024-12-07
Author(s)
Po-Hao Tseng
Shao-Yu Fang
Chi-Tse Huang
Hao-Wei Chiang
Feng-Ming Lee
Yu-Hsuan Lin
Jhe-Yi Liao
Yu-Yu Lin
Hsiang-Yun Cheng
Ming-Hsiu Lee
Kuang-Yeu Hsieh
Keh-Chung Wang
Chih-Yuan Lu
Abstract
We developed a novel analog range in-memory searching technology (AR-IMS) using high dimensional serial-parallel array architecture in 3D-NAND flash chip for applications in few-shot learning (FSL) models and retrieval augmented generation (RAG). A controller (or feature extractor) for analog vector extraction of 492 dimensions (features) was optimized from a common model by in-field data to improve system accuracy. The current-clamping function in the AR-IMS array significantly reduces the matching current variation and keeps the total matching current at low level to meet the design requirements for the sense amplifiers. A special analog range level quantization scheme was developed to ensure in-field performance and matching tolerance for high system reliability. The feature vector data bits stored in the AR-IMS are quantized up to 64 levels, and the query vector is quantized into 100 levels. The proposed integrated system (Optimizing controller + AR-IMS accelerator) enable high performance analog computing for continuous-learning FSL AI systems. As compared to the CPU stand-alone system, the AR-IMS accelerator with CPU or GPU integration provided high speed (57x to 258x), low power computing performance (70x to 158x) for RAG in large language model (LLM) generative process.
Event(s)
2024 IEEE International Electron Devices Meeting, IEDM
Publisher
IEEE
Type
conference paper