LoGIC: Multi-LoRA Guided Importance Consensus for Multi-Task Pruning in Vision Transformers
Journal
Proceedings of the AAAI Conference on Artificial Intelligence
Journal Volume
40
Journal Issue
25
Start Page
20588
End Page
20596
ISSN
2374-3468
2159-5399
Date Issued
2026-03-14
Author(s)
Abstract
Deploying Vision Transformers (ViTs) in real-world multi-task learning remains challenging due to their massive computational costs and the difficulty of pruning shared backbones without harming task performance. Single-task pruning often causes destructive interference by discarding weights critical to other tasks, while existing multi-task pruning strategies remain costly and unscalable for billion-parameter models. We propose Multi-LoRA Guided Importance Consensus (LoGIC), a unified framework for efficient and robust multi-task ViT pruning. LoGIC follows a two-phase procedure: (i) task-consistent pruning of LoRA modules, guided by a task-adaptive gating mechanism that balances shared and task-specific contributions while enforcing structured sparsity for deployment; and (ii) cross-task consensus pruning of the frozen ViT backbone, which retains both universally shared and task-specialized capabilities, enabling aggressive sparsity without sacrificing accuracy. Across five diverse vision benchmarks, LoGIC achieves up to 50% structured sparsity while maintaining competitive accuracy and surpassing all baselines.
Event(s)
40th AAAI Conference on Artificial Intelligence, AAAI 2026
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Type
conference paper