A correlation-permutation approach for speech-music encoders model merging
Journal
2025 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)
Start Page
1
End Page
7
ISBN (of the container)
979-833154426-3
ISBN
[9798331544263]
Date Issued
2025-12-06
Author(s)
Abstract
Creating a unified speech and music model requires expensive pre-training. Model merging can instead create a unified audio model with minimal computational expense. However, direct merging is challenging when the models are not aligned in the weight space. Motivated by Git Re-Basin, we introduce a correlation-permutation approach that aligns a music encoder's internal layers with a speech encoder. We extend previous work to the case of merging transformer layers. The method computes a permutation matrix that maximizes the model's features-wise cross-correlations layer by layer, enabling effective fusion of these otherwise disjoint models. The merged model retains speech capabilities through this method while significantly enhancing music performance, achieving an improvement of 14.83 points in average score compared to the linear interpolation model merging. This work allows the creation of unified audio models from independently trained encoders.
Subjects
audio foundation models
Model merging
self-supervised learning
Publisher
Institute of Electrical and Electronics Engineers(IEEE)
Type
conference paper