Martin Pašen, Vladimír Boža. Merging of neural networks. Technical Report 2204.09973, arXiv, 2022.
Download preprint: not available
Download from publisher: https://doi.org/10.48550/arXiv.2204.09973
Related web page: not available
Bibliography entry: BibTeX
We propose a simple scheme for merging two neural networks trained with different starting initialization into a single one with the same size as the original ones. We do this by carefully selecting channels from each input network. Our procedure might be used as a finalization step after one tries multiple starting seeds to avoid an unlucky one. We also show that training two networks and merging them leads to better performance than training a single network for an extended period of time.