Data-Driven MoE: A Data-Driven Approach to Construct MoE by a Single LLM

Abstract

With the emergence of the Mixtral-8x7B model, Mixture of Experts (MoE) models have gained people’s attention by its ability to outperform Large Language Models (LLMs) with much larger parameter scales while requiring less resource consumption. In details, the construction of a MoE model involves incorporating gate layers and utilizing sparse MoE layers to replace the feed-forward neural network layers of a LLM. Existing methods for constructing MoE models primarily involve reusing some well-trained neural network layers and initializing gate layers randomly. However, this process still demands a considerable amount of training, and additional training techniques must be implemented to ensure that each expert is sufficiently trained. To address the problems of elevated training expenses and low fault tolerance associated with existing MoE methods, we propose a simple, efficient, and effective MoE construction method - data-driven MoE construction method. In this method, rather than relying on random initialization, gate layers are initialized with a series of supervised fine-tuning data that encompass the scenarios in which the MoE model will be utilized. Sparse MoE layers are then constructed by replicating individual LLM’s FFN layer with noises. The proposed method offers a straightforward, convenient, and efficient approach for constructing MoE models. The experimental results indicate that the proposed data-driven MoE model surpasses existing approaches in terms of performance on both domain-specific and general tasks. The maximum improvements observed were 7.26, 9.96, 8.99, and 5.25 in the Rouge calculation for the medical, math, law, and general domains, respectively.