In this paper, we propose a novel language model for Russian large vocabulary speech recognition based on sequence memoizer modeling technique. Sequence memoizer is a long span text dependency model and was initially proposed for character language modeling. Here, we use it to build word level language model (LM) in ASR. We compare its performance with recurrent neural network (RNN) LM, which also models long span word dependencies. A number of experiments were carried out using various amounts of train data and different text data arrangements. According to our experimental results, the sequence memoizer LM outperforms recurrent neural network and standard 3-gram LMs in terms of perplexity, while RNN LM achieves better word error rate. The lowest word error rate is achieved by combining all three language models together using linear interpolation.

Index Terms: sequence memoizer, advanced language modeling, inflective languages