Fast Monte-Carlo Approximation of the Attention Mechanism
DOI:
https://doi.org/10.1609/aaai.v36i7.20679Keywords:
Machine Learning (ML)Abstract
We introduce Monte-Carlo Attention (MCA), a randomized approximation method for reducing the computational cost of self-attention mechanisms in Transformer architectures. MCA exploits the fact that the importance of each token in an input sequence vary with respect to their attention scores; thus, some degree of error can be tolerable when encoding tokens with low attention. Using approximate matrix multiplication, MCA applies different error bounds to encode input tokens such that those with low attention scores are computed with relaxed precision, whereas errors of salient elements are minimized. MCA can operate in parallel with other attention optimization schemes and does not require model modification. We study the theoretical error bounds and demonstrate that MCA reduces attention complexity (in FLOPS) for various Transformer models by up to 11 in GLUE benchmarks without compromising model accuracy. Source code and appendix: https://github.com/eis-lab/monte-carlo-attention
Downloads
Published
2022-06-28
How to Cite
Kim, H., & Ko, J. (2022). Fast Monte-Carlo Approximation of the Attention Mechanism. Proceedings of the AAAI Conference on Artificial Intelligence, 36(7), 7185-7193. https://doi.org/10.1609/aaai.v36i7.20679
Issue
Section
AAAI Technical Track on Machine Learning II
