Abstract
Many approaches to training generative models by distinct training objectives have been proposed in the past. Variational Autoencoder (VAE) is an outstanding model of them based on log-likelihood. In this paper, we propose a novel learnable prior, Pull-back Prior, for VAEs by adjusting the density of the prior through a discriminator that can assess the quality of data. It involves the discriminator from the theory of GANs to enrich the prior in VAEs. Based on it, we propose a more general framework, VAE with a Pull-back Prior (VAEPP), which uses existing techniques of VAEs and WGANs, to improve the log-likelihood, quality of sampling and stability of training. In MNIST and CIFAR-10, the log-likelihood of VAEPP outperforms models without autoregressive components and is comparable to autoregressive models. In MNIST, Fashion-MNIST, CIFAR-10 and CelebA, the FID of VAEPP is comparable to GANs and SOTA of VAEs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Arjovsky, M., Chintala, S., et al.: Wasserstein generative adversarial networks. In: International Conference on Machine Learning, pp. 214–223 (2017)
Bauer, M., Mnih, A.: Resampled priors for variational autoencoders. In: The 22nd International Conference on Artificial Intelligence and Statistics, pp. 66–75 (2019)
Chen, X., Mishra, N., et al.: PixelSNAIL: an improved autoregressive generative model. In: International Conference on Machine Learning, pp. 863–871 (2018)
Dai, B., Wipf, P.D.: Diagnosing and enhancing VAE models. In: ICLR (2019)
Gelfand, I.M., Silverman, R.A., et al.: Calculus of variations. Courier Corporation (2000)
Goodfellow, I., Pouget-Abadie, J., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Gulrajani, I., Ahmed, F., et al.: Improved training of Wasserstein GANs. In: NIPS (2017)
Heusel, M., Ramsauer, H., et al.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: Advances in Neural Information Processing Systems, pp. 6626–6637 (2017)
Hoffman, M.D., Johnson, M.J.: Elbo surgery: yet another way to carve up the variational evidence lower bound. In: Workshop in Advances in Approximate Bayesian Inference, NIPS, vol. 1 (2016)
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: ICLR (2014)
Kingma, D.P., Dhariwal, P.: Glow: generative flow with invertible 1 \(\times \) 1 convolutions. In: Advances in Neural Information Processing Systems, pp. 10215–10224 (2018)
Kingma, D.P., Salimans, T., et al.: Improved variational inference with inverse autoregressive flow. In: Advances in Neural Information Processing Systems, pp. 4743–4751 (2016)
Kumar, R., Goyal, A., et al.: Maximum entropy generators for energy-based models. arXiv preprint arXiv:1901.08508 (2019)
Lake, B.M., Salakhutdinov, R., et al.: Human-level concept learning through probabilistic program induction. Science 350(6266), 1332–1338 (2015)
Larochelle, H., Murray, I.: The neural autoregressive distribution estimator. In: AISTATS, pp. 29–37 (2011)
Liu, Z., Luo, P., et al.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)
Maaløe, L., Fraccaro, M., et al.: BIVA: a very deep hierarchy of latent variables for generative modeling. In: NeurIPS (2019)
Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, J.I.: Adversarial autoencoders. CoRR (2015)
Metz, L., Poole, B., et al.: Unrolled generative adversarial networks. In: ICLR (2017)
Van den Oord, A., Kalchbrenner, N., et al.: Conditional image generation with PixelCNN decoders. In: Advances in Neural Information Processing Systems, pp. 4790–4798 (2016)
Rezende, D.J., Mohamed, S., et al.: Stochastic backpropagation and approximate inference in deep generative models. In: ICML (2014)
Salimans, T., Goodfellow, I., et al.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, pp. 2234–2242 (2016)
Salimans, T., Karpathy, A., Chen, X., Kingma, P.D.: PixelCNN++: improving the PixelCNN with discretized logistic mixture likelihood and other modifications. In: ICLR (2017)
Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution. In: Advances in Neural Information Processing Systems, pp. 11895–11907 (2019)
Takahashi, H., Iwata, T., et al.: Variational autoencoder with implicit optimal priors. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5066–5073 (2019)
Tomczak, J., Welling, M.: VAE with a VampPrior. In: International Conference on Artificial Intelligence and Statistics, pp. 1214–1223 (2018)
Tomczak, J.M., Welling, M.: Improving variational auto-encoders using householder flow. arXiv preprint arXiv:1611.09630 (2016)
Wu, J., Huang, Z., Thoma, J., Acharya, D., Van Gool, L.: Wasserstein divergence for GANs. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11209, pp. 673–688. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01228-1_40
Xiao, H., Rasul, K., et al.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017)
Acknowledgments
This work has been supported by National Key R&D Program of China 2019YFB1802504 and the Beijing National Research Center for Information Science and Technology (BNRist) key projects.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, W., Liu, W., Cai, Z., Xu, H., Pei, D. (2020). VAEPP: Variational Autoencoder with a Pull-Back Prior. In: Yang, H., Pasupa, K., Leung, A.CS., Kwok, J.T., Chan, J.H., King, I. (eds) Neural Information Processing. ICONIP 2020. Lecture Notes in Computer Science(), vol 12534. Springer, Cham. https://doi.org/10.1007/978-3-030-63836-8_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-63836-8_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63835-1
Online ISBN: 978-3-030-63836-8
eBook Packages: Computer ScienceComputer Science (R0)