Abstract
We address the problem of imitation learning with multi-modal demonstrations. Instead of attempting to learn all modes, we argue that in many tasks it is sufficient to imitate any one of them. We show that the state-of-the-art methods such as GAIL and behavior cloning, due to their choice of loss function, often incorrectly interpolate between such modes. Our key insight is to minimize the right divergence between the learner and the expert state-action distributions, namely the reverse KL divergence or I-projection. We propose a general imitation learning framework for estimating and minimizing any f-Divergence. By plugging in different divergences, we are able to recover existing algorithms such as Behavior Cloning (Kullback-Leibler), GAIL (Jensen Shannon) and DAgger (Total Variation). Empirical results show that our approximate I-projection technique is able to imitate multi-modal behaviors more reliably than GAIL and behavior cloning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
We refer the reader to [7] for appendices containing detailed exposition.
- 2.
For a convex function \(f(\cdot )\), the convex conjugate is \(f^*(v) = \sup _{u \in \mathrm {dom}_f} \left( uv - f(u) \right) \). Also \((f^*)^* = f\).
References
Ross, S., Melik-Barkhudarov, N., Shankar, K.S., Wendel, A., Dey, D., Bagnell, J.A., Hebert, M.: Learning monocular reactive UAV control in cluttered natural environments. In: 2013 IEEE International Conference on Robotics and Automation (ICRA) (2013)
Finn, C., Levine, S., Abbeel, P.: Guided cost learning: deep inverse optimal control via policy optimization. In: International Conference on Machine Learning, pp. 49–58 (2016)
Pomerleau, D.A.: ALVINN: an autonomous land vehicle in a neural network. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems, vol. 1, pp. 305–313. Morgan-Kaufmann (1989)
Li, Y., Song, J., Ermon, S.: Infogail: interpretable imitation learning from visual demonstrations. In: Advances in Neural Information Processing Systems, pp. 3812–3822 (2017)
Ho, J., Ermon, S.: Generative adversarial imitation learning. In: Advances in Neural Information Processing Systems, pp. 4565–4573 (2016)
Nowozin, S., Cseke, B., Tomioka, R.: f-gan: training generative neural samplers using variational divergence minimization. In: Advances in Neural Information Processing Systems, pp. 271–279 (2016)
Ke, L., Choudhury, S., Barnes, M., Sun, W., Lee, G., Srinivasa, S.: Imitation learning as \(f\)-divergence minimization. arXiv preprint arXiv:1905.12888v2 (2019)
Osa, T., Pajarinen, J., Neumann, G., Bagnell, J.A., Abbeel, P., Peters, J.: An algorithmic perspective on imitation learning. arXiv preprint arXiv:1811.06711 (2018)
Argall, B.D., Chernova, S., Veloso, M., Browning, B.: A survey of robot learning from demonstration. Robot. Autonom. Syst. 57(5), 469–483 (2009)
Billard, A.G., Calinon, S., Dillmann, R.: Learning from humans. In: Siciliano, B., Khatib, O. (eds.) Springer Handbook of Robotics, pp. 1995–2014. Springer, Heidelberg (2016)
Bagnell, J.A.: An invitation to imitation. Technical Report CMU-RI-TR-15-08, Carnegie Mellon University, Pittsburgh, PA, March 2015
Ross, S., Bagnell, J.A.: Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979 (2014)
Sun, W., Venkatraman, A., Gordon, G.J., Boots, B., Bagnell, J.A.: Deeply aggrevated: differentiable imitation learning for sequential prediction. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 3309–3318. JMLR. org (2017)
Sun, W., Bagnell, J.A., Boots, B.: Truncated horizon policy search: combining reinforcement learning & imitation learning. arXiv:1805.11240 (2018)
Cheng, C.A., Yan, X., Wagener, N., Boots, B.: Fast policy learning through imitation and reinforcement. arXiv:1805.10413 (2018)
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., Levine, S.: Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087 (2017)
Pomerleau, D.A.: Alvinn: an autonomous land vehicle in a neural network. In: Advances in Neural Information Processing Systems, pp. 305–313 (1989)
Ross, S., Gordon, G., Bagnell, D.: A reduction of imitation learning and structured prediction to no-regret online learning. In: AISTATS (2011)
Kim, B., Farahmand, A.M., Pineau, J., Precup, D.: Learning from limited demonstrations. In: Advances in Neural Information Processing Systems, pp. 2859–2867 (2013)
Gupta, S., Davidson, J., Levine, S., Sukthankar, R., Malik, J.: Cognitive mapping and planning for visual navigation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
Laskey, M., Lee, J., Hsieh, W., Liaw, R., Mahler, J., Fox, R., Goldberg, K.: Iterative noise injection for scalable imitation learning. arXiv preprint arXiv:1703.09327 (2017)
Laskey, M., Staszak, S., Hsieh, W.Y.S., Mahler, J., Pokorny, F.T., Dragan, A.D. and Goldberg, K.: Shiv: reducing supervisor burden in dagger using support vectors for efficient learning from demonstrations in high dimensional state spaces. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 462–469. IEEE (2016)
Laskey, M., Chuck, C., Lee, J., Mahler, J., Krishnan, S., Jamieson, K., Dragan, A., Goldberg, K.: Comparing human-centric and robot-centric sampling for robot deep learning from demonstrations. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2017)
Ratliff, N.D., Silver, D., Bagnell, J.A.: Learning to search: functional gradient techniques for imitation learning. Autonom. Robots 27(1), 25–53 (2009)
Ratliff, N.D., Bagnell, J.A., Zinkevich, M.A.: Maximum margin planning. In: International Conference on Machine Learning. ACM (2006)
Piot, B., Geist, M., Pietquin, O.: Bridging the gap between imitation learning and inverse reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. 28(8), 1814–1826 (2017)
Abbeel, P., Ng, A.Y.: Apprenticeship learning via inverse reinforcement learning. In: International Conference on Machine Learning. ACM (2004)
Ziebart, B.D., Maas, A.L., Bagnell, J.A., Dey, A.K.: Maximum entropy inverse reinforcement learning. In: AAAI (2008)
Wulfmeier, M., Ondruska, P., Posner, I.: Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.04888 (2015)
Syed, U., Schapire, R.E.: A game-theoretic approach to apprenticeship learning. In: Advances in Neural Information Processing Systems (2008)
Ho, J., Gupta, J., Ermon, S.: Model-free imitation learning with policy optimization. In: International Conference on Machine Learning (2016)
Finn, C., Christiano, P., Abbeel, P., Levine, S.: A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852 (2016)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Blondé, L., Kalousis, A.: Sample-efficient imitation learning via generative adversarial nets. arXiv preprint arXiv:1809.02064 (2018)
Fu, J., Luo, K., Levine, S.: Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248 (2017)
Qureshi, A.H., Boots, B., Yip, M.C.: Adversarial imitation via variational inverse reinforcement learning. arXiv preprint arXiv:1809.06404 (2018)
Peng, X.B., Kanazawa, A., Toyer, S., Abbeel, P., Levine, S.: Variational discriminator bottleneck: improving imitation learning, inverse RL, and GANS by constraining information flow. arXiv preprint arXiv:1810.00821 (2018)
Torabi, F., Warnell, G., Stone, P.: Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158 (2018)
Torabi, F., Warnell, G., Stone, P.: Behavioral cloning from observation. arXiv preprint arXiv:1805.01954 (2018)
Peng, X.B., Kanazawa, A., Malik, J., Abbeel, P., Levine, S.: Sfv: reinforcement learning of physical skills from videos. In: SIGGRAPH Asia 2018 Technical Papers, p. 178. ACM (2018)
Nguyen, X., Wainwright, M.J., Jordan, M.I.: Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory 56(11), 5847–5861 (2010)
Boularias, A., Kober, J., Peters, J.: Relative entropy inverse reinforcement learning. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 182–189 (2011)
Rhinehart, N., Kitani, K.M., Vernaza, P.: R2p2: a reparameterized pushforward policy for diverse, precise generative path forecasting. In: The European Conference on Computer Vision (ECCV), September 2018
Ghasemipour, S.K.S., Gu, S., Zemel, R.: Understanding the relation between maximum-entropy inverse reinforcement learning and behaviour cloning. In: Workshop ICLR (2018)
Babes, M., Marivate, V.N., Subramanian, K., Littman, M.L.: Apprenticeship learning about multiple intentions. In: International Conference on Machine Learning, pp. 897–904 (2011)
Dimitrakakis, C., Rothkopf, C.A.: Bayesian multitask inverse reinforcement learning. In: European Workshop on Reinforcement Learning, pp. 273–284. Springer (2011)
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: Infogan: interpretable representation learning by information maximizing generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2172–2180 (2016)
Hausman, K., Chebotar, Y., Schaal, S., Sukhatme, G., Lim, J.J.: Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 1235–1245 (2017)
Lee, K., Choi, S. and Oh, S.: Maximum causal tsallis entropy imitation learning. In: Advances in Neural Information Processing Systems (2018)
Lee, K., Choi, S., Oh, S.: Sparse Markov decision processes with causal sparse tsallis entropy regularization for reinforcement learning. IEEE Robot. Autom. Lett. 3(3), 1466–1473 (2018)
Belousov, B., Peters, J.: f-divergence constrained policy improvement. arXiv preprint arXiv:1801.00056 (2017)
Csiszár, I., Shields, P.C.: Information Theory and Statistics: A Tutorial. Now Publishers Inc, Norwell (2004)
Liese, F., Vajda, I.: On divergences and informations in statistics and information theory. IEEE Trans. Inf. Theory 52(10), 4394–4412 (2006)
Kanamori, T., Suzuki, T., Sugiyama, M.: Statistical analysis of kernel-based least-squares density-ratio estimation. Mach. Learn. 86(3), 335–367 (2012)
Zhang, M., Bird, T., Habib, R., Xu, T., Barber, D.: Variational f-divergence minimization. arXiv preprint arXiv:1907.11891 (2019)
Sun, W., Vemula, A., Boots, B., Bagnell, J.A.: Provably efficient imitation learning from observation alone. arXiv preprint arXiv:1905.10948 (2019)
Acknowledgements
This work was (partially) funded by the National Institute of Health R01 (#R01EB019335), National Science Foundation CPS (#1544797), National Science Foundation NRI (#1637748), the Office of Naval Research, the RCTA, Amazon, and Honda Research Institute USA.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ke, L., Choudhury, S., Barnes, M., Sun, W., Lee, G., Srinivasa, S. (2021). Imitation Learning as f-Divergence Minimization. In: LaValle, S.M., Lin, M., Ojala, T., Shell, D., Yu, J. (eds) Algorithmic Foundations of Robotics XIV. WAFR 2020. Springer Proceedings in Advanced Robotics, vol 17. Springer, Cham. https://doi.org/10.1007/978-3-030-66723-8_19
Download citation
DOI: https://doi.org/10.1007/978-3-030-66723-8_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-66722-1
Online ISBN: 978-3-030-66723-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)