Abstract
In this paper we introduce a budgeted knowledge transfer algorithm for non-homogeneous reinforcement learning agents. Here the source and the target agents are completely identical except in their state representations. The algorithm uses functional space (Q-value space) as the transfer-learning media. In this method, the target agent’s functional points (Q-values) are estimated in an automatically selected lower-dimension subspace in order to accelerate knowledge transfer. The target agent searches that subspace using an exploration policy and selects actions accordingly during the period of its knowledge transfer in order to facilitate gaining an appropriate estimate of its Q-table. We show both analytically and empirically that this method decreases the required learning budget for the target agent.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: A survey. The Journal of Machine Learning Research 10, 1633–1685 (2009)
Torrey, L., Shavlik, J.: Transfer learning. In: Handbook of Research on Machine Learning Applications, vol. 3, pp. 17–35. IGI Global (2009)
Lazaric: Knowledge transfer in reinforcement learning. PhD thesis, PhD thesis, Politecnico di Milano (2008)
Tanaka, F., Yamamura, M.: Multitask reinforcement learning on the distribution ofMDPs. In: Proceedings. 2003 IEEE International Symposium on Computational Intelligence in Robotics and Automation, vol. 3, pp. 1108–1113 (2003)
Taylor, M.E., Stone, P., Liu, Y.: Value functions for RL-based behavior transfer: Acomparative study. In: Proceedings of the National Conference on Artificial Intelligence, vol. 20, p. 880 (2005)
Wilson, A., Fern, A., Ray, S., Tadepalli, P.: Multi-task reinforcement learning: a hierarchical bayesian approach. In: Proceedings of the 24th International Conference on Machine learning, pp. 1015–1022 (2007)
Soni, V., Singh, S.: Using homeomorphisms to transfer options across continuous reinforcement learning domains. In: Proceedings of the National Conference on Artificial Inligence, vol. 21, p. 494 (2006)
Mihalkova, L., Huynh, T., Mooney, R.J.: Mapping and revising Markov logic networksfor transfer learning. In: Proceedings of the National Conference on Artificial Intelligence, vol. 22, p. 608 (2007)
Taylor, M.E., Whiteson, S., Stone, P.: Transfer via inter-task mappings in policy search reinforcement learning. In: Proceedings of the 6th International Joint Conference on Autonomous Agents and Multi-agent Systems, p. 37 (2007)
Taylor, M.E., Jong, N.K., Stone, P.: Transferring Instances for Model-Based Reinforcement Learning. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008, Part II. LNCS (LNAI), vol. 5212, pp. 488–505. Springer, Heidelberg (2008)
Driessens, K., Ramon, J., Croonenborghs, T.: Transfer learning for reinforcement learning through goal and policy parameterization. In: Proceedings of the ICML Workshop on Structural Knowledge Transfer for Machine Learning (Online Proceedings), p. 14 (2006)
Pan, S.J., Kwok, J.T., Yang, Q.: Transfer learning via dimensionality reduction. In: Proceedings of the 23rd National Conference on Artificial Intelligence, vol. 2, pp. 677–682 (2008)
Moore, A.W., Atkeson, C.G.: Prioritized sweeping: Reinforcement learning with lessdata and less time. Machine Learning 13(1), 103–130 (1993)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Farshidian, F., Talebpour, Z., Ahmadabadi, M.N. (2012). Budgeted Knowledge Transfer for State-Wise Heterogeneous RL Agents. In: Huang, T., Zeng, Z., Li, C., Leung, C.S. (eds) Neural Information Processing. ICONIP 2012. Lecture Notes in Computer Science, vol 7663. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34475-6_53
Download citation
DOI: https://doi.org/10.1007/978-3-642-34475-6_53
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-34474-9
Online ISBN: 978-3-642-34475-6
eBook Packages: Computer ScienceComputer Science (R0)