Abstract
In this paper we tackle the container allocation problem in multimodal transportation planning under uncertainty in container arrival times, using Deep Reinforcement Learning. The proposed approach can take real-time decisions on allocating individual containers to a truck or to trains, while a transportation plan is being executed. We evaluated our method using data that reflect a realistic scenario, designed on the basis of a case study at a logistics company with three different uncertainty levels based on the probability of delays in container arrivals. The experiments show that Deep Reinforcement Learning methods outperform heuristics, a stochastic programming method, and methods that use periodic re-planning, in terms of total transportation costs at all levels of uncertainty, obtaining an average cost difference with the optimal solution within 0.37% and 0.63%.
The work leading up to this paper is partly funded by the European Commission under the FENIX project (grant nr. INEA/CEF/TRAN/M2018/1793401).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alves, J.C., Mateus, G.R.: Deep reinforcement learning and optimization approach for multi-echelon supply chain with uncertain demands. In: Lalla-Ruiz, E., Mes, M., Voß, S. (eds.) ICCL 2020. LNCS, vol. 12433, pp. 584–599. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59747-4_38
Barron, E., Ishii, H.: The Bellman equation for minimizing the maximum cost. Nonlinear Anal. Theory Methods Appl. 13(9), 1067–1090 (1989)
Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The arcade learning environment: an evaluation platform for general agents. J. Artif. Intell. Res. 47, 253–279 (2013)
Bhargavi, K., Babu, B.S.: Soft-set based DDQ scheduler for optimal task scheduling under uncertainty in the cloud. In: 2017 2nd International Conference On Emerging Computation and Information Technologies (ICECIT), pp. 1–6. IEEE (2017)
Delbart, T., Molenbruch, Y., Braekers, K., Caris, A.: Uncertainty in intermodal and synchromodal transport: Review and future research directions. Sustainability 13(7), 3980 (2021)
Escudero, A., Muñuzuri, J., Guadix, J., Arango, C.: Dynamic approach to solve the daily drayage problem with transit time uncertainty. Comput. Ind 64(2), 165–175 (2013)
Fang, D., Guan, X., Peng, Y., Chen, H., Ohtsuki, T., Han, Z.: Distributed deep reinforcement learning for renewable energy accommodation assessment with communication uncertainty in Internet of Energy. IEEE Internet Things J. 8, 8557–8569 (2020)
Farahani, A., Genga, L., Dijkman, R.: Online multimodal transportation planning using deep reinforcement learning. arXiv preprint arXiv:2105.08374 (2021)
Gao, Y., Yang, J., Yang, M., Li, Z.: Deep reinforcement learning based optimal schedule for a battery swapping station considering uncertainties. IEEE Trans. Ind. Appl. 56(5), 5775–5784 (2020)
Gumuskaya, V., van Jaarsveld, W., Dijkman, R., Grefen, P., Veenstra, A.: Dynamic barge planning with stochastic container arrivals. Transp. Res. Part E Logist. Transp. Rev. 144, 102161 (2020)
Ma, H., Yu, G., She, Y., Gu, Y., et al.: Waterflooding optimization under geological uncertainties by using deep reinforcement learning algorithms. In: SPE Annual Technical Conference and Exhibition (2019). Society of Petroleum Engineers
Mnih, V., et al.: Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Peng, Z., Zhang, Y., Feng, Y., Zhang, T., Wu, Z., Su, H.: Deep reinforcement learning approach for capacitated supply chain optimization under demand uncertainty. In: 2019 Chinese Automation Congress (CAC), pp. 3512–3517. IEEE (2019)
Powell, W.B., Jaillet, P., Odoni, A.: Stochastic and dynamic networks and routing. Handb. Oper. Res. Manag. Sci. 8, 141–295 (1995)
Rivera, A.P., Mes, M.R.: Anticipatory scheduling of freight in a synchromodal transportation network. Transp. Res. Part E Logist. Transp. Rev. 105, 176–194 (2017)
Sakib, N.: Highway lane change under uncertainty with deep reinforcement learning based motion planner (2020)
Shyalika, C., Silva, T.: Reinforcement learning based an integrated approach for uncertainty scheduling in adaptive environments using MARL. In: 2021 6th International Conference on Inventive Computation Technologies (ICICT), pp. 1204–1211. IEEE (2021)
SteadieSeifi, M., Dellaert, N.P., Nuijten, W., Van Woensel, T., Raoufi, R.: Multimodal freight transportation planning: a literature review. Eur. J. Oper. Res. 233(1), 1–15 (2014)
Tokic, M., Palm, G.: Value-difference based exploration: adaptive control between epsilon-greedy and softmax. In: Bach, J., Edelkamp, S. (eds.) KI 2011. LNCS (LNAI), vol. 7006, pp. 335–346. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24455-1_33
Topaloglu, H.: A parallelizable and approximate dynamic programming-based dynamic fleet management model with random travel times and multiple vehicle types. In: Dynamic Fleet Management, pp. 65–93. Springer, Heidelberg (2007). https://doi.org/10.1007/978-0-387-71722-7_4
van Riessen, B., Negenborn, R.R., Dekker, R.: Real-time container transport planning with decision trees based on offline obtained optimal solutions. Decis. Supp. Syst. 89, 1–16 (2016)
Wan, K., Gao, X., Hu, Z., Wu, G.: Robust motion control for UAV in dynamic uncertain environments using deep reinforcement learning. Remote Sens. 12(4), 640 (2020)
Wang, P., Li, Y., Shekhar, S., Northrop, W.F.: Uncertainty estimation with distributional reinforcement learning for applications in intelligent transportation systems: a case study. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 3822–3827. IEEE (2019)
Yang, J., Yang, M., Wang, M., Du, P., Yu, Y.: A deep reinforcement learning method for managing wind farm uncertainties through energy storage system control and external reserve purchasing. Int. J. Electric. Power Energy Syst. 119, 105928 (2020)
Yehia, A.: Understanding uncertainty: a reinforcement learning approach for project-level pavement management systems. PhD thesis, University of British Columbia (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Farahani, A., Genga, L., Dijkman, R. (2021). Tackling Uncertainty in Online Multimodal Transportation Planning Using Deep Reinforcement Learning. In: Mes, M., Lalla-Ruiz, E., Voß, S. (eds) Computational Logistics. ICCL 2021. Lecture Notes in Computer Science(), vol 13004. Springer, Cham. https://doi.org/10.1007/978-3-030-87672-2_38
Download citation
DOI: https://doi.org/10.1007/978-3-030-87672-2_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87671-5
Online ISBN: 978-3-030-87672-2
eBook Packages: Computer ScienceComputer Science (R0)