Abstract
Compositional relations represent a good source of information in the task of scene understanding. However, current approaches in domestic service robotics only scratch the surface of the benefits of compositional relations by leveraging only their spatial component. In this position paper, we propose a new perspective on the use of compositional relations as a means to extract meaning from context in open-ended interactions. We especially design a multi-layer representation based on scene graphs that encapsulates four different dimensions of knowledge. To exploit this new representation, we introduce a new large-scale dataset for indoor service robots with high-quality scene graph annotations. We then argue for the opportunities of using this representation to easily extract a wide range of fine-grained information about human interaction with context (All data and code are available at https://github.com/Maelic/IndoorVG).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Agia, C., et al.: Taskography: evaluating robot task planning over large 3D scene graphs. In: Conference on Robot Learning. PMLR (2022)
Amiri, S., Chandan, K., Zhang, S.: Reasoning with scene graphs for robot planning under partial observability. IEEE Robot. Autom. Lett. 7(2), 5560–5567 (2022)
Beetz, M., et al.: Know rob 2.0: a 2nd generation knowledge processing framework for cognition-enabled robotic agents. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2018)
Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)
Chatpatanasiri, R.: GPT3 and commonsense reasoning (2021). https://agi.miraheze.org/wiki/GPT3_and_Commonsense_Reasoning. Accessed 30 Apr 2023
De Magistris, G., et al.: Vision-based holistic scene understanding for context-aware human-robot interaction. In: Bandini, S., Gasparini, F., Mascardi, V., Palmonari, M., Vizzari, G. (eds.) AIxIA 2021. LNCS, vol. 13196, pp. 310–325. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-08421-8_21
Gadre, S.Y., et al.: Continuous scene representations for embodied AI. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14849–14859 (2022)
Graf, F., et al.: Toward holistic scene understanding: a transfer of human scene perception to mobile robots. IEEE Robot. Autom. Mag. 29(4), 36–49 (2022)
Gupta, R., et al.: Common sense data acquisition for indoor mobile robots. In: AAAI, pp. 605–610 (2004)
Krishna, R., et al.: Visual genome: connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vision 123(1), 32–73 (2017)
Lemaignan, S., et al.: Oro, a knowledge management platform for cognitive architectures in robotics. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3548–3553. IEEE (2010)
Li, L., et al.: The devil is in the labels: Noisy label correction for robust scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18869–18878 (2022)
Li, X., et al.: Embodied semantic scene graph generation. In: Proceedings of the 5th Conference on Robot Learning, pp. 1585–1594. PMLR (2022). ISSN 2640-3498
Lin, X., et al.: GPS-Net: graph property sensing network for scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3746–3753 (2020)
Cewu, L., Krishna, R., Bernstein, M., Fei-Fei, L.: Visual relationship detection with language priors. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 852–869. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_51
Paulius, D., Jelodar, A.B., Sun, Y.: Functional object-oriented network: construction & expansion. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5935–5941 (2018)
Ren, S., et al.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)
Sado, F., et al.: Explainable goal-driven agents and robots-a comprehensive review. ACM Comput. Surv. 55(10), 1–41 (2023)
Saxena, A., et al.: Robobrain: large-scale knowledge engine for robots (2015)
Tang, K., et al.: Learning to compose dynamic tree structures for visual contexts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6619–6628 (2019)
Tang, K., et al.: Unbiased scene graph generation from biased training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3716–3725 (2020)
Tang, K., et al.: Unbiased scene graph generation from biased training. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, pp. 3713–3722. IEEE (2020)
Xu, D., et al.: Scene graph generation by iterative message passing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5410–5419 (2017)
Yan, S., et al.: PCPL: predicate-correlation perception learning for unbiased scene graph generation. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 265–273 (2020)
Zellers, R., et al.: Neural motifs: scene graph parsing with global context. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5831–5840 (2018)
Zhang, A., et al.: Fine-grained scene graph generation with data transfer. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13687, pp. 409–424. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19812-0_24
Acknowledgment
This work benefits from the support of Britanny region.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Neau, M., Santos, P., Bosser, AG., Buche, C. (2024). In Defense of Scene Graph Generation for Human-Robot Open-Ended Interaction in Service Robotics. In: Buche, C., Rossi, A., Simões, M., Visser, U. (eds) RoboCup 2023: Robot World Cup XXVI. RoboCup 2023. Lecture Notes in Computer Science(), vol 14140. Springer, Cham. https://doi.org/10.1007/978-3-031-55015-7_25
Download citation
DOI: https://doi.org/10.1007/978-3-031-55015-7_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-55014-0
Online ISBN: 978-3-031-55015-7
eBook Packages: Computer ScienceComputer Science (R0)