ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations | SpringerLink
Skip to main content
Springer Nature Link
Log in

ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations

  • Conference paper
  • First Online:
Performance Evaluation Methodologies and Tools (VALUETOOLS 2021)

Abstract

Efficient and dependable communication is a highly relevant aspect for Internet of Things (IoT) systems in which tiny sensors, actuators, wearables, or other smart devices exchange messages. Various publish/subscribe protocols address the challenges of communication in IoT systems. The selection process of a suitable protocol should consider the communication behavior of the application, the protocol’s performance, the resource requirements on the end device, and the network connection quality, as IoT environments often rely on wireless networks. Benchmarking is a common approach to evaluate and compare systems, considering the performance and aspects like dependability or security. In this paper, we present our IoT communication benchmarking framework ComBench for publish/subscribe protocols focusing on constrained networks with varying quality conditions. The benchmarking framework supports system designers, software engineers, and application developers to select and investigate the behavior of communication protocols. Our benchmarking framework contributes to (i) show the impact of fluctuating network quality on communication, (ii) compare multiple protocols, protocol features, and protocol implementations, and (iii) analyze scalability, robustness, and dependability of clients, networks, and brokers in different scenarios. Our case study demonstrates the applicability of our framework to support the decision for the best-suited protocol in various scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 5719
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 7149
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://pypi.org/project/asyncio-mqtt/.

  2. 2.

    https://pika.readthedocs.io/.

  3. 3.

    https://github.com/chrysn/aiocoap.

  4. 4.

    https://hub.docker.com/r/descartesresearch/iot-pubsub-benchmark-controller.

  5. 5.

    https://hub.docker.com/r/descartesresearch/iot-pubsub-benchmark-client.

References

  1. Al-Fuqaha, A., Guizani, M., Mohammadi, M., Aledhari, M., Ayyash, M.: Internet of things: a survey on enabling technologies, protocols, and applications. IEEE Commun. Surv. Tutor. 17(4), 2347–2376 (2015). https://doi.org/10.1109/COMST.2015.2444095

    Article  Google Scholar 

  2. Appel, S., Sachs, K., Buchmann, A.: Towards benchmarking of AMQP. In: Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems, pp. 99–100 (2010)

    Google Scholar 

  3. Bansal, S., Kumar, D.: IoT application layer protocols: performance analysis and significance in smart city. In: 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT), pp. 1–6. IEEE (2019)

    Google Scholar 

  4. Boano, C.A., et al.: IoTBench: towards a benchmark for low-power wireless networking. In: 2018 IEEE Workshop on Benchmarking Cyber-Physical Networks and Systems (CPSBench). IEEE (2018)

    Google Scholar 

  5. Chen, Y., Kunz, T.: Performance evaluation of IoT protocols under a constrained wireless access network. In: 2016 International Conference on Selected Topics in Mobile & Wireless Networking (MoWNeT), pp. 1–7. IEEE (2016)

    Google Scholar 

  6. Collina, M., Bartolucci, M., Vanelli-Coralli, A., Corazza, G.E.: Internet of Things application layer protocol analysis over error and delay prone links. In: 2014 7th Advanced Satellite Multimedia Systems Conference and the 13th Signal Processing for Space Communications Workshop (ASMS/SPSC), pp. 398–404. IEEE (2014)

    Google Scholar 

  7. Community: Locust Website (2021). https://locust.io/, Accessed 15 Apr 2021

  8. Community: MZBench Website (2021). https://github.com/satori-com/mzbench, Accessed 13 Apr 2021

  9. Corp, G.: Gatling Website (2021). https://gatling.io/, Accessed 13 Apr 2021

  10. Dizdarević, J., Carpio, F., Jukan, A., Masip-Bruin, X.: A survey of communication protocols for internet of things and related challenges of fog and cloud computing integration. ACM Comput. Surv. (CSUR) 51, 1–29 (2019)

    Article  Google Scholar 

  11. Eugster, P.T., Felber, P.A., Guerraoui, R., Kermarrec, A.M.: The many faces of publish/subscribe. ACM Comput. Surv. (CSUR) 35(2), 114–131 (2003)

    Article  Google Scholar 

  12. Gündoğran, C., Kietzmann, P., Lenders, M., Petersen, H., Schmidt, T.C., Wählisch, M.: NDN, CoAP, and MQTT: a comparative measurement study in the IoT. In: Proceedings of the 5th ACM Conference on Information-Centric Networking, pp. 159–171 (2018)

    Google Scholar 

  13. Halili, E.: Apache JMeter. Packt Publishing, Birmingham (2008)

    Google Scholar 

  14. Herrnleben, S., et al.: Evaluation results of ComBench as open data. Technical report, University of Wuerzburg (2021). https://doi.org/10.5281/zenodo.4723344, Accessed 30 Apr 2021

  15. Iglesias-Urkia, M., Orive, A., Barcelo, M., Moran, A., Bilbao, J., Urbieta, A.: Towards a lightweight protocol for industry 4.0: an implementation based benchmark. In: 2017 IEEE International Workshop of Electronics, Control, Measurement, Signals and their Application to Mechatronics (ECMSM) (2017)

    Google Scholar 

  16. Jacob, R., Boano, C.A., Raza, U., Zimmerling, M., Thiele, L.: Towards a methodology for experimental evaluation in low-power wireless networking. In: Proceedings of the 2nd Workshop on Benchmarking Cyber-Physical Systems and Internet of Things, pp. 18–23 (2019)

    Google Scholar 

  17. Kayal, P., Perros, H.: A comparison of IoT application layer protocols through a smart parking implementation. In: 2017 20th Conference on Innovations in Clouds, Internet and Networks (ICIN), pp. 331–336. IEEE (2017)

    Google Scholar 

  18. von Kistowski, J., Arnold, J.A., Huppler, K., Lange, K.D., Henning, J.L., Cao, P.: How to build a benchmark. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015), ICPE ’15. ACM, New York (2015)

    Google Scholar 

  19. Kounev, S., Lange, K.D., von Kistowski, J.: Systems Benchmarking. Springer, Heidelberg (2020). https://doi.org/10.1007/978-3-030-41705-5

    Book  Google Scholar 

  20. Naik, N.: Choice of effective messaging protocols for IoT systems: MQTT, CoAP, AMQP and HTTP. In: 2017 IEEE International Systems Engineering Symposium (ISSE), pp. 1–7. IEEE (2017)

    Google Scholar 

  21. Neagoe, T., Cristea, V., Banica, L.: NTP versus PTP in computer networks clock synchronization. In: 2006 IEEE International Symposium on Industrial Electronics, vol. 1, pp. 317–362. IEEE (2006)

    Google Scholar 

  22. Pohl, M., Kubela, J., Bosse, S., Turowski, K.: Performance evaluation of application layer protocols for the internet-of-things. In: 2018 Sixth International Conference on Enterprise Systems (ES), pp. 180–187. IEEE (2018)

    Google Scholar 

  23. Profanter, S., Tekat, A., Dorofeev, K., Rickert, M., Knoll, A.: OPC UA versus ROS, DDS, and MQTT: performance evaluation of industry 4.0 protocols. In: Proceedings of the IEEE International Conference on Industrial Technology (ICIT) (2019)

    Google Scholar 

  24. Sachs, K., Appel, S., Kounev, S., Buchmann, A.: Benchmarking publish/subscribe-based messaging systems. In: Yoshikawa, M., Meng, X., Yumoto, T., Ma, Q., Sun, L., Watanabe, C. (eds.) DASFAA 2010. LNCS, vol. 6193, pp. 203–214. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14589-6_21

    Chapter  Google Scholar 

  25. Sachs, K., Kounev, S., Bacon, J., Buchmann, A.: Performance evaluation of message-oriented middleware using the SPECjms2007 benchmark. Perf. Eval. 66(8), 410–434 (2009)

    Article  Google Scholar 

  26. Shukla, A., Chaturvedi, S., Simmhan, Y.: RIoTBench: an IoT benchmark for distributed stream processing systems. Concurr. Comput. Pract. Exp. 29(21), e4257 (2017)

    Article  Google Scholar 

  27. Statista, IHS: Internet of Things - number of connected devices worldwide 2015–2025 (2018). https://www.statista.com/statistics/471264/iot-number-of-connected-devices-worldwide/

  28. Talaminos-Barroso, A., Estudillo-Valderrama, M.A., Roa, L.M., Reina-Tosina, J., Ortega-Ruiz, F.: A machine-to-machine protocol benchmark for ehealth applications - use case: respiratory rehabilitation. Comput. Methods Prog. Biomed. 129, 1–11 (2016)

    Article  Google Scholar 

  29. University of Wuerzburg, Institute of Computer Science, Germany, Chair of Software Engineering: Git repository of ComBench (2021). https://github.com/DescartesResearch/ComBench

  30. Wirawan, I.M., Wahyono, I.D., Idfi, G., Kusumo, G.R.: IoT communication system using publish-subscribe. In: 2018 International Seminar on Application for Technology of Information and Communication, pp. 61–65. IEEE (2018)

    Google Scholar 

  31. Zhang, H.L., Zhang, S., Li, X.J., Zhang, P., Liu, S.B.: Research of load testing and result application based on LoadRunner. In: 2012 National Conference on Information Technology and Computer Science. Atlantis Press (2012)

    Google Scholar 

  32. Zhang, K., Rabl, T., Sun, Y.P., Kumar, R., Zen, N., Jacobsen, H.A.: PSBench: a benchmark for content-and topic-based publish/subscribe systems. In: Proceedings of the Posters & Demos Session, pp. 17–18. Association for Computing Machinery (2014)

    Google Scholar 

Download references

Acknowledgement

This project is funded by the Bavarian State Ministry of Science and the Arts and coordinated by the Bavarian Research Institute for Digital Transformation (bidt).

Author information

Authors and Affiliations

  1. University of Wuerzburg, 97074, Wuerzburg, Germany

    Stefan Herrnleben, Maximilian Leidinger, Veronika Lesch, Thomas Prantl, Johannes Grohmann & Samuel Kounev

  2. University of Hohenheim, 70599, Stuttgart, Germany

    Christian Krupitzer

Authors
  1. Stefan Herrnleben
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maximilian Leidinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Veronika Lesch
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thomas Prantl
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Johannes Grohmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Christian Krupitzer
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Samuel Kounev
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan Herrnleben .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Qianchuan Zhao

  2. Sun Yat-sen University, Guangzhou, China

    Li Xia

Rights and permissions

Reprints and permissions

Copyright information

© 2021 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Herrnleben, S. et al. (2021). ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations. In: Zhao, Q., Xia, L. (eds) Performance Evaluation Methodologies and Tools. VALUETOOLS 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 404. Springer, Cham. https://doi.org/10.1007/978-3-030-92511-6_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-92511-6_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-92510-9

  • Online ISBN: 978-3-030-92511-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation