Benchmarking and Performance Analysis for Distributed Cache Systems: A Comparative Case Study

Abstract

Caching critical pieces of information in memory or local hard drive is important for applications’ performance. Critical pieces of information could include, for example, information returned from I/O-intensive queries or computationally-intensive calculations. Apart from such, storing large amounts of data in a single memory is expensive and sometimes infeasible. Distributed cache systems come to offer faster access by exploiting the memory of more than one machine but they appear as one logical large cache. Therefore, analyzing and benchmarking these systems are necessary to study what and how factors, such as number of clients and data sizes, affect the performance. The majority of current benchmarks deal with the number of clients as “multiple-threads but all over one client connection”; this does not reflect the real scenarios where each thread has its own connection. This paper considered several benchmarking mechanisms and selected one for performance analysis. It also studied the performance of two popular open source distributed cache systems (Hazelcast and Infinispan). Using the selected benchmarking mechanism, results show that the performance of distributed cache systems is significantly affected by the number of concurrent clients accessing the distributed cache as well as by the size of the data managed by the cache. Furthermore, the conducted performance analysis shows that Infinispan outperforms Hazelcast in the simple data retrieval scenarios as well as most SQL-like queries scenarios, whereas Hazelcast outperforms Infinispan in SQL-like queries for small data sizes.