A Systematic Approach for Benchmarking of Container Orchestration Frameworks (Research Paper Artifact)

This record contains two supplementary artifacts for our ICPE 2023 research paper "A systematic approach for benchmarking of container orchestration frameworks". First, the source code of the presented benchmarking framework COFFEE is provided. Second, two VM images show the execution of COFFEE as a minimal example in a Kubernetes and a Nomad cluster, respectively. In the following, we describe each of these artifacts in more detail.

COFFEE Framework

We are interested in our benchmarking framework COFFEE being used and extended by other researchers. Therefore, we provide the source code including some documentation and utility files in the following GitHub repository:

https://github.com/DescartesResearch/COFFEE

This record contains the main branch of the Git project as of February 3, 2023. With the files in this repository, the source code and documentation of COFFEE can be examined. However, for most users, it might be challenging to run COFFEE from scratch in a meaningful way. COFFEE is a benchmarking framework of container orchestration frameworks. Consequently, the user would need to have access to a Kubernetes or Nomad cluster, as well as to a dedicated container image repository. To reduce this effort, to show that COFFEE is executable, and to provide minimal running examples, we provide VM images with COFFEE working with Kubernetes and Nomad.

VM Images

As mentioned in the previous section, we created two VM images to show basic executable examples for using COFFEE. One VM image contains COFFEE configured for a Kubernetes cluster; the other contains COFFEE configured for a Nomad cluster. For this, Kubernetes and Nomad, as well as all dependencies, are installed locally on these VMs. Also, the required container images are already installed on the VMs to eliminate the dependency on a container registry. For this purpose, the build scripts have been slightly modified compared to the version in the GitHub repository. Both VMs contain a README file in the user's home directory with instructions for execution. The example is configured so that COFFEE starts a test container in the cluster and measures its readiness time and removal time. The results are then exported as CSV and TXT files. The VM images use the OVA format.

In addition to this record, we created a CodeOcean capsule containing the paper's measurement data and processing scripts. This capsule can be found on https://doi.org/10.24433/CO.8875394.v3