Investigating the rendering capability of embedded devices for graphical-user-interfaces in mobile machines

2.1 Graphical-user-interface development

Two main branches of GUI development and deployment can be identified in the state-of-the-technology. A GUI is developed using web technologies like HTML or JavaScript and is interpreted by a web browser. Otherwise, many frameworks and libraries exist to build a GUI, which is compiled and executed directly on a hardware device. For completeness, it is mentioned that a more granular classification of GUI development methods is possible for mobile platforms, which use combinations of domain-specific languages for code generation, web technologies, interpreters and compilers [4]. Nevertheless, in mobile machines, embedded devices up to the industrial computer (IPC) are used as hardware platforms due to the high requirements for robustness, real-time capability and low power consumption. These hardware requirements are also reflected in the GUI development. The GUI must have a high degree of reliability due to safety reasons because the GUI is the primary interface between operator and machine. Therefore, GUIs for mobile machine applications are developed using GUI frameworks and are directly compiled and executed on the embedded device, which takes over the GUI rendering. The GUI visualization is realized via an external screen. A significant advantage of building a low-level GUI directly on the embedded device is that the GUI can be optimized during runtime and adapted to the hardware platform. However, GUI development is more time-consuming and changes in the GUI application are often more complicated, which further increases the importance of appropriate GUI design recommendations.

2.2 Benchmarks of embedded devices

Literature research is conducted about GUI and embedded device design concepts that provide detailed information about the capability of an embedded device to render a GUI and provide GUI design decisions for hardware target platforms. The literature review is limited to embedded devices and GUI toolkits for compilation according to Section 2.1. Both challenges in hardware-oriented GUI design are directly connected and are covered by application-based GUI benchmarks of embedded devices. Application-oriented GUI benchmarks of hardware devices offer the opportunity to compare the capability of embedded devices to render a reference GUI.

For embedded systems, the embedded microprocessor benchmark consortium (EEMBC) and standard performance evaluation corporation (SPEC) provide a variety of benchmark suites for the performance investigation [5]. For example, the EEMBC DENBench performance benchmark suite allows performance analysis of devices for multimedia tasks such as image compression and processing as well as audio encoding and decoding [6]. This benchmark suite deals with GUI related content and analyses other processes which are irrelevant to GUI applications in mobile machines (R1: variety-rich GUI). No benchmarks for variety-rich GUI applications are defined. Hormi [7] presents a GUI benchmark suite implemented using the Qt GUI toolkit for embedded systems that uses eight separate benchmark cases to classify an embedded system. Each test case includes a GUI process common in many applications. These test cases include, among other parts, the creation of scenes with three-dimensional objects, textures, resource-intensive GPU processes and multidimensional vector matrix operation. Some test cases can cover GUI in mobile machines, but three-dimensional animations are not standard (R1). The number of FPS is measured to describe the workload of the embedded device, which can be loaded in the full range using the test cases (R2: full range of workload). However, neither the source code nor the test cases are modeled and sufficiently documented to reuse the benchmark suite. In addition, the focus is exclusively on Qt applications, so it is questionable whether other cross-platform frameworks can also be used for GUI creation (R3: applicability). Besides the frame rate being calculated, the device workload is not further broken down into CPU and memory load (R4: interpretability). Another benchmark of embedded devices, motivated by [7], is presented by [8]. The benchmark also consists of individual test cases. Eleven test cases are designed to benchmark the rendering of typical Qt elements like rectangles, color gradients, opaque components and animations. These test cases include GUI components that are common, according to R1. The frame rate, CPU and memory load are measured so that the benchmarks provide modular and interpretable results (R4). Nevertheless, the test cases still need to be implemented or modeled, so the documentation and applicability are questionable (R3). Moreover, an evaluation of embedded devices still needs to be included. Whether these test cases will fully utilize embedded devices, is still being determined (R2). Although essential requirements are met, this work can only be considered as a first draft. The benchmark suite is challenging to apply without generalizable documentation and evaluation of the test cases. Furthermore, the Qt documentation already contains initial estimates of how resource-intensive the creation of some Qt elements is. Quantitative results for selected target devices are, however, essential to clarify the research questions (RQ1: GUI capability of devices) and (RQ2: GUI design recommendation).

In summary, first approaches for a GUI benchmark of embedded devices exist in the literature. Different GUI parts are benchmarked instead of only one reference GUI application. This modularity allows the first GUI design recommendations after the benchmark execution on different embedded systems. This leads to the central problem. Either the GUI test cases differ significantly from common GUI in mobile machines or they are neither documented, modeled nor evaluated. The state-of-the-art lacks a well-modeled, documented and evaluated benchmark suite for embedded systems usable for GUI developers in the mobile machine domain.

3.1 Design and execution

The benchmark concept is presented first, followed by a discussion on how the idea is aligned with the requirements and constraints. For the benchmark of the GUI rendering capability of embedded devices, nine GUI components are rendered individually and independently from each other. In Figure 1 (right) these nine GUI components are listed based on which nine benchmark test cases are specified. The starting point of the benchmark is that for each GUI component, the load on the device is increased by increasing the number of GUI components to be rendered in a specific time. The generated number of FPS shows the workload. When the frame rate drops below a certain threshold, the device is considered to be fully utilized. The number of objects, frame rate and CPU and memory performance metrics are tracked. The benchmark approach is modeled using UML activity diagrams for each test case. The user must initialize all variables (independent variables) that are declared for the test cases. The embedded device is connected to a screen and the test cases are executed. For the evaluation of the benchmark results, four findings (F) can be distinguished regarding the benchmark initialization and the upper limit of the generated frame rate. The findings are dependent on the implementation as well as the benchmark initialization and are discussed in detail in Section 4.1. In the following, the findings are discussed and some suggestions are submitted on how the GUI rendering capability of an embedded device can be compared for each finding.

Figure 1:

(Left) GUI, adapted from [9], of a large-scale mobile construction machine. The GUI (left) represents the upper limit of GUI complexity, that is addressed in this paper, and can be subdivided into separated GUI components (right).

Finding 1: The upper limit of the frame rate is narrowed by the technical constraints of the hardware setup (F1: FPS limit caused by hardware setup). The hardware (cable, display) used for the benchmark can also influence the upper limit of the frame rate. For example, the transmitted number of FPS for an HDMI 2.0 cable is limited to 60. If F1 occurs, the frame rate is expected to form a plateau at the beginning of the measurement because the embedded device can generate enough FPS. The frame rate drops as soon as the embedded device is too overloaded to hold that FPS plateau value. In the further process it is expected that the FPS will continue to fall until the critical minimum value is reached. Two different metrics can be used to compare the performance of several embedded devices with F1. At first, the number of objects rendered before the frame rate drops below this plateau can be compared for different embedded devices. An overall ranking of the GUI rendering capability is possible by averaging the number of rendered objects through all test cases or by selecting these GUI components and test cases close to a GUI application of interest. For example, if a GUI of interest contains mainly text elements and one-colored geometry, the number of objects generated for precisely these two test cases can be compared. Secondly, a frame rate threshold between the FPS plateau and the critical FPS value, which leads to the abortion of the test cases, can be used and the number of objects that are rendered at the frame rate of interest can be compared analogously.

Finding 2: The GUI performance on the device limits the maximum frame rate (F2: FPS limit caused by embedded device). This case can be seen as a subcase of F1. As soon as the frame rate drops from the plateau in F1, F2 occurs.

Finding 3: The device is immediately loaded so that the minimum number of frames that must be rendered is not reached (F3: embedded device immediately overloaded). F3 describes the case where the benchmark has been insufficiently initialized. The number of GUI objects that must be rendered in a specific time interval needs to be lowered and must be decreased. The benchmark must be reinitialized again.

Finding 4: The benchmark of a particular GUI component does not increase the embedded devices’ workload and the frame rate keeps constant (F4: no FPS decrease). In case of F4, an increase in the number of objects that must be rendered does not lead to a reduction in the frame rate. This can be caused, for example, by benchmark implementation issues. The benchmark of the certain GUI component must be rejected and the GUI capability must be described using other GUI component benchmarks.

In summary, for F1, F2 the frame rate allows a performance ranking of embedded devices for a GUI of interest and addresses (RQ1: GUI capability of devices). If F3 occurs, F1 or F2 can be induced by a reparameterization of the benchmark. GUI design recommendations can also be taken from the number of generated objects until the frame rate drops under a fixed threshold. Comparing the number of objects among all test cases enables to rank which GUI component loads a given embedded device and how much compared to other GUI components. A hardware performance-related GUI design recommendation can be made according to (RQ2: GUI design recommendation).

3.2 Discussion on constraints and requirements

The presented concept is designed according to R1–R4. The frame rate is used to operationalize the embedded device workload and (C1: FPS-measurement) is fulfilled per definition as far as the evaluation confirms that the frame rate is suitable for the device workload. (R1: variety-rich GUI) is considered because these nine GUI components can be extracted from extensive GUI applications for mobile machines. It is assumed that, if complex GUIs can be considered in the benchmark, less complex ones are also automatically covered. Less complex GUI applications can be seen as a subset of complex GUIs. A market study of standard GUI in mobile machines is performed to expose which typical properties a common GUI has. GUI for large-scale construction machines tends to be more extensive with multiple components compared to GUIs for lightweight, compact machines like vibration plates or rollers. A wide range of GUI in mobile machines is the state-of-technology. For the demonstration proposed in Figure 1 (left), the upper limit of a complex and extensive GUI is presented and the GUI components are assigned to the GUI by using roman numerals I–IX. The proposed GUI in Figure 1 (left) belongs to a large-scale diaphragm wall grab, a large-scale construction machine that is challenging to operate and can be composed entirely of the stated GUI components.

The benchmark is also designed according to (R2: full range of workload) because the workload of the embedded device is increased by an increase of objects to be rendered. (R3: applicability) is also taken into account because the test cases are separately modeled in UML activity diagrams with predeclared variables that the user must initialize. Therefore, the benchmark can be implemented in a straightforward manner. Nevertheless, expert knowledge is necessary for a proper benchmark initialization so that the embedded devices are loaded meaningfully. However, technical hardware specifications can give a first hint on initializing the benchmark. As a rule of thumb, the lower the CPU and GPU power, the lower the number of objects in each test case that can be rendered in a time span. For the implementation, framework-specific differences can also be taken into account and the test cases can be subdivided even more granularly. (R4: interpretability) is addressed as each test case is performed individually and some additional performance metrics of the embedded device are tracked.

3.3 Test case modeling

The benchmark test cases (i)–(ix) to load an embedded device are derived from the identified GUI components (I)–(IX) (Figure 1). The lower and upper case numbers are used to assign the GUI components to a test case. In Table 1 the GUI components and test cases are shown.

In Figure 2(a), the test cases (i)–(viii) are modeled. All test cases are modular and should be executed standalone successively. The device workload should be increased through the number of GUI components to be rendered. To distinguish the test cases (i)–(viii), they are defined as classes with names and attributes. The classes and attributes are described qualitatively and may differ depending on the implementation framework. The test cases (i), (iii), (iv), (vii) belong to related geometries. The benchmark is designed as simply as possible so that only one underlying geometrical object is considered, assuming all geometrical objects will lead to a comparable device workload with low deviations. The test cases start with rendering randomly positioned GUI components (class name). The frame rate, CPU and memory utilization are measured step by step according to the number of objects to be rendered. The test case is aborted if the frame rate repeatedly falls below a critical value from a minimum number of objects to be created. The list test case (ix) is described in Figure 2(b). Using several lists (IX) is unusual for GUIs in mobile machines. Therefore, the device workload will be increased by increasing the scroll speed through a list. It is scrolled through a list with a fixed number of values until the last element is reached. The frame rate, CPU and memory usage are measured. The scrolling speed is increased up to a certain limit. The test case ends when the critical run-through speed or the minimum frame rate is reached.

Figure 2:

Activity diagrams (UML) for the benchmark test cases (i)–(ix). The test cases (i)–(viii) are modeled in (a) and the list (ix) test case in (b).

4.1 Evaluation criteria and hardware setup

In general, the proposed benchmark approach to address the research questions is evaluated using a prototypical benchmark implementation and executing the benchmark on embedded devices. The authors are aware that these procedures only demonstrate the applicability for implementation in a common GUI development language and for the tested embedded devices. However, the benchmark design phase makes no requirements regarding a specific embedded device, GUI implementation or variable initialization. Therefore, it is assumed that the benchmark approach can also be applied to other embedded devices or GUI implementations if it delivers prototypically meaningful results. The evaluation of the proposed benchmark approach takes place in three stages. At first, the test cases from Section 3.3 are evaluated (criterion: Can the F1–F4 occur?). Only if this evaluation confirms the applicability of the overall concept, it can be analyzed using the quantitative results from the test cases (criterion: Are the recommendations from F1–F4 helpful for evaluating the GUI rendering capability of embedded devices?). The evaluation is limited to assessing the benchmark regarding the research questions (RQ1: GUI capability of devices) and (RQ2: GUI design recommendation). It does not focus on a detailed phenomenological analysis of the GUI capability of a specific embedded device.

Only the finding (F1: FPS limit caused by hardware setup) is evaluated in detail. (F2: FPS limit caused by embedded device) is a subset of F1 and already included in the evaluation. If finding (F4: no FPS decrease) occurs for one of the test cases, the other test cases must be assigned to F1 or F2. Otherwise, the benchmark can be considered insufficient in advance. (F3: embedded device immediately overloaded) does not have to be evaluated separately, provided that F1 or F2 confirm the benchmark approach. Supposed F1 or F2 can be achieved. In that case, it is demonstrated that a benchmark with proper initializations can point out the performance differences between embedded devices and the benchmark can be reinitialized in a meaningful manner as described in the recommendations for F3.

The hardware setup contains an embedded device connected to an external screen over an HDMI 2.0 cable. The benchmark is executed on an NVIDIA Jetson Nano, a Raspberry Pi 3B+, a BeagleBone AI and a BeagleBone Black. The upper limit of the number of FPS, that can be transmitted via HDMI 2.0, is 60. These hardware devices are selected because they are standard embedded systems and belong to different performance classes. All devices are set up with a graphical Debian operating system, which is also common for embedded devices. Based on the technical CPU and GPU specification in Table 2, the authors expect that the NVIDIA Jetson Nano has the best GUI-rendering performance, followed by the Raspberry Pi 3B+. The BeagleBone AI has a CPU and GPU which perform significantly worse than the NVIDIA Jetson Nano and Raspberry Pi 3B+. The BeagleBone Black is not equipped with a GPU, so the CPU takes over the rendering. Devices without GPU might have the lowest GUI performance. Based on the hardware specifications, it can be estimated that the devices belong to four different performance classes. For all devices, the benchmark software is initialized with the same setting. The authors know these initializations could lead to a high workload for the lower-performant devices. However, the same settings are used to discuss the prerequisites of the proposed benchmark and to induce F1–F4.

4.2 Test case implementation and initialization

(R3: applicability) requires that the benchmark approach is applicable to many devices. Of course, the benchmark should be implemented by applying a framework that is used close to real GUI applications of interest. However, a platform-independent GUI toolkit is suitable for the prototypical implementation and benchmark application. Therefore, the benchmark is prototypically implemented using Qt,^[1] a cross-platform framework designed for GUI development. Qt software is supported by nearly all standard operating systems and state-of-technology for embedded systems. In detail, the Qt Quick^[2] application is used that includes QML,^[3] a declarative Qt Meta-Object Language. Of course, without loss of generality, other cross-platform toolkits like Tk^[4] are also suitable for the implementation. The test cases in Figure 2 are analyzed in the context of their implementation in Qt. All test cases can be realized directly in Qt. For the RotatingGeometry (vii) test, two main branches of implementation exist. Therefore, the test case (vii) is extended to a RotatingGeometry test case using QML (viia) and JavaScript (viib). In summary, ten test cases (i)–(ix) with (viia) and (viib) are derived based on Figure 2.

The benchmarks are initialized using the rectangle as reference geometry for test cases (i), (iii), (iv), (viia,b). After each description, the labels (names) refer to the variables introduced in the UML diagrams in Figure 2. The minimum number of objects that must be rendered is set to 100 (objNum). The benchmark measurement is terminated if 10 FPS (minFPS) are undercut three times in a row (maxOverload). The CPU utilization [%] (cpuLoad) is tracked, as well as the virtual memory VM [MegaByte, MB] and memory residence RSS [MB] (memLoad). For test cases (i)–(viii) every 10 ms a GUI component is created (creationDelay) and the logging starts after 10 generated objects so that the workload logging is initiated again after 100 ms (logDelay). The list test case (ix) starts with a time delay of 100 ms until scrolling can continue (jumpDelay), which is deducted in 10 ms steps (delayDeduct). In Figure 3, an example of GUI components, that are rendered using the initialized benchmark, are presented.

Figure 3:

Exemplary presentation of the GUI test cases (i), (iii), (iv), which are initialized using rectangles. For each test case, rectangles are rendered and randomly placed on the screen with randomly selected sizes. The frame rate is measured and the number of rectangles is increased. In test case (i/iii/iv) the rectangles are unicolored/semi-transparent/multicolored.

4.3 Test case evaluation

The test case evaluation can be conducted if F1–F4 occur. The initialized benchmark executed on the four devices (Table 2) from different performance levels leads to F1–F4. For the NVIDIA Jetson Nano, (F1: FPS limit caused by hardware setup) and (F4: no FPS decrease) enter. The benchmark execution on the Raspberry Pi 3B+ leads to (F2: FPS limit caused by embedded device) and (F4: no FPS decrease), which is also expected due to the lower device performance in comparison to the NVIDIA Jetson Nano. The highest number of FPS for the Raspberry Pi 3B+ for all test cases (except list ix) is 30–40 FPS. The BeagleBone AI and BeagleBone Black are immediately fully utilized (F3: embedded device immediately overloaded). In the following, only NVIDIA Jetson Nano with (F1: FPS limit caused by hardware setup) and (F4: no FPS decrease) are discussed in detail and aligned to the evaluation method (Section 4.1).

The upper limit of 60 FPS can be reached for all test cases (60-FPS plateau in Figure 4) (F1: FPS limit caused by hardware setup). Furthermore, 60 FPS can be held despite an increase in the number of objects. If a critical number of objects is rendered, the frame rate drops significantly and decreases with further objects. The frame rate falls continuously (ii), (v), (viia) or stepwise (i), (iv), (vi), (viii). For test cases (viib), (iii), the frame rate does not decrease monotonously and individual stepwise rises can be identified. However, the frame rate trend decreases. The interaction of CPU and memory may cause this deviation. Test case list (ix) shows a constant frame rate and device workload independent of the number of list elements. Therefore, the test case list (ix) is unsuitable for a benchmark and excluded from further analyses (F4: no FPS decrease). In summary, test cases (i)–(viii) lead to the full range of workload, measured by using the frame rate.

Figure 4:

Benchmark results for NVIDIA Jetson Nano.

Furthermore, more insights into embedded device utilization can be extracted from the benchmark. The measured CPU utilization fluctuates in the range of 10 %. For the test cases (vi), (viia), (viib) the CPU load seems constant over the full range of generated objects. A linear decrease in CPU utilization can be identified for the tests (iii), (vi), (viii). A higher order polynomial decrease of the CPU load might occur for (i), (iv), (v). In summary, the CPU load tends to increase, which fits the increased embedded device workload. Nevertheless, the CPU load is not the limiting factor because the embedded device is utilized to a frame rate of 10 FPS, whereas the CPU load is not higher than 30–40 %. Memory utilization (RSS and VM) also increases for all test cases. In summary, CPU and memory do not limit the GUI rendering capability of an NVIDIA Jetson Nano since they need to be fully loaded. However, CPU, RSS and VM increase with a decrease in the frame rate and can give more hardware-applied programming hints for GUI developers.

To summarize, for most of the test cases the benchmark initialization is meaningful and it is demonstrated that the GUI rendering capability of the embedded device depends on the type and number of GUI components. The promising test case evaluation confirms that the test cases have been selected reasonably and that a sufficient initialization of the benchmark is in principle possible so that (F1: FPS limit caused by hardware setup) can enter.

4.4 Benchmark concept evaluation

After the confirmation that the findings F1–F4 occur, the recommendations discussed for F1–F4 (Section 3.1) are evaluated, the GUI rendering capability of embedded devices can be compared and GUI design recommendations can be provided. At first, it is analyzed whether the benchmarks allow a GUI rendering benchmark of embedded devices. Therefore, the benchmarks on the NVIDIA Jetson Nano and Raspberry Pi 3B+ are considered. The frame rate threshold is arbitrarily set to 10 FPS and the number of objects, that are rendered until the frame rate drops below that threshold, is counted in Table 3. The number of objects is compared for the NVIDIA Jetson Nano and Raspberry Pi 3B+.

In all test cases, the GUI rendering capability of the NVIDIA Jetson Nano is significantly higher than that of the Raspberry Pi 3B+. Depending on the test case and the object to be rendered, the difference in size is between a factor of 2 and a factor of 85. If an arbitrary GUI application contains many images and multi-colored geometries, an NVIDIA Jetson Nano is much better suited than a Raspberry Pi 3B+. Furthermore, the Raspberry Pi 3B+ does not reach the frame rate of 60 FPS. The lower frame rate is also aligned with the performance expectations based on the technical hardware documentation (see Table 2). Therefore, based on the benchmark, the NVIDIA Jetson Nano has a higher GUI performance than the Raspberry Pi 3B+. In summary, the benchmark can be used to highlight the performance differences between various embedded devices.

Secondly, it is investigated whether the benchmark enables GUI design recommendations. Therefore, the NVIDIA Jetson Nano and Raspberry Pi 3B+ are considered. To cover (F1: FPS limit caused by hardware setup), the FPS threshold for the NVIDIA Jetson Nano for rendered GUI component counting is set to 60 FPS, the upper limit of the HDMI 2.0 provided frame rate. (F2: FPS limit caused by embedded device) is achieved by the Raspberry Pi 3B+ with a 30 FPS threshold. For both devices, the number of rendered objects is counted and visualized in Figure 5.

Figure 5:

(Left) NVIDIA Jetson Nano: Rendered number of objects for each test case for a 60 FPS threshold. (Right) Raspberry Pi 3B+: Rendered number of objects for each test case for a 30 FPS threshold. The ranges of the y-axes vary for the left and right figure.

For the NVIDIA Jetson Nano, rendering rectangles and pictures tends to be low-resource consuming. The generation of text and rectangles with color gradients lead to a higher device workload, followed by the cluster containing animations, JavaScript and control elements. The rendering of windows and opaque rectangles shows the most increased workload. This ranking gives a first design decision, on which object type requires more or fewer hardware resources. To demonstrate that the FPS threshold helps to characterize the workload of each element compared to the other type of objects, the same analysis can be done for the Raspberry Pi 3B+. Creating text elements has the lowest workload, followed by windows and control buttons rendering. All other objects lead to a similar utilization of the embedded device.

In summary, the FPS threshold can be used to derive a first design decision for a GUI, which is executed on a given embedded device, and allows a weighing if it is essential to use an opaque rectangle instead of a standard rectangle. However, the NVIDIA Jetson Nano and Raspberry Pi 3B+ show a different workload based on the selected test cases. This investigation supports the central thesis of this work, that the embedded devices have to be benchmarked individually and that general statements about the renderability of individual objects cannot be made with low deviations.

4.5 Discussion on requirements, constraints and research questions

The benchmark approach is discussed according to the constraints and requirements. (C1: FPS measurement) is fulfilled because the frame rate generally decreases during an increase of the number of objects to be rendered. (R1: variety-rich GUI) is approximately fulfilled by design and nearly all selected GUI components lead to suitable test cases which load the embedded device. In further research, some industrial GUI applications could be considered. (R2: full range of workload) is also adequately addressed and the benchmark user can select an appropriate FPS threshold from the full range of performance workload. A suitable level of granularity can be proposed, so that individual GUI components can be benchmarked individually (R4: interpretability). Thus, this paper contributes a novel benchmark approach to the state-of-the-art. Table 4 margins the proposed benchmark from state-of-the-art.

The benchmark approach is furthermore a suitable method to answer the research questions.

Research question 1: Which commercial embedded devices can be used to render a GUI of interest sufficiently and how do these devices differ in their GUI rendering performance (RQ1: GUI capability of devices)?

Answer to RQ1: The proposed benchmark should be meaningfully initialized and should be executed on embedded devices. The number of rendered objects until the frame rate drops below an FPS-threshold should be counted. The GUI of interest should be analyzed regarding the type and amount of included GUI components that are listed in Figure 1. The higher the number of generated GUI components of interest, the higher the GUI capability of the embedded device.

Research question 2: Which GUI components should preferably be used to utilize a given embedded device only as much as necessary (RQ2: GUI design recommendation)?

Answer to RQ2: The benchmark is initialized and executed as described for RQ1. A comparison of number of objects that are rendered for a given FPS-threshold states which GUI components lead to a higher workload. In the following, the presented benchmark approach is discussed according to its advantages and obstacles. At first, the benchmark approach allows quantifying the embedded device workload in dependency of multiple scalable reference GUIs by measuring the frame rate. Several test cases are derived which have a high number of variables that the user must initialize. On the one hand, the benchmark is adaptable to many devices with different performance levels. On the other hand, it is impossible to initialize the benchmark, so that all devices can be compared. Expert knowledge is necessary for a meaningful initialization. Pre-classification of the devices must take place to initialize the benchmark adequately. In summary, the benchmark fulfills its purpose and makes it possible to compare the performance of devices and derive GUI design recommendations.