Keywords

1 Introduction

Technological developments of collaborative systems have increased demands on resources for evaluating virtual environments that allow multiple modes such as visualization, verbal communication and sound. Evaluating usability of interactive systems designed for Computer Supported Cooperative Work (CSCW) has been found challenging. Because of collaborating users, the evaluation requires more observers than in a single user scenario [1]. Increasingly, sound plays a large role in virtual environments in the form of natural or synthetic sound, music or voice communication [2]. Evaluation of the effect of sound on voice communication has been carried out but mostly in controlled laboratory settings [3].

To gain valid results of a usability evaluation, method selection continues to be a critical activity. The increased complexity of systems has called for ways to choose methods that are most appropriate for usability evaluations. One such framework has been proposed by Antunes et al. [4] who devised a selection strategy for choosing an appropriate method of evaluation. Unfortunately, empirical research on usability evaluation of collaborative virtual environment has not been extensive [4]. While evaluations of collaborative environments have been mostly studied theoretically [5], studies have been reported on the effectiveness of 3D collaborative virtual environments [6]. Recognizing that multiple dimensions of interactive systems call for more than an adoption of one or two evaluation methods, it has been suggested that a collection of resources, that are a part of a whole method, return an efficient evaluation [7]. Examples of resources of methods are participant recruitment, task scenario, reporting format, problem identification, problem classification and a thinking-aloud protocol [7]. This paper uses the term method-resources to describe such resources [8].

Thus motivated, the main objective of this research is to gain empirical knowledge on the suitability of resources for usability evaluation of a collaborative system, set in a virtual environment, where players communicate verbally in a noisy environment. To achieve this objective, we studied a usability evaluation of a prototype virtual environment for training crisis management personnel dealing with mass-casualty accidents.

Crisis management training is organized and developed according to an accurate predefined system for immediate responses to mass-casualty incidents. First responders are personnel from volunteers to professionals with a few to many years of experience. Professional responders come from different organizations, e.g., rescue, police, medical and firefighting. Training these people to obtain efficient skills is crucial.

The application (see Fig. 1) that was evaluated is a virtual training simulator for crisis response to a mass-casualty incident at an airport. Trainees are commanders, the On Scene Commander (OSC) managing the responses overall on the scene and the Rescue Coordinator (RC) managing the rescue work on the scene that exercise progress report, rescue resources, information requests and task delegation. They can navigate with an on scene view and a zooming in effect through the virtual environment that has a changing high-fidelity scene. They are wearing earphones with a microphone. A design based on empirical data consisted of three voice communication metaphors reflecting the communication spaces used in a real crisis event and its training, i.e. one way radio, mobile phone and face-to-face (F2F) communication for the two persons to speak to one another [9]. To broadcast a message a GUI button is pressed with a mouse (i.e. Push-to-talk) and in response, a transfer is reversed from receiving to sending with the player microphone no longer muted. The prototype has a high fidelity soundscape with sound from fire trucks, other resources, players talking, fire and wind. Any type of sound is relayed over communication channels, i.e. players can hear noises at the far end through the channel.

Fig. 1.
figure 1

A snapshot of the virtual environment showing the scene of the accident.

Full size image

2 Selection of Method-Resources

2.1 Evaluation Methods

To select an evaluation method we applied the CSCW evaluation framework by Antunes et al. [4], where a variety of evaluation methods are presented for each stage of the software development. We selected Groupware Observational User Testing (GOT) as an end-user oriented evaluation method focusing on realism and usability as the main objective [4, 10]. Gutwin and Greenberg [10] proposed GOT, as a cost effective usability method, which was based on a set of fundamental Mechanics Of Collaboration (MOC). The GOT technique is an observational user testing method focusing on usability in a planned situation, collaboration where users perform predefined tasks. The framework of MOC includes seven categories of important collaboration activities: Communication as Explicit communication and Consequential communication (information unintentionally given off by others); Coordination of action; Planning; Monitoring and gathering information in the workspace; Assistance to one another; and Protection of resources in the workspace [10, 11]. The MOC model has been evolving [12], but the original set was more appropriate for our study. Besides MOC, the method-resources needed for GOT are think-aloud, observers, users and tasks. Furthermore, to record observations, screen-captures were used and audio recorded of the communication in the virtual environment and of the think-aloud. The tasks and the users are described in the next section.

2.2 Collaborative Scenario and Users

Based on extensive observations of crisis management training on-site, interviews and workshops, a collaborative scenario comprising several tasks was written and validated by an experienced crisis management instructor. The scenario aimed to secure a situation at an accident scene and allowing commanders to ask for resources, such as fire fighters, using verbal communication (see Table 1). Each commander was located in a separate room in front of a screen wearing a head-set with a microphone.

Table 1. Tasks in a collaborative scenario for OSC and RC.
Full size table

Six employees of a rescue and fire organisation at an airport with experience in crisis response were recruited as participants for the study and divided into three pairs of OSC and RC. Participants had all received a one day introduction to training in a virtual environment, but not to the prototype used in this study. The same session was repeated three times, once for each pair of collaborators which were followed by an observer.

2.3 Analysis of Usability Problems and Protocol Analysis According to MOC

The data was analysed in two ways, analysing usability issues and collaborations using MOC. A bottom up qualitative analysis was performed identifying usability issues that were consolidated into unique usability problems. Before analysing the data, the second author transcribed the audio data into text while listening to it and observing the video capture. Comments that observers (the first and third authors) had written down during the sessions were integrated to the transcript protocol. The third author analysed the protocol for problems that participants faced that were then verified by the first author. In addition to problems, an observer looked for successful interactions, activities and comments raised by the participants. After analysing the protocol, the observations were categorised into groups emerging from the data. Reviewing the transcribed conversations and the videos of the screen capture, the second author and an engineer analysed the data using MOC. After analysing them independently, they discussed differences and came to a consensus. Additional method-resources are observers with expertise in human-computer interaction and moderate expertise in the domain of crisis management.

3 Results

The experience gained from applying the method-resources will help analyse their usefulness. The sessions lasted 17–20 min each, which gave 109 min of transcribed audio and video recordings. The number of conversations per pair was 10–12, or a total of 32 for all sessions. Observations, capturing problems or successful interactions, e.g. successful training, were 112.

The data analysis uncovered 13 unique usability problems of 84 problem instances divided into eight categories, communication, information/communication, collaboration, navigation, discrepancies between virtual and real world, sound, wrong or inappropriate tasks and following scripts. The first three problem categories of communication and collaboration are discussed in this paper.

Altogether 71 collaboration instances from the conversations of the three pairs of participants were analysed with respect to the MOC categories (see Fig. 2). It is noteworthy that the players seemed to monitor their environment and the situation extensively, learning where people are and what they are doing. That no assistance took place could be attributed to low complexity of the scenario and that the players saw no need to protect their workspaces indicates that there were no threats imposed on them in the environment. To analyse the collaboration protocol according to MOC an exact transcript of the collaboration was required.

Fig. 2.
figure 2

Collaborations analysed using MOC.

Full size image

The method-resources proved to be helpful for uncovering issues and usability problems of categories addressed here. Table 2 shows examples of how the method-resources were used to uncover particular issues. The issues in the table are labelled with issue IDs (e.g. A01) which are used in the following text and the numbers of the MOC categories are identified according to Fig. 2.

Table 2. Examples of use of method-resources.
Full size table

To show how the different method-resources have helped, a few examples of the collaboration protocol are given. In several cases, we noticed that the trainees were able to practice skills successfully. One of them is a coordination mechanic (A01):

OSC: [Thinking aloud] I will meet him. The wind actually is like that. I will call. Oops. I will go and meet him like, somewhere here. Not too close to the fire though. I would meet him around here.

The Think-aloud protocol helped the observer to gain insight into what the user is thinking and discovered that he was practicing skills successfully. Such a scenario could not have been practiced without expert users in the domain.

In two of the three experiments, it took participants some time to realise that they should collaborate and take turns, but after a short while they got used to it (A02). Four usability problem instances were observed affecting two participants:

OSC: [Thinking-aloud] So, Do I then… – do I wait for him to call a backup team or help. Or do I? Is that my decision, or does he…

The current method-resources make it difficult to conclude the nature of this problem and the cause of the confusion. It could be that the commander is unsure of the collaborative scenario, that the virtual environment does not provide adequate affordance, or even that he has not been trained adequately in his role. An additional method-resource would be needed to inquire about certain critical points.

Some problems were about providing information verbally. Users relayed wrong information to their partners, or missed to respond when being addressed, either altogether or not responding accurately (A03):

OSC: “RC we have a plane crash on runway 19 the intersection. Plane is on fire. We have 35 people on board. It is a mini-jumbo jet. You go on scene with your team.”

RC: “RC got that. A plane crashed down on fire, an intersection 11. No 19 and 01. And 38 people on board” “What type of aircraft is that?”

The participant acknowledged inaccurately and wrongly, or he did not hear correctly the number of people on board, i.e., 38 instead of 35. The video was checked to confirm conditions that showed good sound and no noise disturbing. Five participants had such a problem in a total of 15 instances. It is not analysed as a usability problem and it may even reflect an accurate picture of a normal training scenario. Another thing we noticed is that the expert users in the domain were able to play their role and act out from the given scenario (“What type of aircraft is that?”), thus indicating that the virtual environment is a useful training tool. The MOC analysis proved useful in separating the explicit communication issues from others. The advanced collaborative scenario and expert users in the domain are vital method-resources to create a dialogue and reveal such a scenario.

It was observed that OSC talked while navigating to the gate in the virtual environment and kept the radio channel open, allowing RC to hear what he said (A04, A05, A06, and A07). RC told the observer that he could hear everything OSC said (A08). Later, OSC resolved how to use the radio and called his partner successfully by identifying himself and the receiver (A09). We noticed that participants used the phone much less but encountered similar problems. Two of the three pairs tried to talk F2F, one of them took a few minutes to make it work smoothly but a second pair used it without problems. Expertise of the users in using communication devices in crisis management, e.g. radios, was crucial to understand how they used the communication metaphors in the virtual environment. Observers are expensive resources for evaluating how much certain features are used and could be replaced with automatic monitors. Finally, we see in this example, when noting that the OSC talked on his way to the gate, that it is essential to have a screen-capture of the experiment.

The GOT method and its resources worked well for focusing on usability evaluation in a planned situation, especially when focusing on collaboration and users performing particular predefined tasks. The factors of collaboration and communication were the focus of this study and using the MOC analysis of collaborations fits well for them. The examples show that observation, transcripts and the think-aloud technique are fundamental in researching verbal communication in a collaborative environment. Experts in the domain and the task scenarios are used to uncover fewer issues, but are nonetheless essential. The captured voice is used to produce exact transcript of the collaboration, but as Table 2 shows the sound and the video is used less to understand the issues.

4 Conclusion

The contribution of the paper is twofold. First, we have shown a method for analysing the usefulness of method-resources. Such a method can be useful for other researchers analysing method-resources. Second, its application shows that the GOT method and the MOC analysis are appropriate for analysing factors of collaboration and communication. An essential part of that is to include the think-aloud technique and observers. In only a few cases sound and video were necessary to analyse issues. Other resources that were especially important were expert users in the domain and advanced task scenario.

A few ideas emerged for decreasing the cost of method-resources and raising their effectiveness. Developing software tools for monitoring the scenario, e.g. the frequency of use of features could help decrease the expenses of observation in the collaborative scenario and data analysis. In an environment where the dialogue is rich, the domain is complex and user domain expertise is high, it may be more difficult than ever to understand the causes of users’ actions. A method-resource to analyse critical points of understanding causes may be needed.