Interactive Narrative in Augmented Reality: An Extended Reality of the Holocaust

Abstract

In this research, the author descripted new narrative media known as Immersive Augmented Reality Environment (IARE) with HoloLens. Aarseth’s narrative model [17] and all available input design in IARE were reviewed and summarised. Based on these findings, The AR Journey, a HoloLens app aiming at interactive narrative for moral education purpose, was developed and assessed. Qualitative methods of interview and observation were used and the results were analysed. In general, narrative in IARE were proved to be valid for moral education purpose, and findings including valid narrative structure, input model, design guidelines were revealed.

1 Introduction

Stories can be told in various media such as books, drama, films, video games and Virtual Reality (VR). Video games are powerful as they provide an active, exploratory, multi-sensory process for participants, while VR is also noticeable for its immersive experience of presence and sensory vividness [1]. With the invention of HoloLens, a Head Mounted Display (HMD) for Augmented Reality (AR), a new form and media for narratives has emerged. This new AR device differs from the previous hand-held devices (e.g. mobile phone) or VR because it overwhelms participants sense by filling the space with virtual holograms and spatial sound through the HMD, but still keeping them connected with the real world. HoloLens can be considered as a new media which creates the unique Immersive Augmented Reality Environment (IARE) where virtually holographic characters and objects can be mixed into real-world space seamlessly while participants observe, walk and interact with things in the real-world environment. IARE enables people to navigate and interact with a physical space freely while watching a performance of holographic characters. Unfortunately, existing researches mainly focused on practical usage of AR with HoloLens, such as architectures and automobile design, medical training, touring guide, etc. [2, 3]. There is limited research on how to use IARE for storytelling. Though a research from George Tech University suggested that mixing character into real-world environment had no improvement for narrative [4], the technology they used didn’t create an authentic IARE, because they just overlaid a flat 2D character instead of mixing a holographic 3D character in a physical space. Hence the authors plan to investigate the potentials of narrative in IARE based on HoloLens and assess the effectiveness of this narrative.

2 Literature Review

2.1 Challenges of Narrative in Museums for Moral Education

Narratives are proved to be a powerful method for empathy such as perspective taking and emotional engagement [5]. Many researchers claim positive association between empathy and prosocial behaviors [6, 7]. Jin et al. [8] summarised empirical evidences that prosocial video games are beneficial for youth’s moral development. All these findings suggest narrative is valid and powerful for moral education.

The National Holocaust Centre and Museum (NHCM) is one of the few museums employing the narrative technique to unveil the history and enable young generations to carefully examine, commemorate and learn from the tragedy of the Holocaust. The Journey, one of its permanent exhibitions, tells the story using environment storytelling technique through the eyes of a fictional Jewish boy Leo who survived the Holocaust and came to the UK via the Kindertransport¹. Six rooms are restored to show how Jewish’s life look like including Leo’s family living room, Leo’s classroom in his school, the street after Night of Broken Glass, the tailor’s shop of Leo’s family, a train carriage for Kindertransport and refuge in the UK. In each room, audience can watch a short video of Leo giving a monologue of what he saw, heard, experienced and felt at that time. The visitors can experience the complete story gradually by going through each room, interacting with objects and watching videos. The Journey is a text free and tactile exhibition, designed mainly for young audience. However, most visitors experienced the Journey as part of their visit.

The major challenges of the Journey exhibition are identified according to the NHCM website [8], literature [9] and the authors’ observation on site:

• Inclusiveness: The Journey is originally designed as a group visit experience for young audience with a tour guide, thus storytelling may be plain and shallow for individual adult visitors.

• Accessibility: the NHCM is far from the downtown area of a small city with limited public transportation. It is difficult to make the learning experience accessible in the widest range of places and formats, to continually reach new audiences and provoke attitudinal change across all communities.

• Attractiveness: as younger generations prefer modern interactive methods derived from their evolving personal technologies [10], storytelling via interactive digital technologies such as video games, VR and AR would be more effective and appealing.

2.2 Recent Advance of AR/MR HMD

Hololens 1 was released at the end of 2016 offering a solution for fully-immersive experience mixing virtuality and reality. Hololens is an HMD, featuring a 35° viewing angle see-through holographic lenses (waveguides), spatial understanding by real-time 3D scanning, gaze tracking, hand gestures input, voice recognition and built-in speakers for spatial sound [11]. Hololens 1 can partly understand and recognise real world information such as walls, floors, ceilings, chairs and put a stable hologram into the real-world space, e.g. audience can watch a holographic virtual character “seating” on a real-world sofa talking to them.

In July 2018, a similar AR-HMD device named Magic Leap One was launched in the market. It is equipped with a LCOS screen with a higher definition of 1280 × 960, offering a wider viewing angle of 50°, larger RAM of 8 GB and better CPU [12]. It also has the several functions similar to HoloLens 1 and the additional eye tracking function. However, reviewers discovered that Magic Leap One achieved higher Field-of-View (FOV) by sacrificing image resolution and brightness, which made text cloudy and the virtual image darker.

In February 2019, HoloLens 2 was released as the most advanced AR/MR device on the market. The HoloLens 2 catches up in terms of FOV with a 52° viewing angle and screen resolution of 2K per eye [13]. Moreover, HoloLens 2 stands out with a much improved hand-tracking technology, eye tracking, voice recognition and better ergonomic design. Users can directly manipulate virtual hologram by hand (without any symbolic hand gestures) and perform dictation recognition offline. HoloLens 2 outperforms Magic Leap One on almost every aspect except it is heavier.

In addition, Microsoft also released specific software development kit known as Mixed Reality Toolkit (MRTK) [14] while other companies like HP Inc. released their own immersive AR headset with the motion controllers in the past two years.

In summary, the hardware and software development kit are keeping improving during the past five years. Though there are still limitations which affect audience’s immersive experience, such as narrow FOV, limited CPU computing power, great enhancement has achieved in terms of display quality, input methods, software development modules, which makes narrative in IARE using real-time 3D characters possible. Currently, HoloLens 2 is not available yet in the consumer market. Hololens 1 is still the best available choice for research and study purpose considering better development environment of MRTK and future migration to HoloLens 2.

2.3 Related Research on Narrative in AR

There are two distinct properties of HMD AR as well as HMD VR compared with other medias. Presence is one of them, which refers to a subjective user experience of being into the virtual or mixed-reality environment rather than the actual physical locale [15]. Different from the flat screen, HMD is a binocular device which can not only help user perceive the accurate size of an object, but also cover the large part of user’s vision to generate an overwhelming sensory feeling.

The other feature is agency, which refers to the ability to “do something” in the experience—to interact or react rather than simply perceive. Agency is the core of VR/AR experience because virtual environment (VE) within headset gives the audience the native agency of looking [16]. In other words, IARE has the built-in nature of interaction as audience would like to have more ability to interact with the environment rather than looking.

Regarding the feature of presence and agency, one important feature of narrative in IARE is real time. Real time here doesn’t indicate the same concept of video games, which means to update the content 60 times per second. Real time in this case addresses that the time within the story is contingent, e.g. the enactment of an actor should be in accordance with the real world so that it can speed up or slow down. Real time is in fact incompatible with certain narrative forms such as literature or film. Because in film, the performance of an actor can easily accelerate or decelerate via editing and montage. R. Aylett and S. Louchart claimed real time as an important feature in VR in their research [17]. Though they had a different interpretation of real time as “From an authorial point of view, it would imply the author writing, telling and displaying the story at the same time as the reader is reading or viewing it.”, they had an insight for the influence of real time in narrative:

• It could bring certain constraints on the dramatic intensity of any narrative.

• The theatre’s typical episodic structure and the concept of ‘off-stage’ activity can be used to produce some sense of temporal and spatial richness.

• A narrative in real time must be either multiple, interactive or exceptionally rich in dramatic features.

• Participative forms of narrative such as video games, Interactive and Improvisational theatre (IT/IMPROV), can enlighten the potential approach and methods for storytelling in VR.

Due to the lack of theory and barrier of the technical issues, there are few attempts made for narrative based on HoloLens. Fortunately, Fragments, a suspense & adventure narrative experience of HoloLens developed by Asobo Studio, is a good exploration and many people reported positive feedback about it. The success of Fragments further revealed potential for narratives in IARE.

Besides, IARE is probably effective for narratives of serious purpose rather than suspense or adventure types if the above three issues of narrative in museum (in Sect. 2.1) are taken into consideration. IARE can involve hologram of virtual characters that enable the richness and more possibilities of a profound storytelling suitable for adults. Further, IARE can also adapt easily to a new real-world space and projects the virtual characters, furniture and object into the new one. Lastly, the experience of narrative in IARE aligns with the experience of video games and immersive theatre, which is affable and approachable for young adults. However, the valid narrative structure and input design for the narrative in IARE aiming at moral education remains unclear.

2.4 Research Questions

As Sect. 2.2 summrises that the computing power of AR device is limited for now, the vividness of hologram can’t be guaranteed. Section 2.3 makes a hypothesis that narrative in IARE could be valid for moral education purpose and further points out the narrative structure needs to be investigated as well as the input design. Therefore, the author proposes the following Research Questions (RQ):

• RQ1: How much dose the vividness of character in IARE affect audience’s perception?

• RQ2: What’s the valid narrative structure for narrative in IARE in terms of moral education?

• RQ3: What’s the valid input design for narrative in IARE?

• RQ4: Are narratives in IARE engaging for audience?

• RQ5: What are the design guidelines for narrative in IARE to achieve moral education goals?

In order to answer the above questions, an interactive narrative application on HoloLens named The AR Journey was scheduled to be developed by the author for the NHCM. It aimed at Holocaust education by telling the story of Jewish boy Leo through an augmented reality experience.

3 Design of The AR Journey

The design process started with a series of observations (Stage 1) to investigate the behavioural traits of users wearing HoloLens for a storytelling experience. This process revealed users’ common behavior and feedback in IARE with HoloLens. Based on these information, the second stage explored Aarseth’s narrative model for games and proposed a modified “quest game” structure for narrative in IARE. The third stage concluded current input types and input models for IARE and selected two input models among them. Visual content design and programming were then completed and the alpha version was published (Stage 4). In the final stage, The AR Journey was tested in NHCM and later evaluated (Stage 5) by a small group of experts, and the app was revised according to their feedbacks to be ready for final experiment.

3.1 Stage 1: Preliminary Study

User observation: The observational analysis revealed the general behavioural patterns of the user who were invited to experience the HoloLens suspense & adventure game, Fragments.

• Pattern 1: most users felt uncomfortable physically when wearing HoloLens 1^st headset over 20 min.

• Pattern 2: many users had difficulty learning and implementing the air tap gesture, the HoloLens input gesture.

• Pattern 3: most users tended to stand still when watching the enactment of characters.

• Pattern 4: most users reported the narrow FOV issue was a distraction.

These behavioural patterns form the initial guideline of The AR Journey.

3.2 Stage 2: The AR Journey Narrative Structure

Following the suggestion of examining narrative model for games (Sect. 2.3), Aarseth’s model [18], which is grounded on Chatman’s concept of kernels and satellites² [19], is interesting. As Table 1 showed, he clarified the difference between linear story, nonlinear story, linear game, quest game and “pure” game. As the purpose of The AR Journey is moral education and the story of a Leo’s family involved in Kindertransport should be conveyed clearly, audience aren’t allowed to have the full influence on the kernel of the story, which means the “Pure” game type is excluded. On the other hand, in order to improve the inclusiveness of the experience, audience should have some limited influence on the story and some constitutive events can then contribute to the diversity and depth of the narrative experience. Therefore, quest game type is the idea choice for The AR Journey, which allows alternative branch of the kernel and possible supplementary events (satellites).

Table 1. Classification of narratives based on audience’s influence on kernels and satellites

Coincidentally, IARE has the built-in nature that allows the user walk within a real world space and touch, interact with real world props and objects where satellites can be embed. Additional stories and information can be revealed and unfolded when users explore the room. By this way, audience can keep moving instead of standing still all the time (Pattern 1 in preliminary study).

In summary, as Fig. 1 shows, the narrative structure of The AR Journey consists of a kernel, which employs a branched narrative structure, and several satellites. Storytelling of the kernel part or satellites part are triggered by props in Leo’s living room. The diary is the key prop that can activate the kernel part while other props such as radio, gramophone, suitcase, newspaper, etc. can launch the satellites part. Audience are able to perceive the story by exploring different props and make different choice leading to different branch of the story. As a result, using this narrative structure, different audience can have their own path to experience the story to get their own understanding and reflection.

Fig. 1.

Narrative structure of The AR Journey

3.3 Stage 3: The AR Journey Interaction Design

As described in the last section, all storytelling is triggered by audience through interaction and audience need to choose branches of the story via an input interface. Thus, interaction design is important not only for natural experience of the story, but also for suspension of disbelief³ (Fig. 2). Presence in IARE, which shares the similar concept of suspension of disbelief, refers to experiencing the mixture of the virtual environment and real world, even rather than the actual physical locale. Witmer [15] found that responsive and controllable interaction is critical to produce presence in a virtual environment. As a result, simple interactions with straightforward feedback like turning on/off a virtual light, opening/closing a virtual drawer, pulling/closing a curtain, can suspend audience’s disbelief. It is not necessary to trigger an event or introduce a story piece after every interaction. Meanwhile, interactions with complex feedback, such as determining the story branch, triggering a voice record, reading a diary, can help audience understand the narrative when they put together all the story fragments.

Fig. 2.

Interaction design in IARE (Solid line is the path adopted by The AR Journey)

Secondly, it is essential to design secondary interactions that can generate (unlock) new input targets to trigger subsequent interactions (Fig. 2), e.g. one opens a drawer and then find a diary in the drawer; after the audience opens the diary, a postcard slips on the floor; the audience picks up the postcard to read words on it. In this case, picking up the postcard is the secondary interaction of opening the diary, which is also the secondary interaction of pulling the drawer. Significantly, secondary interaction in IARE can make audience feel the environment more convincible as it is exactly the same way in the real world, this is different from many user interaction design concepts which tend to feedback the information as direct as possible.

Thirdly, audience communicate within IARE through input on input targets and output. It is important to recongise that input targets in IARE could be real world targets as well as virtual world targets, and so are the output. Besides, input on virtual input targets can output real world feedback, and vice versa, e.g. turning off the light in real world can change the lighting on virtual characters, throwing the virtual switch can turn on the floor lamp in real world. By this way, the boundary between virtual world and real world can be further blurred. However, in order to center on the research questions and ensure the accessibility of The AR Journey, this study focus on input on virtual targets to get output in virtual world.

Lastly, the input, which consists of input type and input model, is analysed. The input type refers to the fundamental genre of input in IARE which are gaze, manipulate/point/commit, and voice command. Audience can perform the same type using different apparatus, e.g. action of gaze can be performed via head or eye, action of pointing and manipulation can be executed via hand or controllers. Table 2 compares the merit and limitation of different apparatus aiming at same input type. In summary, it reveals that eye gaze and head gaze are different. Eye gaze is more implicit, inaccurate, faster, easier, and usually used as an alternative input channel while head gaze is more accurate, reliable but slower and discomfort. For manipulation/point/commit, there are several possible apparatus including direct manipulation by hand, symbolic hand gestures, and different controllers (the motion controllers⁴, the HoloLens clicker⁵, the Xbox controller, etc.) Direct manipulation by hand, which is only supported for HoloLens 2, is the ideal choice as it is nature, institutive to use and consistent with the real world manipulation [20]. The motion controllers are more precise and stable with tactical feedback but draw a clear line between virtuality and reality [21]. Hand gestures are not recommended in most cases as they are inconvenient to learn and remember, and fatigue users easily [22]. Voice command has great potential as it is hand-free, natural and low effort but unreliable at present, especially in non-English or noisy environment [23].

Table 2. Comparison of different input types and their apparatus

It is important to understand that different input types can be combined or used alone with their own conventions to form an input model, which are listed as followings [25]:

• Direct manipulation is an input model that involves touching holograms/real world objects directly with one’s hands or controllers. (Hololens 2 only)

• Point and commit is a ‘far’ input model that enables users to target, select and manipulate 2D content and 3D objects that are out of reach using hand gestures or controllers.

• Gaze and commit is a far input model using eye/head gaze and commit action via hand gesture, controllers or voice command.

• Gaze and dwell is a hand free input model. The user keeps gazing (with their head or eyes) at the target and lingers there for a moment to activate it.

• Voice input is a hand free input model by using one’s voice to command and control an interface, e.g. the user can read a button’s name out to activate it.

As HoloLens 2 is still not available in the consumer market and the motion controllers are too large to eliminate the gap between virtual world and real world, input models of direct manipulation and point and commit are filtered out. Eye gaze is also filtered out because it is only supported by HoloLens 2. Voice input is rejected as the Chinese voice input is poorly supported and hand gestures have proved problematic in our preliminary study. In summary, there are two path of interaction design left for the AR Journey, which are head gaze and commit with the clicker and head gaze and dwell (Fig. 2). Both interactions aim at virtual targets and lead to output in virtual world. To make the environment more convincible subjectively, some secondary interactions and interactions only with straightforward feedback are supposed to be included.

3.4 Stage 4: The AR Journey Development Process

Script Development

The script and dialogue were rewritten based on an existing script from a parallel project of the virtual journey app [26] in collaboration with experts from the NHCM to ensure historical accuracy. The dialogues and plots are designed based on survivors’ testimonies and facts of the past. In light of the above narrative structure and interaction design, the story focused on a debate between Leo’s parents happened in the living room. The main story employed a branched structure which raised three questions to the audience who can make choices for the protagonist Leo. The story can be triggered by opening Leo’s diary, while some fragments, such as isolation in school, dad’s favorite music, were hidden in props in the room including a gramophone, newspaper, telephone, radio, etc.

Asset Development

4 characters, approximate 15 min’ character animation, 11 props, and around 20 UI elements were created for the AR app. The visual asset development followed the same rule of historical accuracy as the script development. Most references came from the NHCM and the online United States Holocaust Memorial Museum. Character modelling and animation were the key part of this project. 3D software including Zbrush, 3ds Max, Substance Painter, Marvelous Designer were used for character modelling, texturing, UV unwrapping, cloth simulation and rigging. All models were tested and refined in Unity 3D engine with optimised material for mobile using texture baking (Fig. 3).

Fig. 3.

Procedures of character design for Leo

Software Development

The programing was developed in Unity 3D game engine. The main challenge for programming was the character’s natural and autonomous behaviour and the HoloLens’ input module. To enable the audience to communicate with Leo using gaze, gesture and voice, Leo needs natural eye contact, face to face conversation, and some social intelligence, e.g. to respond, look at and walk towards the user. In order to achieve the autonomous response, characters were based on program-controlled motion clips with Inverse Kinematics (IK). There are also many pre-defined keyframe animations. Thus, how to make seamless blending between the above two animation systems was the key. The main technique here is to use the Playable module of Unity 3D for seamless blending of character animations. Each pre-defined keyframe animation and program-controlled motion clip is a playable note and could be connected via a mixer note. The strength of each channel of the mixer could be adjusted dynamically in real time. There are two timeline playables in this design in order to ensure a smooth blend between different pre-defined animations. A special Animation Layer Mixer is used here to have a separate control of upper body and expression known as an avatar mask.

Moreover, a gaze and dwell module was developed, since it is not built-in for HoloLens 1. Gaze and commit with a clicker is a built-in module. Consequently, the audience can interact with virtual characters and objects either by gaze and dwell or gaze and click.

3.5 Stage 5: Quick Assessment and Iteration

When the development was done, it was tested in the NHCM (Fig. 4), and different input models were assessed and bugs were identified. 5 users including students and teachers from the game or design department were invited as testers. The main findings according to the common feedbacks from the testers are as followings:

Fig. 4.

Screen captures of The AR Journey in NHCM

• Input model of gaze and dwell was difficult to use in this case, as there is an issue of “Observation vs. commit”. Because the FOV is narrow in HoloLens, Audience tend to put the choice in the middle of the view to read, which would accidently also trigger the choice while still reading the it.

• The low framerate caused discomfort.

• Ambient sound was too noisy to hear the character’s dialog.

• Leo’s voice sounded too old for a boy.

• Expression was stiff.

• Leo’s skin was too oily.

Later, input model of gaze and dwell was abandoned, and the voice acting, material of skin, ambient sound and expression were modified. Framerate was improved above 50 fps in accordance with official guidelines [27] and suggestions from the developer community [28].

4 Experiment

Participants

31 university students were recruited to participate in the experiment, and 29 of them (14 females, 15 males, Tongji University, Shanghai, China) has completed the experiment while 2 of them quitted due to the hardware failure. Participants aged from 17 to 26 years old (M = 21.97, SD = 2.15) came from all kinds of majors like film, engineering, journalism, animation, etc. There was no grading associated with the exercise but 10 dollars’ voucher as compensation for completing the experiment and the interview.

Procedure and Materials

The materials used in the study included an interactive HoloLens app named The AR Journey, a semi-structured interview regarding perception of mixed reality, narrative, and user experience. In order to ensure the reliability of the experiment, the voice acting and subtitles were reproduced in the participants’ own language: Chinese.

As Fig. 5 showed, the room, a lab in Tongji University, was selected for this experiment, which was decorated with cherry wood floor and some European-style furniture to be consistent with Leo’s living room of the exhibition in NHCM. AR headset HoloLens 1^st was used featuring a 35° viewing angle see-through holographic lenses (waveguides) with HD definition screen (720p), Intel Atom x5-Z8100 CPU (1.04 GHz), 2 GB RAM, built-in speakers capable of spatial sound, and a HoloLens clicker (its peripheral).

Fig. 5.

Participant is taking part in the experiment in China (left); the hologram mixed into the real-world space participants can see with HoloLens (right)

Participants entered the room and took part in the experiment one by one. Firstly, they had 2 min to go through a short paragraph about Holocaust to get the idea of the general background of the story. Then participants were guided to get familiar with the HoloLens headset and the operation of Hololens clicker less than 1 min. When the tutorial was finished, participant could then explore the room, find the story behind the items, and interact with the story. The process of exploration lasted from 12 min to 16 min depending on the number of items the participants explored. Lastly, the participants were asked to take off the headset and had an 8-min interviewed with author.

Measures

Qualitative methods including interview and observation were used for assessment and evaluation. The semi-structured interview was designed based on questions from Narrative Engagement Scale [5], Presence Questionnaire (PQ) [15], IBM’s Computer System Usability Questionnaire (CSUQ) [29], and The Intrinsic Motivation Inventory (IMI) [30]. The interview questions focused on aspect of narrative understanding, perceived presence, user experience of input, engagement, enjoyment and suggestions. The process of the experiment was video recorded for further behavior analysis.

5 Results and Discussion

5.1 Interview Results

In this section, a brief descriptive analysis of the interview results is shown before being brought into the context of the discussion in the next section.

Interview questions are as follow:

• Q1: Do you feel the characters or environment are unnatural? If so, what aspect is unnatural?

• Q2: Do you feel the above unnatural characters or environment intervened your overall experience? If so, to what extent?

• Q3: Do you feel the interaction are uncomfortable? If so, what aspect is uncomfortable?

• Q4: Do you remember any impressive plot? please describe it.

• Q5: What do you think of Leo’s story? please describe it.

• Q6: How do you like the experience? Please describe it and give some examples.

• Q7: How do you dislike the experience? Please describe it and give details.

• Q8: Would you recommend this experience to your friends? If so, why?

To analyze the results, a two-cycle, simultaneous coding procedure was applied. The first cycle established an initial, in vivo coding. The second cycle used a holistic pattern coding method in order to provide both a summary of the content and a point of reference for the discussion of specific quotes [31]. The second cycle codes and results are shown in Fig. 6.

Fig. 6.

Coded answers to interview questions (Horizontal axis shows percentage of participants who mentioned these in their responses)

In Q1 and Q2, the participants were asked to assess general experience for holograms and holographic character. 26 subjects (89.6%) thought the experience was acceptable (Code 1), and 22 subjects (75.9%) spotted the flaws of the holograms or character but claimed this flaw cause little interference on them (Code 2). The reasons were interesting. Some statements were:

“…I noticed the motion of character was unnatural, but I felt my attention was mainly focused on the story, therefore, I quickly ignored these flaws…”

“…I thought it was because my sense was overwhelmed by these holograms. Consequently, I became more tolerant to the 3D characters’ unnatural expression …”

In Q3, the participants were asked to assess the input method. 27 subjects (93%) thought the input method was easy to learn and appropriate for this interactive narrative (Code 10). However, 2 subjects disagreed with this input, their statements were:

“…I felt more like directly using my hand to touch, press or grab the virtual hologram instead of using this clicker, maybe hand gestures could be better than the clicker…”

“…I wanted to use the clicker just like a laser pointer instead of using head gaze to control the ‘mouse cursor’…”

In Q4 and Q5, the participants were asked to assess the narrative with HoloLens. 27 subjects (93%) were interested in this choice-based storytelling (Code 5). 13 subjects (45%) were inspired to reflect about meaningful issues (Code 6). 10 subjects (34.5%) thought the story itself was flat with few dramatic conflicts (Code 3). 9 subjects (31%) mentioned that the story affected them emotionally (Code 4). Some statements were:

“…I was struggling to make decision to support mom or dad. Their statement both sounded reasonable. I chose to support mom in the end, but I was also curious about what if I chose to support dad …”

“…I wanted to know more about Hitler Youth, so I made the first choice. Actually, I was still curious about the second choice…”

“…I had a deep impression on the choice for issue of equality. It made me think our own society…”

“…The dialogue were too much and too flat, I felt I was distracted…”

“…The story was a little bit boring. I would like to see a Nazi officer breaking into the house. That can be breath-taking…”

“…It also reminded me a movie about Holocaust I just watched, I felt so sorry about Leo…”

“I felt so connected with Leo. When he walked towards me, I also stepped towards him…”

In Q6, the participants were asked to describe the positive experience during the narrative. 14 subjects (48%) thought it was empathetic and interesting that Leo could detect, look at and walk towards them (Code 8). 13 subjects (45%) reported to perceive immersion or presence (Code 7). 6 subjects (20.1%) preferred the way to explore props within a real-world space (Code 9). Some statements were:

“…I felt I was talking to Leo, because he was looking at me when he talked…”

“…I noticed Leo was gazing at me, but sometimes I would feel a little uncomfortable under the gaze of his eyes…”

“…I had the illusion that Leo was really there, especially when he walked towards me…”

“…I didn’t pay too much attention on what’s the virtual stuff and what’s the real stuff, I mean I was immersed into this experience…”

“…I felt like I was transferred to Leo’s room. I thought more furniture’s like carpet on the floor, window curtains could let you feel more real…”

“…The dialogue were too long, I liked to explore stuffs in the room, more explorative props could be even better…”

“…I liked to explore different clues to get the whole piece of the story. I thought the idea of investigating props in the room was great. On the other hand, a branched storytelling was not interesting as the choices were predefined and limited…”

In Q7 the participants were asked to describe the negative experience during the narrative. 23 subjects (79.3%) thought HoloLens was bad for ergonomic design (Code 11). and 12 subjects (41.3%) pointed out the narrow FOV issue (Code 12). Some comments were:

“…The headset was too loose. I used my hand to hold it all the time and my arm ached now…”

“…The headset was too tight to squeeze my head, especially in the later part I was distracted badly…”

“…I felt good at the beginning. The headset became heavier and heavier after a while…”

“…It was odd that I could only see the hologram in front of my head. I saw nothing out of the corner of my eye. That was weird …”

Lastly, all participants (100%) described this activity as an interesting experience and would like to recommend it to their friends (Code 13). Though the reasons were different. Some statements were:

“…because the experience was very fresh and fancy…”

“…I thought the story was also good and invoking, I would like to share with my friend …”

“…because it was more engaging than a flat-screen film …”

“…because I wanted to know the other branch of the story …”

5.2 Discussion

Several interesting findings were revealed based on the above data analysis. The main issues for narrative in IARE are problems with hardware including the ergonomics and narrow FOV. Though most participants (79.3%) complained about the comfort of HoloLens and many of them (41.3%) recognised the disruptive influence of the low FOV, majority of participants still (89.6%) thought the IARE with current device (HoloLens 1^st) was acceptable, and all of them (100%) gave positive comment in terms of enjoyment and was willing to recommend the experience to others. As described in Sect. 2.2, HoloLens 2 has improved the viewing angle and ergonomic design, it is believed that IARE can be further enhanced with HoloLens 2 and has a positive future as the hardware is keeping advancing in recent years.

The computing power of HoloLens is still limited because of the headset-embed CPU, which can only afford the artificial 3D characters and environment. However, most participants (75.9%) seemed to have high tolerance of these artefacts. Attention attracted on narrative, low expectation of AR content and novelty of HoloLens were mentioned most as the underlying reasons. The criticism focused on the body movement and facial expression of the 3D character. This provides guidance for developers that less effort can be placed on asset development, especially on realistic texturing and material. It is also important to note that the weak computing power issue can be possibly solved in the near future as the rapid development of 5G technology.

The modified “quest game” structure employed in this study was proved to be effective. Almost all the participants (93%) hold the positive attitude towards the selectable branched story design while a small number of them (20%) was keen on exploring props and searching details of the story. Meanwhile, there were arguments about the organisation of the props which were supposed to have more inter-connection to achieve a more consistent storytelling. Moreover, almost half of the participant (45%) were inspired to make a reflection of equality and discrimination. Briefly speaking, the narrative was valid in this study for moral education purpose. Interestingly, there were the contradictory attitude on story itself: one third of participants (34.5%) thought the story was flat due to the tedious dialogue and lack of dramatic conflicts while another one third (31%) described the story as touching. One possible reason was that students had different background knowledge of moral issues. Students who were interested in moral issues inclined towards the story.

Almost all the participants (93%) thought the input design is feasible in this study as they felt it was very easy to learn and to use. In addition, natural and autonomous response of virtual characters was vital for interaction as half of the participants (48%) experienced empathy or connectedness with Leo because they either noticed the eye contact or Leo could turn around and find them to talk. Besides, HoloLens 2 is close to be released into the market, manipulation directly by hand is possibly the best input choice and more versatile for narrative in IARE.

6 Conclusion and Future Work

This research introduced a new narrative media known as IARE with HoloLens. The AR Journey, a HoloLens app aiming at interactive narrative, was developed for the NHCM and evaluated through qualitative methods.

To conclude, modified “quest game” type is proved to be a valid narrative structure in IARE, and the input model of “gaze and commit with clicker” is valid for a click-through experiences. The design guidelines were discussed and summarised according to the analysis of collected data,

This research hopes to serve as an initial attempt to release the potential of a new form of narrative using AR technologies, and further explorations in this direction, e.g. to expand understanding or to develop design guidelines, to further investigate the interaction with real world objects, to evaluate the new input models in HoloLens 2, are bound to be fruitful and exciting in the future.