1 Introduction and Related Work

Vibrotactile feedback (VF) is a popular haptic feedback modality for virtual reality (VR) interaction because it combines relatively low technological complexity in its implementation with a wide variety of achievable haptic effects [5]. There is evidence for a positive impact of VF on many success metrics for interactions with virtual environments (VEs), such as improved task performances [2], improved user immersion [2], increased perceived realism [17] and increased presence [6, 12]. VF can be used to communicate both physical cues relating to the VE (e.g. vibrating objects [16], contacts [4], impacts [8], interaction forces [3], texture roughness [7]) as well as abstract cues (e.g. for guidance, notification or communication [5]). Many technologies can deliver VF in VR, such as wearable [5], grounded [19], hand-held [1], and even mid-air haptic devices [10].

Despite the wide use of VF delivered through ungrounded hand-held devices in VR, the approach is mostly restricted to monolithic VF [5]. This has the advantage of simplicity as it requires only a single actuator. However it remains inadequate for providing spatial information. That is, cues originating from different directions relative to the user are identical and thus indiscriminable for the user, unless a mapping is created between direction and waveform parameters.

Conversely, localized VF through multiple actuators is widely used to convey spatial information to the user, in particular in wearables (e.g. [5, 11, 12]) and surface haptics (e.g. [15]). In this paper, we begin to explore the possibilities offered by localized VF within handheld tangible objects. In particular, we focus on rendering spatialized impacts happening on a virtual hand-held object larger than the tangible held by the user. To render the impacts, the tangible houses two vibrotactile motors at its extremities. We hypothesize that by using two actuators, we can provide localized vibrotactile feedback which can inform the user about where the impact occurred on the larger virtual object they are manipulating (see Fig. 1).

Fig. 1.
figure 1

(A) Close up CAD of the tangible handle being held by the user’s avatar; (B) User manipulating the handle in VR; (C, D) VR interactions causing impacts at different distances and in different directions from the hand.

Full size image

In early work on impact rendering in interactions within VEs, Wellman et al. [21] used a data-driven approach to play back recorded impact vibrations during virtual contacts on a voice-coil actuator embedded into a grounded force-feedback device handle. Okamura et al. expanded on this, compiling a vibration waveform library for impacts generated by fitting a simplified vibration model based on an exponentially decaying sinusoid to recorded impact data [14]. Because this model (see Sect. 2.2) provided an interesting compromise between perceived realism, impact property discrimination, and computing requirements, it has since been widely adopted in interactions with VEs [13, 19]. Some work on spatialization in VR was performed by Gongora et al. [9]. They studied vibrotactile impacts delivered in a bimanual task using a pair of monolithic handheld vibrotactile devices, with the aim of rendering localized vibrotactile impacts along a virtual bar connecting both hands. There have also been a few research attempts at systems spatializing vibrotactile cues inside hand-held devices using multiple vibrotactors [18] or asymmetric vibrations [20] but to our knowledge none have been leveraged in VR interactions.

2 Experimental Design

2.1 Research Questions and Hypotheses

We seek to provide VF to render impacts between one manipulated virtual object and other virtual objects in a VE. Our question concerns the extent to which spatializing impact cues by distributing them between two actuators embedded in a cylindrical tangible handle (see Fig. 1-B) is effective in providing users with information on impact direction. We also seek to understand how this approach affects perceived realism and impact properties, and whether it is compatible with existing approaches to rendering impact distance in a setup using a single actuator (e.g. [9, 19]). To investigate this, we compare distance and direction discrimination performances, as well as perceived realism and virtual object material properties in VR, using different impact vibration models (see Table 1). We formulate the following hypotheses:

H1:

Spatialization of impacts in hand by assigning impact waveforms to distinct vibrotactors will allow discrimination of impact direction, regardless of the chosen impact vibration model.

H2:

Impact models coding distance with more redundant parameters (see Sect. 2.2 for the details of the models) will yield better distance discrimination performance.

Fig. 2.
figure 2

(A) Virtual rod manipulated in the experiment with 4 possible impact distances extending symmetrically around the virtual hand. \(x_{th}\) and \(x_p\) respectively denote the thumb and pinkie side actuator positions. (B) Possible evolution of vibration amplitude A, decay \(\beta \) and frequency \(f=2\pi \omega \) as a function of impact distance for both actuators. Values were determined based on literature and a pilot study. We do not consider any impact occurring within the hand, hence the null values between \(x_{th}\) and \(x_p\).

Full size image

2.2 Rendering Impacts Distance and Direction

We use the simplified impact vibration model introduced by Okamura et al. [14], where \(\alpha (x,t)\) denotes the waveform amplitude at the instant t for an impact at a distance x from the hand (see Fig. 2): \(\alpha (x,t) = A(x) e^{-\beta (x)t} sin(\omega (x) t).\) In realistic impacts, the peak amplitude A, decay \(\beta \) and angular frequency \(\omega \) would all be functions of impact distance as well as impact dynamics and properties of the materials involved. However, such impact models can sometimes be less effective at communicating usable information on impact distance [19]. An alternative is to select a subset of model parameters (A, \(\beta \), \(\omega \)) to encode impact distance, possibly leaving the remainder free for encoding other impact properties. Given these three parameters, there are seven different possibilities (see Table 1) for encoding impact distance (see Fig. 2).

2.3 Materials and Methods

To investigate the formulated hypotheses, we designed a pair of experiments assessing impact direction and distance perception in VR.

Hardware. Subjects sat at a table, wearing an HTC Vive Pro head-mounted display (HMD). They held the vibrotactile handle in their dominant hand which was tracked using an HTC Vive Tracker attached using an adhesive fixture to keep the palm and inside of the fingers unobstructed. They used an HTC Vive Controller held in their non-dominant hand to answer experimental questions (see Fig. 3-A). The handle was equipped with a pair of symmetrically mounted Actronika HapCoil One voice-coil actuators (see Fig. 1-A).

Fig. 3.
figure 3

(A) A subject performing the experiment. (B) VR view of the familiarization task, where the impacted objects are visible. (C) VR view of the actual task, where the impacted objects are hidden and only haptic feedback of impacts is provided.

Full size image
Table 1. Impact vibration models for encoding impact distance x from the hand studied in our experiments. The model names indicate the vibration parameters that vary as a function of impact distance, with Amp referring to amplitude A, Dec referring to the decay \(\beta \) and Freq referring to the frequency \(\omega \). (Left) Models used in experiment 1; (Right) Models used in experiment 2; AmpDecFreq was common to both experiments.
Full size table

Experimental Task. The common experimental task for both experiments was inspired from Sreng et al. [19]. Subjects were asked to hold the tangible handle in their dominant hand. They observed the VE showing their virtual hand holding a virtual rod with the same diameter as the tangible handle but extending symmetrically outward 0.5m beyond the edges of the tangible handle. By moving this virtual rod up and down, it could impact a lightweight and unconstrained object at one of four distances \(d_i = {0.05, 0.2, 0.35, 0.50}\,\mathrm {m}\) from either the thumb or the pinkie side of the hand (see Fig. 2-B,C). These impacts were rendered according to one of the impact models summarized in Table 1. During the experiment, the impacted object was obstructed from view so as to provide no visual feedback of the impact location (see Fig. 3-C). Subjects placed the rod at the starting location, then were prompted to move it downward. On the way down, the stick impacted a first virtual object which appeared randomly on the left or right at one of the distances \(d_i\). Upon reaching the target location, subjects were prompted to return the stick to the start location and repeat the process. A second object appeared on the same side as the first, at one of the four possible distances, generating a second impact, after which subjects answered a pair of experimental questions:

Q1:

Which side did the impacts occur on? (Left/Right)

Q2:

Was the second impact further away from the hand than the first? (Y/N)

Experimental Design and Protocol. For achieving a shorter experiment, we split our investigation into two identical experiments containing 4 blocks each. Impacts were rendered respectively using Amp, Dec, AmpDec, AmpDecFreq for experiment 1 and Freq, AmpFreq, DecFreq, AmpDecFreq for experiment 2 (see Table 1). 24 subjects (19 m., 5 f., ages 21–30 (Mean:24.9 y), 20 right-handed) took part in the study after providing written informed consent. Subjects were randomly assigned to one of the two experiments.

In each experiment, subjects first performed a familiarisation task where the VE was fully visible, showing the hand-held virtual stick and the impacted virtual objects (see Fig. 3-B). During this task we ensured that subjects moved at a similar speed, though the vibration did not depend on it. They were informed that the rod and impacted object properties might vary during the course of the subsequent experiment. Subjects filled out an initial questionnaire indicating personal data and prior experience with haptics, VR and perception studies.

Each experiment contained one block per impact model, whose order was counterbalanced between subjects. Within each block, subjects performed 3 repetitions of the task for each of the 16 combinations of impact distances occurring on either side, totalling 96 trials presented in a fully random order. Post-block questionnaires assessed perception of the stick and impacted object’s material and geometric properties, their variability, and perceived impact realism.

3 Results

Impact directions were consistently correctly identified between 94% and 97% of the time across all impact models. Most errors occurred for pairs of low amplitude and duration stimuli.

To test for H2, subjects were assigned, for each impact model, to one of two groups based on whether they interpreted the impact model as intended (increased impact distance perceived as an increased impact distance) or in an inverted manner (increased impact distance perceived as a decreased impact distance). Inversion rates (percentage of subjects interpreting an increase in impact distance as a decrease) were around 50% for all models not involving Freq, and varied between 92% and 100% for all models involving Freq.

Fig. 4.
figure 4

JNDs for impact distance, expressed as Weber fractions, for both experiments.

Full size image

We then computed the 75%-just-noticeable-difference (JND) for distance discrimination as a Weber fraction for each subject by fitting cumulative Gaussians to the data. Finally, we compared the distribution of JNDs across impact vibration models (see Fig. 4). Data from the experiment 1 (Fig. 4-B) were not normally distributed, and a Friedman test showed no significant differences between conditions. Data from the experiment 2 were normally distributed, and a 2-way ANOVA showed a significant effect of impact model (\(F(3) = 4.132\), \(p = 0.021\)) but no significant differences between participants. A post-hoc Tukey HSD test revealed the only significant difference to lie between the JNDs for the Freq and AmpDecFreq conditions (\(p=0.016\)).

Rod properties were rated most consistent (median 2 of 7) in all conditions but Amp (median 3 of 7) and AmpDecFreq (median 4 of 7), however none of these differences were significant. The properties reported as changing between trials were rod material (Freq, AmpDec, AmpDecFreq), stiffness (all models except Dec), length (Dec, AmpDec, AmpFreq), fill (Dec, AmpDec), weight (Freq, AmpDecFreq). Subjectively reported rod materials were dominated by “metal” and “plastic” for all models involving Amp, as well as the Dec model, with qualifiers such as “resonating” and “tube”. Models involving Freq but not Amp yielded more “wood” and “plastic” responses, with qualifiers such as “soft”, “damped” and “warm”. AmpDecFreq yielded an almost even mix of all three material categories. Realism was consistently rated as average across all models (median 3 of 7) and was considered slightly variable across all models (median 3 of 7).

4 Discussion

The impact direction identification rates between 94% and 97% indicate that regardless of the chosen impact model, spatializing the impacts between two actuators allowed subjects to correctly and intuitively identify the side on which the impact occurred with a high degree of accuracy. Hypothesis H1 is therefore verified. Looking at inversion rates, it is interesting to note that all models involving Freq tended to be systematically inverted (92% to 100% of subjects perceived an increase in distance as a decrease) which would indicate the evolution of \(\omega \) may be the cause for this.

Weber fractions for distance discrimination were consistently high across all impact models except AmpDecFreq (m = 0.17), ranging from 0.6 (DecFreq, experiment 2) to 1.32 (Amp, experiment 1). This indicates that while distance discrimination was mostly possible, it was far from an easy task. The only statistically significant difference observed (Freq-AmpDecFreq, experiment 2) is in favor of hypothesis H2, and the mean JNDs seem to also support the hypothesis. However, given the poor performance of AmpDecFreq in experiment 1 and the fact that none but one of the differences are statistically significant, we cannot conclude that H2 is supported. This may be due to H2 being wrong, or to flaws in the stimulus or experimental task design. If H2 is not verified, there may be a lot of headroom to encode various impact properties by distributing them across different parameters without adversely impacting performance.

The high inversion rates due to using \(\omega \) as a parameter led us to hypothesize that models combining \(\omega \) with A, \(\beta \) or both may have been confusing to half the subjects that did not invert their interpretation of Amp and Dec. This hypothesis cannot be easily tested because subjects that performed Amp and Dec did not perform AmpFreq and DecFreq. Yet, analysing the results from experiment 1 revealed that 6 out of 12 subjects had inverted both Amp and Dec while 5 of 12 had not (the remaining subject inverted only one of both models). By looking at the JNDs for each of these groups of subjects in the AmpDecFreq condition, we note that the group that inverted both Amp and Dec performed better at AmpDecFreq (JNDs: 0.07 to 1.96, mean = 0.83) than the group that did not invert Amp and Dec (JNDs: 2.15 to 9.33, mean = 4.63). This would tend to support our interpretation and argue for the need to redesign the function \(\omega (x)\) in our rendering approach. Furthermore, it may be necessary to consider the frequency dependence of vibration amplitude perception in such a redesign. However, given the very small sample size, this conclusion must be seen as tentative.

The spread in JNDs between subjects indicates a large inter-subject variability in the ability to perform the task. During the experiment, several subjects noted that the task was really difficult until they “chose” a way to understand the mapping of the stimuli to impact distance. Thus, we believe this variability shows that subjects displayed different capacities for adapting to the difficulty of the experimental tasks and choosing an effective response strategy. This means that the haptic representation of impact distance is far from intuitive or natural with the chosen models, although AmpDecFreq shows some promise in experiment 2. This may indicate poor model design, or the fact distance discrimination is really hard without any context such as e.g. visual feedback of impacts.

All models were perceived as equally (un)realistic, indicating that either the impact model used is unrealistic, that spatialization impacted realism, or both.

5 Conclusion and Perspectives

We presented an investigation into the use of spatialized in-hand vibrotactile feedback for VR interactions. Our study focused on the ability of a handle equipped with two vibrotactors to deliver realistic, discriminable and understandable sensations of impacts through which users could determine the location (direction and distance) of impacts on a virtual manipulated object.

We determined direction identification scores, JNDs for impact distance, and perceived impact realism for 7 impact vibration models. Results showed excellent direction identification performances, but distance discrimination performances were mediocre. Impacts were perceived as only moderately realistic, which may be due to the impact models studied as well as our spatialization technique.

In future work, we plan to investigate whether differently distributing the vibrations between both actuators can improve perceived realism and consistency between impacts while preserving distance and direction discrimination performance. This study also highlighted certain avenues for improving the perception of vibration impact models which we intend to investigate. Finally, we plan to extend the approach to 2D and 3D spatialization using more actuators.

Because of the good direction discrimination performance observed in this initial study, we believe there is potential for using multiple actuators in manipulated tangible objects or controllers. This seems particularly promising for VR applications which could benefit from the use of in-hand directional cues.