From Virtual Reality to Immersive Analytics in Bioinformatics

1.1 A Workshop with Live and Immersive Demonstrations

The special session ’From Virtual Reality to Immersive Analytics in Bioinformatics’ was organized as part of two conferences: Stereoscopic displays and applications (SD&A) XXIX, taking place 29 Jan – 31 Jan 2018, and The engineering reality of virtual reality 2018 (ERVR), taking place 30 Jan – 1 Feb 2018, both in Burlingame, California as part of the 2018 IS&T International Symposium on Electronic Imaging – often just known as the Electronic Imaging Symposium.

Established in 1990, and now in its 29th year, SD&A focuses on developments covering the entire stereoscopic 3D imaging pipeline from capture, processing, and display, to perception. The conference brings together practitioners and researchers from industry and academia to facilitate an exchange of current information on stereoscopic imaging topics. The website is located at: http://stereoscopic.org.

The ERVR conference focuses on the creation of virtual and augmented reality systems. In addition to research in this area, the trend toward content building continues and practitioners find that technologies and disciplines must be tailored and integrated for specific visualization and interactive applications. This conference serves as a forum where advances and practical advice toward both creative activity and scientific investigation are presented and discussed. Research results as well as applications can be presented.

Most of the authors of the manuscripts presented in this contribution provided a live demonstration of their implementation at the conference venue. This demo provided a first hands experience and furthermore enabled us to gauge the public’s enthusiasm for such technologies, which was strong. Furthermore, videos of the implementations were shown, user feedback was described and use cases were reported.

Other manuscripts presented at the 2018 SD&A and ERVR conferences will be published in the Electronic Imaging Symposium conference proceedings available open access at: http://ist.publisher.ingentaconnect.com/content/ist/ei.

4.1 Interactive Molecular Graphics for Augmented Reality Using HoloLens

An important device in the field of Augmented Reality applications is the Microsoft HoloLens^®. This device can be used to place virtual objects into real-world environments. The manuscript shows proteins floating over a table or projected in front of a tree. In this way, the observer can inspect the spatial structure of the protein from all perspectives using natural displacement and body movement. Similar technologies may be used in future approaches to elaborate protein structures collaboratively in the context of a regular environment, liberated from the limited perspective provided by regular computer displays. In case of the HoloLens, the big advantage of an integrated computing device comes with a drawback: the compute performance is quite low. In case of molecular data, where usually thousands to millions of atoms are visualized as spheres, this limitation is problematic. In this work, the performance of the HoloLens is analyzed, extended by a good overview concerning related application cases. A natively developed Universal Windows Platform application with different rendering techniques is compared to a Unity application in terms of performance, varying the number of atoms to be visualized [24].

4.2 Heuristic Modeling and 3D Stereoscopic Visualization of a Chlamydomonas reinhardtii Cell

Chlamydomonas reinhardtii is an unicellular organism which plays an important role in the field of metabolic engineering as it can be used as an alternative energy source. In this work a spatial mesoscopic model of this cell was created using an interpretative cell visualization approach. The idea is to combine the knowledge of different information sources – such as publications, microscopic images, database information – with the purpose to create a well-evaluated cell model which can be used for visualization and exploration purposes. The so-created model was used as a basis for a stereoscopic 3D cell animation, discussing the underlying metabolic processes to extract energy from a biological cell. The publication discusses the nature of the different information sources, the design choices involved in the process (e.g. in terms of coloring, texturing and structure building), the way the flagella-based movement of the cell was animated, as well as the special requirements to create a stereoscopic 3D animation. The underlying observations are relevant for content creation for virtual environments and should encourage visualization researchers to use stereoscopic 3D visualization in the context of their work which also plays an important role in Virtual Reality-related applications [25].

4.3 Molecular Dynamics Visualization (MDV): Stereoscopic 3D Display of Biomolecular Structure and Interactions Using the Unity Game Engine

An important field in Bioinformatics as well as Chemoinformatics research is the simulation of molecular structures. One of the most established techniques is the molecular dynamics (MD) simulation approach. Such simulations were performed in the context of this work with the Gromacs [26] software. The visualization of molecular dynamics has a long tradition, using standard tools such as VMD or PyMol. However, these visualization packages are not optimized to be used in the context of multiscale virtual 3D stereoscopic environments. The molecular dynamics visualization (MDV) software has been developed on top of the Unity development platform and can be used with a range of large-scale immersive displays. The HIVE Cylinder at Curtin University for example consists of a 3 m high × 8 m diameter screen with a 180° field-of-view. The screen is illuminated using three 1920 × 1200 DLP projectors warped and blended to provide a continuous image across the screen, supporting 3D stereoscopic visualization with (active) liquid-crystal shutter 3D glasses. Using MDV, MD simulations can be collaboratively as well as immersively analyzed with groups of up to 50 people. Standard molecular visualization metaphors, such as ball & stick and secondary structure representations are implemented and can be selected via a 3D user interface [27].

4.4 Mesoscopic Rigid Body Modelling of the Extracellular Matrix Self-Assembly

As the structure of the extracellular matrix (ECM) is not well known due to technological limitations, alternative approaches are required to predict and model its structural properties. Using the game and physics engines of Unity 3D^®, a framework was developed to model large molecules as dynamic chains of interacting rigid bodies and to observe self-assembly processes. Thereby, the ECM components can be modeled and evaluated at the mesoscopic scale. Properties like fibre flexibility as well as intermolecular interaction types and occurrences can be analyzed. Using an HMD, the user of this framework is immersed into the spatial simulation which can be easily explored by using the 3D controllers of the HMD. For this purpose, a gaming machine like device was placed into a virtual environment which can be triggered to run the ECM simulation. It is possible to observe the structural changes of the different ECM components and compare their structures to each other. The publication further compares the rendering and CPU performance of rigid bodies representing one to three million atoms [28].

4.5 MinOmics, an Integrative and Immersive Tool for Multi-omics Analysis

MinOmics is an integrated analysis pipeline and visualization framework for multi-omics analysis. In this work, a stereoscopic 25 MPixel display wall was used to display proteomic data on 1417 proteins of Chlamydomonas reinhardtii. For this purpose, UnityMol was used, a Unity 3D^®-based software application which visualizes proteins in 3D [29]. A number of hardware setups are discussed which could be used with MinOmics: a UnityMol-based WebGL and WebVR application using a regular monitor, a split-screen approach combined to a stereoscopic monitor, and a full-immersive environment using head-mounted displays. A number of scientific application cases for these setups are discussed. For example, the MinOmics framework can be used to explore multiple omics datasets. The authors discuss the exploration of redox proteomic datasets in the context of Chlamydomonas reinhardtii for glutathionylation, nitrosylation and disulphide bonds. MinOmics can be used to explore protein specificity of multiple redox post translational modifications (PTM), the cysteine specificity of multiple redox PTMs, and to explore structural determinants of redox-modified cysteines [30].

4.6 Semantics for an Integrative and Immersive Pipeline Combining Visualization and Analysis of Molecular Data

Most bioinformatics-related workflows require the use of a number of different tools. Often this diversity means that data has to be transformed, converted to other formats and transferred leading to a huge overhead of time and costs. This overhead applies especially to virtual environments, where sometimes the data to be immersively visualized has to be optimized by visualization experts to make it compatible to the virtual environment, such as the aforementioned HIVE Cylinder display. In this work, an integrated pipeline is introduced, promoting direct interaction on semantically-linked 2D and 3D heterogeneous data, which is immersively displayed in a virtual working place. The focus here is on the interactive visualization of heterogeneous molecular data which is combined based on existing or inferred links following hierarchical concept definitions. The user is supported by the developed system with adaptive analyzes customized towards the required tasks. The developed ontology describes both structural biology as well as interaction concepts [31].