Integrating virtual reality (VR) directly into the radiology workflow can offer radiologists new, intuitive ways to interact with medical imaging data, both individually and in concert with other physicians, say researchers from Canada in an article recently published online in the Journal of Digital Imaging.
In their review, the investigators from the University of Ottawa discussed the predominant medical applications of virtual reality and augmented reality (AR) in radiology, as well as the potential benefits associated with implementing these technologies in clinical practice. They also proposed a conceptual framework that outlines the ways in which clinicians can view medical images in virtual reality within the radiology workflow (JDI, September 13, 2018).
"VR and AR offer a gateway to visualizing complex anatomy with a true 3D perspective," senior author Daniel La Russa, PhD, told AuntMinnie.com. "They provide a means of blending 3D models with the images from which they were derived, resulting in novel and informative views not accessible on traditional platforms."
Virtual reality device allows for visualization and manipulation of a 3D heart model. All images courtesy of Dr. Daniel La Russa.Clinical applications of VR, AR
Technological breakthroughs have continued to increase the quality and accessibility of virtual and augmented reality -- stimulating the development of numerous applications for these advanced visualization techniques in healthcare, according to La Russa and colleagues. The most common application of virtual reality in medicine seems to be its use as an alternative therapeutic and educational tool for patients.
More recently, clinical studies have demonstrated a variety of additional medical applications of virtual reality geared toward the clinician as user, including the following:
-
Training: A growing number of medical centers have developed training programs that allow residents to learn and practice new procedures on a "virtual workstation." The fully immersive environment has helped trainees improve their understanding of complex anatomy as well as enhance their interventional skillset, the authors noted. One such program is a Web-based simulated training environment, RadIQ, developed by Massachusetts General Hospital (MGH).
Several medical schools have incorporated radiology teaching programs that not only educate students about the field but also enable them to practice basic radiographic techniques in a virtual simulation.
-
Surgery: Clinicians have been using virtual reality and augmented reality headsets to plan and simulate surgical and interventional procedures. Research has shown that preprocedural preparation with these technologies may increase clinician confidence and speed.
-
Medical image interpretation: Virtual reality software has also shown promise as a way to boost efficiency for a wide range of tasks in radiology, from diagnosing brain aneurysms on CT scans to correcting segmentation errors on MRI scans.
VR in the radiology workflow
In terms of radiology workflow, there are several different ways to process medical images so that clinicians can visualize them in virtual reality, the authors noted. These workflows could be categorized based on the type of image -- i.e., segmented or not segmented -- a clinician wants to see.
Users interact with medical datasets in 3D via virtual reality.To view segmented models using virtual reality software, a radiologist or clinician needs to use processing software to create and edit a 3D model of a medical image, usually from a dataset of MRI or CT scans. The workflow is not unlike what is used to fabricate 3D-printed models.
"These technologies share many of the same image segmentation and preprocessing workflows, so models created for 3D printing purposes can also be used for VR/AR applications with minimal additional overhead," La Russa said.
The differences lie in the way users need to format the image file and edit the 3D virtual model. For virtual reality models, clinicians should save the data as a Wavefront or object (OBJ) file, rather than the stereolithography (STL) format used in 3D printing. Virtual reality models also require users to select material properties -- such as glossiness and transparency -- as well as the type of shading desired (smooth or flat). The models only need minimal segmentation because they generally include most of the surrounding anatomy, whereas 3D-printed models tend to consist of more focused anatomy.
Segmented 3D models visualized in virtual reality differ from 3D-printed models in that users can manipulate and move through the virtual reality models in real-time, the authors noted. This invites the development of inventive ways to visualize 3D medical image datasets.
Viewing raw images in VR
La Russa and colleagues also introduced three distinct workflows to view nonsegmented, or raw, medical images:
-
2D visualization: Examining nonsegmented 2D medical images in virtual reality calls for converting the pixels of DICOM images into "texture" format. This allows clinicians to look at 2D images without the size limitations of a typical screen or any distractions in the field-of-view.
-
2.5D visualization: After converting DICOM files to the image texture format, clinicians can model a 3D moveable viewing plane that allows them to view a 2D image in a 3D plane. The main benefit of 2.5D is that it enables clinicians to intuitively choose among different image slices within the context of the entire 3D dataset, the authors noted.
-
3D visualization: Finally, clinicians can preprocess images and then use volume rendering software to create and visualize 3D models in virtual reality.
"To reach our full potential as radiologists, we need experience working with images in VR/AR, including knowledge of which tools and settings will optimize the display and manipulation of information," said co-author Dr. Frank Rybicki, PhD. "Armed with the knowledge of how to build VR/AR applications that are compatible and performant with their preferred hardware systems, radiologists can begin to use the technology to benefit patients."
Tool for collaboration
Despite being a fully immersive environment, virtual reality can create new opportunities for collaboration among clinicians from different specialties, the authors noted.
On the one hand, radiologists can partner with scientists such as medical physicists and computer engineers to leverage the capabilities of virtual reality and augmented reality. On the other hand, radiologists can inhabit a shared virtual environment with other clinicians and examine the same medical images from a variety of perspectives while maintaining a sense of the position and orientation of the entire 3D dataset.
"We can put geographically separated radiologists and surgeons into the same virtual environment to simultaneously interpret a patient scan and plan complex interventions," Rybicki said. "This approach to virtual conferencing and planning is entirely novel and will bring care teams to a better understanding of how best to treat patients."
