The combination of augmented reality/virtual reality (AR/VR) technology and cinematic rendering could transform communication between radiologists and referring physicians, as well as education, according to research in the Journal of Computer Assisted Tomography.
In a technical note article published September 22, a team of researchers led by Dr. Steven Rowe, PhD, and senior author Dr. Elliot Fishman of Johns Hopkins University School of Medicine in Baltimore shared their initial experience with using AR/VR to view and manipulate the 3D photorealistic images.
"The intersection of these technologies suggests many potential advances, including the ability of interpreting radiologists to look at photorealistic images of patient pathology in real time with surgeons and other referring providers, so long as VR/AR headsets are deployed and readily available," wrote Rowe and co-authors from Johns Hopkins and Siemens Healthineers.
The researchers imported cinematic-rendered images produced on a SyngoVia VB40 workstation (Siemens) to a workstation linked to a Microsoft HoloLens 2 Development Edition headset. A Cinematic Reality software prototype (Siemens) was then paired with the HoloLens.
With hand motions, users can zoom in and zoom out on the image, as well as rotate it and apply cut-planes, according to the researchers. In addition, they can also walk around the cinematic rendering hologram to see obscured anatomy or pathology.
Members of multidisciplinary teams could use the AR/VR environment to discuss anatomy, normal variants, and complex pathology. For example, an emergency radiologist could -- virtually or in-person -- meet with a vascular surgeon to discuss CT angiography findings, according to the researchers. Using HoloLens headsets, both providers would be able to discuss any salient findings.
"Particularly in the context of the ongoing pandemic, it is imperative for radiologists to find ways to communicate subtle findings and function as viable consultants for our clinical colleagues," the authors wrote. "In the era of the COVID-19 pandemic, it is more difficult than ever before for radiologists to be available to their colleagues, due to a combination of working from home and maintaining social distancing at work. The emerging technological intersection of [cinematic rendering] and AR/VR is tailored to the new reality of the COVID era, with the ability to have multiple users, all remote from each other, logged into the AR/VR environment as the [cinematic rendering] images are discussed."
The combination of cinematic rendering and AR/VR could also offer other potential advantages.
"The rise of artificial intelligence (AI) as a driving force in the future of radiology suggests that new visualization methods for volumetric data may be important as data inputs for graphical processing unit-driven AI workstations," they wrote. "Adding AR/ VR to CR visualizations may allow nonradiologists to have input into the images that are sent to the picture archiving and communication system and facilitate the incorporation of clinical data into ensemble AI algorithms."
Other hardware and software could also be used to replicate the group's approach. Photorealistic images could be generated using other rendering software modified with a cinematic rendering lighting model, according to the researchers. Other AR/VR technology available in the market, including the Oculus headset (Oculus Studios) or EchoPixel's True 3D tablet and pen.
"The potential availability of competitive platforms should help make the combination of CR with AR/VR more widely accessible," they wrote.
The researchers noted that their study is descriptive and doesn't provide specific data that would demonstrate the added value of cinematic rendering and AR/VR.
"Ultimately, well-designed prospective studies will be necessary to evaluate the utility of the combination of CR and AR/VR to medical diagnostics," the authors wrote.
