Playing Doctor: This VR Could Walk Physicians Through The Patient’s Body

Ludovic Avot and Yannick Le Berre are big video game fans with Avot’s favorite being “Fallout 4, which guides players through post-apocalyptic Boston”. The game is bleak but immersive, and playing it gave the GE Healthcare designer an idea. What if doctors could use video game technology to step inside the human body — like the heroes of the sci-fi movie “The Fantastic Voyage” — to inspect organs and tissues and search for disease? “We were inspired by the photo-realistic rendering techniques of the high-quality games,” Avot says.

Le Berre, an engineer in medical imaging, wanted to create a way for doctors to walk through the human body using virtual reality. After a few simple but promising prototypes, he decided, with Ludovic, to move things further.

Working at GE Healthcare’s Global Center of Excellence in Medical Imaging Software in Buc, France, the pair spent a hack week — time where GE designers can work on a project outside of their current work — with VR design tools and other gaming software. They used detailed 3D information from CT and MRI body scans to build a virtual experience complete with color, texture, light and other features.

Doctors can use it to observe the glistening pleura of the lungs or ashen pink matter of the brain. They can also “enter” a specific part of the body and closely examine it for things like polyps, tumors and lesions.

Le Berre explained that they used a tracking system capable of following movements so accurately, it doesn’t make users dizzy, like some other VR environments. The prototype, which can be used with off-the-shelf VR headsets like the Oculus Rift®, could be a vital turning point in offering doctors a fresh interpretation of a medical image or to better prepare them for surgery.

“Imagine walking into a virtual room that is your patient’s brain,” says Francois Lenfant, general manager for Global UX Brand & Design Language at GE Healthcare. “You can enlarge the area of interest to the point where you can go inside the brain cells. When you’re immersed in an image, you can work intuitively and find new perspectives.”

Up until now, 3D images have been at the forefront of imaging technology. The center of excellence in Buc is a pioneer in anatomical 3D modeling using images taken with GE scanners. Doctors can use the models to get targeted views of specific organs. They can, for example, focus on coronary artery stenosis and see deeply into the space of the cavity. They can also use 3D printers to make hand-held models of organs.

The next step on this journey into the body is virtual reality. VR is becoming common in gaming, but for it’s still the early days for industry. But Avot and Le Berre have used “open innovation and design thinking processes,” to help VR become an important new tool in healthcare and diagnostics.

The pair presented their VR prototype during the Journées Francophones de Radiologie conference in Paris last year. They say the feedback from approximately 200 radiologists and surgeons who tested it was “entirely positive.”

Next, GE will start testing the VR system with a customer in France this year before expanding to the United States and Asia. The first prototype will be used as a tool to help doctors learn more about anatomy and how to better identify pathologies. Eventually, Avot expects doctors to use the technology not only for diagnostics but also to practice procedures before surgery and to double check results post-surgery.

Le Berre and Avot are already thinking about other ways to help doctors see inside the body. “We’re looking at augmented reality,” Le Berre says. “The idea is to overlay 3D and technical information into the physical world that you’re seeing. This technology could [eventually] be used to guide a doctor during surgery so that he or she can see the tool in the body of the patient in real time.”

GE engineers are also working with artificial intelligence. The idea is that one day machines may be able to recognize cancer on their own.

But for now, human judgment is still the key ingredient. To that end, Le Berre and Avot have tried to make their prototype feel as much like a video game as possible. “We would like to use the multiplayer capabilities to allow multiple users to review the same case,” Avot says. “It’s a video game that all doctors will want to play.”

[youtube https://www.youtube.com/watch?v=bRPAGhpJGZw?feature=oembed&enablejsapi=1&origin=https://safe.txmblr.com&wmode=opaque]

Top image and above: “Imagine walking into a virtual room that is your patient’s brain,” says Francois Lenfant, general manager for Global UX Brand & Design Language at GE Healthcare. Images credit: GE Healthcare