Medical Training Comes to Life in UMD’s VR Simulation

The 76-year-old patient in the doctor’s office suddenly becomes mute and can’t move her right arm—classic symptoms of a stroke.

Cheri Hendrix, director of the physician assistant (PA) program at the University of Maryland, Baltimore goes into assessment mode, her voice growing urgent. “Tell me what day it is,” she says as she checks for a heart rate. “Tell me where you are.”

The scene isn’t playing out in real life. It’s a 3-D training simulation representing a common stroke scenario. Soon, PA program students outfitted with virtual reality (VR) goggles will be able to immerse themselves in the experience, watching Hendrix evaluate the patient as if they were with her, seeing what Hendrix sees and even “walking around” in virtual space to view the exam from different vantages.

The imaging facility used to create the scene, called the HoloCamera, was built by University of Maryland, College Park computer visualization experts in response to studies showing that people retain more information in immersive training environments. The 3-D medical training simulation project using the HoloCamera is a partnership between the UMD and UMB campuses.

“Sitting still and using a desktop and mouse doesn’t aid in forming memories as well as experiencing things in an embodied context,” said Amitabh Varshney, a professor of computer science, dean of UMD’s College of Computer, Mathematical, and Natural Sciences and lead investigator of the project. “Immersive interactivity is more educationally engaging than watching a video or flipping through a book.”

In September, Varshney’s team unveiled the technology to a pair of PA students. At the HoloCamera studio in College Park, second-year clinical student Ashlee Leshinski donned goggles and started moving her head from side to side. A wall monitor captured her fluid view as she observed Hendrix assessing the patient, a professional actor recruited for the project.

“Oh, wow, that’s super cool,” a grinning Leshinski said as she moved around the room and looked down, as if perched above the patient’s chair. “You can see her perspective and all the angles. It’s so clear, too.”

Later during an anatomy education demonstration, Leshinski and a fellow clinical student stepped into a second immersive environment. This one featured an animation of the human cardiovascular and neurological systems overlaid on the patient from head to toe. A UMD team member guiding Leshinski through the simulation was able to trigger a visual representation of an irregular heartbeat that sent a blood clot to the patient’s brain and impaired nerve impulses down her right arm, connecting the outward symptom of immobilization with internal causes.

The HoloCamera features volumetric capture technology, relying on 300 high-resolution cameras precision-mounted in a green room studio to film subjects in 360 degrees. The collective ultra-high-definition footage was fused together using UMD-developed hardware and software featuring artificial intelligence.

“Medicine is one area where precision matters,” said Varshney. “That’s not possible with low resolution.”

The project was partially funded by a $1 million National Science Foundation grant led by Varshney. Development kicked off several months ago when Hendrix filmed the stroke scenario with the actor inside UMD’s studio. The team worked with medical professionals to create the anatomical overlays.

The HoloCamera was developed amid an increased demand by clinical training facilities for instructional tools that feature lifelike environments. Varshney’s research has shown that people in immersive environments recall information almost 9% better than people who watch traditional instructional videos or read textbooks.

To study the HoloCamera’s effectiveness in a stroke patient setting, Hendrix later this fall will compare a group of students trained with the tool with a control group receiving traditional classroom instruction. In the years ahead, the team plans to apply the tool to anatomy education and surgical training at the University of Maryland Medical Center in Baltimore.

Not only will the technology help students in College Park and Baltimore, but it can also be scaled to assist practitioners in rural areas and far-flung countries, bringing the talents of Maryland professors and clinicians to populations in need, the developers say.

“This will be a boon to education all over,” said Hendrix. “Sky’s the limit.”

—Story by Maryland Today