The Right to Make Mistakes… in Virtual Surgery

“I think I just killed him. Oh, no.”

“There’s blood everywhere! What do I do?”

“Where is it? This one? Which one is it!?”

The anguish and panic in my colleague’s voice was palpable.

“The one with the glowing arrow.”

She listened to my instructions. A few gestures later, the bleeding had stopped, and, with it, the simulation. She removed the Virtual Reality headset from her red face and let out a sigh of relief before exclaiming, “That was awesome!” She had just removed a gall bladder for the first time and stopped a hemorrhage on the way.

While our Mixed Reality gall bladder removal surgery simulation gave our colleague renewed appreciation for her non-medical career choice (she claimed she would have made a lousy surgeon), we as a team believe that this same technology can undoubtedly help countless others, young and old, to explore, understand and master their skills in the very demanding field of medicine, especially in surgery.

Over the last year, we’ve had the privilege of working with several surgical experts to develop a few exploratory proof-of-concept projects using XR technology in surgical training.

Below, we share some insights gained from these explorations.

Challenges in surgical training today

It should come as no surprise that it takes time to train good doctors, and evidently, it takes even more time to train good surgeons.

A veteran cardiac surgeon and surgery professor from a major French public university shared with us that “Every ten years or so, I might have one… maybe two students who will go on to become great surgeons. Fewer and fewer medical students are going into surgery because it is even tougher and more stressful.” Indeed, according to the OECD Health Statistics of 2022, France has one of the lowest doctors per capita scores within the OECD, ranking ahead of only Hungary and Luxembourg. If we couple long training time with shrinking pools of candidates, the gap appears difficult to bridge mathematically.

The situation appears similar in Canada. According to the Job Bank website of the National Employment Service (Government of Canada), “This occupational group [surgeons] is expected to face labour shortage conditions over the period of 2022-2031 at the national level.”

Regardless of the nature or the number of the root causes of these shortages, be they systemic or otherwise, we can all agree that improvements in the training methods deserve to be studied and explored. In comes XR.

The question of an all-encompassing solution

Before unpacking the number of improvements that XR technology can bring to surgical training, it is important to clarify a few reasonable misconceptions that may come to mind when we think about using XR in surgical training. Even we, as non-surgery technology experts, had them.

Every surgeon we’ve consulted with has emphasized the primordial importance of tactile and haptic feedback in their work and training. The surgeon must be able to physically feel the tissue that they are working on. They have to feel the tool they are holding in their hands like they’ve become an extension of their body: it can become as technical as which joint of which finger is touching which part of the tool.

An example of a recreated laparoscopic tool being held and used in XR.
Note the tool is entirely virtual but can be held in the user’s real physical hand.

While certain XR accessories do promise such capabilities (like a variety of haptic glove models), the technology today for haptic feedback is still nascent and will require additional testing and fine-tuning to meet the level of precision and dexterity required for surgical training. Considering these accessories’ complexity, scalability and cost, using XR to target basic training can appear out of reach right now. In fact, it might even seem like overkill.

A heart valve replacement specialist humorously remarked that “if you need to use XR to learn how to hold your tools or how to cut, you have no business being in surgery school.” What he really meant to say was that training for the basics of surgery is a mostly solved question until there is a technological breakthrough. It is not unheard of for students to practice cuts and sutures on pieces of pig or simulated silicone skin, which are relatively low-tech, low-cost solutions. These solutions offer the tactile and haptic feedback crucial to surgeons in their training.

One VR anecdote I often like to share is that of a colleague who told me a few years back that they tested a VR application where you could open a virtual book (in VR), flip through the pages and read it. When asked if there was anything else to it, they declared, “No, you just read the books. It could encourage people to read more, perhaps.” I don’t know if that application became a running success or not, as I no longer recall the name of it (more power to them if so!), but I immediately thought about the optics on the impactful use of the technology: one can also grab a physical copy of the book, flip through the pages and read them, or use an e-reader and “flip” through the pages, electronically. Did the perceived or alleged added value of VR justify the development resources and time spent on the application?

As XR developers affiliated with Concordia University, we believe adding this dimension to our cost-benefit analysis is an important part of evaluating concepts and projects. Should we use XR to solve an issue that has already been solved? Can we be certain that our XR solution will bring significant improvement? Or should we target difficult challenges that other technologies have not yet attempted to solve? These are questions we wrestle with when we collaborate with experts to produce learning material in XR.

In the context of limited resources (as is the case in many academic circles) and given the cost and complexity of XR development, we believe maximizing impact is a judicious guiding principle to keep in mind when vetting concepts. It is also our small way of preserving the technology’s reputation, as to dampen accusations of XR being a gimmick, a toy that we try to jam into every use case in existence. How many times have we met people whose lack of enthusiasm for XR was because “I tried it years ago, and it was bad.” From the many public showcases and events that we do, we can say that it is a noticeable group.

This principle is certainly applicable to the medical field as well. To circle back to our heart valve specialist friend: no, we do not need to teach surgeons how to hold a scalpel or cut skin with XR; it’s a waste of time. At least, not until the technology can provide extremely precise haptic feedback that can replicate the feel of human tissue in a cost-effective manner.

So, what did we try to solve?

The potential benefits of XR in medical training

We don’t believe that XR will replace traditional training. Instead, it will complement it by focusing on practical questions, as opposed to didactic questions. It’s about shortening training by helping the student prepare for their training because training resources in the medical field can also be very precious.

Producing memorable learning experiences

With basic manipulations out of the picture, our exploration focused on three promising areas of training identified by our surgeon partners: learning procedures, stress management and remote mentoring.

Procedures

This is less about learning how to manipulate the tools and more about helping the student remember the steps to a complicated, multi-step procedure. What are the steps required to perform a valve replacement surgery? What are the steps to deploy the new replacement valve? What about the gall bladder removal surgery? What are the things to watch out for? What are the risks associated with a particular procedure?

Some may object. Can’t this be achieved via traditional means? Is it just a matter of having the student memorize steps? Why do this in VR, as opposed to having the students read a book, watch a video or even watch a mentor surgeon perform the surgery in person?

According to the report The PwC 2022 US Metaverse Survey, PricewaterhouseCoopers noted that VR training has been proven to increase learner engagement and improve learning outcomes compared to traditional methods in many ways, namely that VR training is 4 times faster when compared to in-classroom training.

It creates powerful memories in the learners that they can later refer to during in-person training or during their practice. The goal is to help the students internalize the steps to a complex procedure through countless simulations so that there will be less hesitation when they are training in person. Remember: the objective is to maximize the impact of in-person training (due to limited resources – more on this later) by priming the students beforehand through their training in XR. Having the student perform the procedure in XR engages more of their senses and focus than having them watch or rewatch videos.

Stress management

Though anecdotal, my colleague’s response to having “killed a patient” is a natural and very real reaction. The same PricewaterhouseCoopers report states that VR learners are 3.75 times more emotionally connected to the content than classroom learners. This is being leveraged by Professor Teresa Hernandez Gonzalez (Phd) of Concordia University, who aims to use VR training to emotionally prepare future teachers.

While Professor Hernandez Gonzalez’s focus is on training teachers, her observations are particularly poignant: “Despite the education they get in their university programs, new teachers tend to teach the way they were taught themselves. This is because once they face the stressful situation of a real classroom, they don’t reach for the knowledge acquired at the university; their brain reaches for the memories stored from previous experiences as a student, and they replicate that. Therefore, creating immersive experiences that can be stored as memories helps new teachers activate those memories when teaching in a real classroom.” (Please read more about her ambitious and fascinating project here.)

Surgeons also go through stressful experiences in the operating room and can encounter unexpected “incidents” during surgery: internal bleeding, a tool malfunction, a wrong cut, etc. These events’ very “unexpected” nature makes in-person training much more challenging. Therefore, what XR can offer are environments that are realistic enough to elicit an emotional response yet remain totally safe for the student. The objective is to repeatedly expose them to difficult situations and allow them to internalize proper responses to build up a library of memories they can rely on once they encounter their own “incidents,” be it in training or the real world.

Remote assistance

XR cannot replace the guiding hand of a seasoned mentor. What it can do, however, is massively extend the reach of the mentor’s arm. The great promise of XR training is to allow surgeons, who may be separated by large distances for whatever reasons (travel, conferences, work, etc.), to remotely connect to a headset of a student and be able to monitor their progress directly by seeing what the student sees, and by letting the student see what the mentor does with their hands through their headset.

This goes beyond the context of training and applies to the use cases where one surgeon could remotely connect to the device of another surgeon to provide direct assistance during a surgery in real-time, where the remote surgeon can clearly see what the other surgeon sees and provide visual guidance. In effect, a surgeon in Montreal could receive remote assistance from a colleague in France in real time.

Visual mock-up of what such an intervention could look like. Note that the user will be able to see the hands of the remote surgeon, who will also see what the user sees. This can be in a virtual training environment or during a real surgery.

Practical considerations

What are the practical benefits of XR training anyway?

Portability: bringing the operating room to the student

We had mentioned the idea of limited training resources. Operating rooms (OR) are not a dime a dozen, no matter which part of the world we’re in. One of our surgeon experts said, “It’s not as simple as booking an operating room for a class. It’s usually occupied for obvious reasons and at wildly different hours. Some hospitals even have trouble tracking which room is booked at what time. It’s not that practical.”

One of the recent advances of XR hardware is its portability. Whether the headset is connected to a laptop with a cable or is entirely wireless and standalone, the user is able to bring the device with them into different spaces, be it inside a regular classroom or even at home. It is so portable that, in fact, one of our colleagues simply packed up one of our headsets (containing the simulation) into their carry-on bag and flew to France to meet with our surgeon experts.

Headsets have mostly gone wireless or can run while being tethered to a laptop, making them much more portable than real medical simulation devices or 1^st generation headsets.

The immersive nature of XR offers the student the flexibility to be exposed to different types of environments, ranging from a fully immersive space in VR to a partially immersive space in mixed reality, where the student can see both the virtual and their physical space simultaneously. It is flexible enough to target the specific learning goals of the training. The student can get accustomed to these environments virtually so that when they do show up at the physical location, the familiarity is already there.

The image on the left shows a fully immersive operating room with walls, floors and ceilings. The image on the right is a Mixed Reality example (note the regular office door and window in the background), allowing the user to see virtual objects overlaid on top of their physical space.

This isn’t unique to the medical field: for example, energy and mining companies have long used 3D models of their assets to train future crew members, for example, allowing the crew to be virtually trained inside a 3D model of a refinery or an oil tanker, sometimes while it is still being built. This ensures that the crew will already have familiarized themselves with the layout of their workspace when they set foot on it, considerably speeding up the training process.

Repeatability: infinite training time

Simulations are, by definition, repeatable: the student can go through the (possibly randomized) scenarios as many times as they wish without requiring any additional resources or time from the instructors, giving them a powerful self-study tool. Repetition helps the student better absorb and internalize the material on their own terms.

Scalability: cost-effectiveness

There are countless specialized professional tools to help surgeons practice. They can be large, unwieldy, highly sophisticated machines, and their price tags often reflect that. To reiterate our earlier point, we believe XR training is unlikely to replace these sophisticated tools in the near future. What XR can offer is the ability to bridge the self-directed part of the training with the actual training using real tools.

A heart-lung machine expert shared: “It’s not that easy to ship this new model of a 400-pound machine to the hospital ahead of time just so that they can practice.” Suddenly, the prospect of allowing perfusionists to practice operating a digital twin of the new machine in XR before receiving the physical machine becomes interesting. A headset, after all, weighs only a few pounds. By the time the perfusionists receive the machines, they would have operated it dozens, if not hundreds of times in XR, gaining a strong understanding of the tool’s functions and usage ahead of time.

The initial investment primarily focuses on developing a simulation that is accurate. Any subsequent updates or modifications to the training are done at the software level. Once that is complete, the distribution costs are primarily associated with that of a headset, which can fluctuate between 500 to 3500 USD per unit depending on the quality or needs.

The Right to Make Mistakes

To make mistakes is to be human; there’s no way to avoid them. The stigma is exacerbated when some mistakes are much more scrutinized than others because not all mistakes have the same consequences. This is all the more poignant in the medical field, as a mistake could have a significant impact on the patient’s well-being, including life and death.

Just like everything else in life, risk can be mitigated but cannot be fully eliminated. Entire careers and industries exist to manage risk. Medical professionals consciously make efforts to mitigate the risk as well, and XR is a tool that can help in this effort. Why?

Because XR provides a realistic environment that can make the students feel the consequences of their mistakes without harming a real patient and, ultimately, without harming the student. We want the students to make all the mistakes they can make, and we want their mistakes to happen during training as much as possible when the consequences are virtual. We want to provide them with an environment where they fail often and fail fast so that they can return to the physical world with a body of experience that will contribute to the building of their confidence and professionalism.

Like all training efforts, XR embraces the future doctors’ right to make mistakes.

The Path Forward

What else does XR have in store for us? We believe it brings a lot of potential.

When we created the VR experience A Conversation with Newton (where the user can meet and freely converse with a GenAI-powered 3D virtual avatar of the famed Isaac Newton – please read more about it here), we realized the natural synergies that exist between immersive XR worlds and ultra-responsive AIs. GenAI is in the process of breathing life into immersive environments and giving users almost true free will by providing learning content without the scripted aspects of more traditional learning or interactive experiences. The user is truly able to ask whatever questions they have in mind and receive answers that are curated in real time by the AI.

It is, therefore, not a leap to believe that this same technology could be used to create specialized, unique, memorable virtual mentors who will accompany the user in their learning journey and provide lively, contextually relevant feedback to help guide the student in their quest for knowledge and wisdom.

Are we talking about creating digital clones of surgery experts so that they can simultaneously train many more students when they are continents or even generations apart? Sort of, yes! This idea was floated by one of our veteran experts himself.

“In 20-30 years, I probably won’t be around anymore. Unfortunately, all my experience and knowledge will be gone, too. Wouldn’t it be great if we could somehow store all this knowledge so that future doctors can still access it?”

Could future doctors meet current and past doctors and tap into their knowledge… virtually? Meet and exchange with their virtual avatar in the virtual world?

A grand idea for a noble cause.

As the iconic Isaac Newton wrote himself, “If I have seen further [than others], it is by standing on the shoulders of giants.”

Perhaps the time for virtual giant shoulder rides has come.

Author:
Ping Ng

Learning Experience Designer @KnowledgeOne. Artmaker. Tinkerer. 3D Printing enthusiast. Gamer. Night owl.