We met Olivier Palmieri, Director of XR Workshop & Game Director at Ubisoft Montreal, to discuss Virtual Reality (VR) and Augmented Reality (AR), including their latest developments and their current and future applications as learning tools. Here is the first interview of the series.
Olivier has been in the video game industry for over 20 years. He has worked on various projects including the Rayman series, Ghost Recon, Assassin’s Creed and Far Cry. He is the creator of Ubisoft’s first virtual reality game, Eagle Flight, which won a D.I.C.E. 2017 for its technical realization in immersive reality.
We hear more and more about the potential of augmented reality (AR) for training. Is it better than virtual reality (VR)?
I would not say that one of the two technologies lends itself better to training than the other. They each have their advantages and disadvantages, but above all, they are relevant in different cases. The media likes to give the impression that AR and VR are at war, that one will eventually take over the other, but they are two sister technologies. They can also be experienced in the near future with the same pair of glasses that let more or less light to pass through to see our immediate environment.
VR cuts us off from the real world, and we are projected elsewhere, while AR adds information over our real environment. AR and VR are not competing, but rather complementary to each other; that’s how you should see them, in my opinion. For example, in the context of a training program, one can use the AR for the learning part “out of context” and then use the VR for the practice “in context.”
In training, in which cases is it relevant to opt for AR and in which cases is it better to opt for VR?
The advantage of AR is that we are not cut off from the world. We stay in our immediate environment enriched with information, characters, and explanations that enable us to learn about where we are or the elements that surround us. One can learn, for example, to troubleshoot a plumbing pipe by seeing explanations superimposed on the pipe in front of them. In another context, a group of people meeting around a table each having their AR glasses can bring up virtual information on the table; everyone sees the elements and can interact with them.
On the other hand, if we want to learn to master tasks in an environment that is not immediately accessible, we will have to project ourselves virtually, and in this case, we will use the VR. It’s the same thing if you have to train in a high-risk environment, the case of a firefighter, for example. The fact of being projected into another world, into the world in which we will have to act eventually and to live an immersive experience brings a real interest in learning.
What exactly do you mean by “real interest in learning”?
When we have lived something, we memorize it better and have a deeper understanding of it. To have experienced it, learning that is done in VR is worth a thousand words. I think it goes beyond what any other media can offer. I will tell you an anecdote that I share often. During my work, I had to visit a game developer. When I arrived at their office, I had a strong feeling of déjà vu; I said to myself “I am certain I’ve been here before,” but I’ve never been there prior… I did not understand what was going on. I recognized things; I understood the dimensions, the spaces; I wasn’t even thinking, it was my body, my brain that told me unconsciously: “Yes, you’ve already come here, you recognize the location, you recognize the spaces.”
Afterward, I remembered that I had visited this place in a VR environment; it had thus imprinted memories, an understanding of the space in my mind, in my subconscious, which had resurfaced. If I had seen a picture or had the place been described to me, my brain would not have told me the same thing subconsciously. I am quite convinced of the effectiveness of VR for training, to learn things and imprint them in our memory.
I have another anecdote. There is a long-standing virtual experience called Titans of Space that allows us to explore space and better understand the solar system. At a certain point in the experiment, we were led to compare the size of the Sun with that of other supergiant stars in the universe. You had to bend your head backward to begin to perceive the entirety of each star in relation to the Sun, which is already in our field of vision. This experience, which I completed a long time ago, I remember it today as if it was done yesterday! I remember the immensity of such and such star in relation to our Sun. If I had read in a book, “this star is a million times bigger than the Sun,” I would have not remembered the number. Besides, I really do not remember the number; what is essential is that I remember the size of the star compared to that of the Sun. That is imprinted in my memory! There is no limit to the learning that can be done with this type of VR… we can go to Mars as well as the summit of Machu Picchu!
We talk a lot about artificial intelligence (AI). Does the AI have something to contribute to VR in the coming years?
Entirely and on a multi-level, but I would focus on one example: a research that I launched as part of XR: Workshop. We collaborate with a group of Concordia University students who want to work on better understanding AI and its surroundings. I asked them to try to design virtual characters “able” to create a social connection with the user. It’s important to know that VR controllers and headsets detect motion and orientation in space with great accuracy: they detect movements of less than a millimeter and a quarter-degree orientation change.
My goal is to explore the possibilities of AI in relation to VR to develop virtual characters, historical figures or known individuals for example, who can look at and analyze how the human user acts. With AI, the “character” could analyze the behavior of his/her interlocutor by asking itself: “Is he/she looking me in the eye? If so, which eye? Does he/she sometimes look elsewhere? If so, where? How does he/she react? How does he/she speak? Is he/she approaching? Going away? etc.”
Would virtual characters react differently depending on each user?
Yes. Each user would have a unique experience, different from others. A virtual Cleopatra could talk to me about different things depending on whether I look her in the eyes, if I look at a pyramid or if I look at the Sphinx. In the latter case, she would say: “I see that you are looking at the Sphinx, do you know that the Sphinx was built in such and such way…” To another visitor who would not have looked at the Sphinx at that moment, she would not have necessarily said the same thing. If I spent my time looking elsewhere, she might ask me, “Is it in your habit to avoid looking at the people you are talking to?” This kind of recognition of what we do, the idea that suddenly an AI character has somehow a little awareness of who we are, the way we react, it would create a real connection, a social dimension.
It would be exciting to do that, including meeting historical or known characters. It’s somewhat of a dream that I have too, to meet in VR a Cleopatra, an Einstein, a Marie-Curie. I would like to have as professor Marie-Curie or Einstein; that they explain to me how they made their discoveries, that they teach me things. We would not have just a video, a character who speaks in a vacuum, but one that recognizes us and reacts in relation to that. Also, here too, it creates a social connection.
The idea of creating a social connection often comes up in your discussions about VR…
We often hear that the VR it closes us in, it makes us antisocial, it cuts social ties … but that’s something I do not believe. Moreover, when Facebook bought Oculus, Mark Zuckerberg said of the VR that it will become the most social platform we have ever had in the world, mainly because through it we will be able to see each other, see virtual beings who will speak to us, who will recognize us, who will interact with us, etc. This aspect will not come immediately, but I think it is highly interesting. And AI, which is bubbling at the moment, particularly in Montreal, is at the heart of this development. And, in this case too, either we remain a spectator, and we watch all this develop, or we jump on the train!
In addition:
