Q. What are Virtual Worlds?

The key to understanding virtual worlds is that there’s nothing necessarily new or high-tech about this idea. It’s core to human existence.

When you dream, you’re in a virtual world. When you read a book, you’re building a virtual world in your mind based on the author’s prose plus your imagination. When you talk to someone about another time or place or person, there is an encoding of the speaker’s mental world into words and sounds and then a decoding of those words and sounds into the listener’s internal world.

These mental worlds don’t always (and often don’t) match up exactly, but we do our best to understand one another.

Computer mediated virtual worlds work the same way, except there is a computer in the middle of the conversation.  And instead of words, it’s most often 3D graphics and sound that are used to deliver the information, closer to movies than books.

Q. What is Virtual Reality?

The term is widely attributed to Jaron Lanier. Virtual Reality is a way to translate information from a computer’s internal models, essentially numbers and instructions, into the most natural forms humans can perceive: sensory inputs like vision, hearing, and touch, and outputs like speech and touch.

By using these most human inputs and outputs, VR can be extremely natural and therefore provide very high bandwidth in and out of your mind. It takes very little work for our brains to interpret a 3D scene. On the other hand, it’s a very literal representation of some world, whereas a book is more abstract, where the reader fills in the details. So in some sense, a book carries more information in fewer bits, though it’s a “lossier” translation requiring a very sophisticated computer (your brain) to decode.

VR typically is portrayed with a Head Mounted Display (HMD) that shuts out the real world and immerses a person in a fully simulated environment. That’s a bit misleading, it turns out, because a person is still very aware of their own body pose (proprioception) and their ever-present sense of self. Problems in VR generally stem from mismatches between the natural senses and the artificial representations. And in many ways, the HMD detracts from the naturalness of virtual worlds by isolating a person from social contact, e.g., seeing other people’s eyes, or holding hands.

On the other hand, the VR experience must be tied to a participant’s actual point of view, that is, where they are and what they’re looking at. So it is by definition a very individual experience. But it’s one that can be shared with others through even more computer mediation, e.g., using avatars to represent other people. There is quite a lot of technology in the middle just to make it seem to disappear, as when two people talk to one another.

In other embodiments, VR comes as a projected environment, where people wear very light to no hardware and the graphics and sound come from the walls of the room (e.g., CAVEs). Multiple people in the same environment can see each other directly and easily in those environments. This kind of VR is most like a Disneyland ride (e.g., Soaring over California is the most literal version of this, but Pirates of the Caribbean is too). All of Disneyland, for what it’s worth, is a virtual reality experience rendered in solid matter.

Q. What is Augmented Reality?

This term is attributable to Ron Azuma. It works in many ways like Virtual Reality does, rendering unique images for each eye that are reconstructed in your brain as mental virtual worlds of 3D scenes. But, by definition, it overlays this new information on top of the real world instead of replacing the real world with a fictional one. Since our brains are already processing the natural world, well naturally, adding information that looks physical becomes very easy for us to consume and use in our daily interactions.

However, it has technical challenges beyond VR, given the need to know exactly where a user is looking while better understanding the shape (and often lighting/colors) of what they’re looking at in order to register the virtual imagery onto the real. Without that registration, results can be poor. Even with ideal registration, many AR technologies are only capable of rendering “ghosts” that are 50% translucent, due to how their optical components mix natural and computer generated light.

The promise of AR when it reaches maturity is that we can walk anywhere in the real world and experience an overlay of interesting content. Real things can be made smarter through AR interactions, virtual things that can me made to seem physical and almost tangible, and smart objects (e.g., phones) can have useful augmentations, like user interfaces floating in space.

Just how much of the real world is modified or replaced by augmentations is entirely up to a given experience. Some AR, like Pranav Mistry’s “Sixth Sense” is meant to be subtle. We may just want to see a recommended book “glow” for attention. Other AR experiences projects a musical show on the side of an entire building, making the building come alive and dance.

Q. What is the difference between AR and VR?

In VR, it’s possible to put big displays like LCD screens right in front of your eyes, because we don’t need to see through them. In AR, where we generally want to have a clear view of the real world, we have to use extra tricks to move the displays out of the way and bend the light (glass waveguide, refractive, bulk optic, fiber optics, etc..) into your eyes. But bulky VR displays can also add cameras to turn into AR displays; and AR displays can add opacity filters to turn into VR displays.

Both AR and VR want to track the position and rotation of your head in space, perhaps even track your eyes for best results. Audio rendering is almost exactly the same for both kinds of experiences, either using speakers, headphones or bone conduction.

One can imagine the ultimate AR/VR device as a pair of contact lenses that generate imagery and beam those rays of light straight back to your retinas. In this scheme, if you close your eyes, you’re in VR. If you open them, you’re in AR. It’s the same exact hardware, with your eyelids as part of the “technical stack.”

Short of that eventual future, in a CAVE or “illumiroom” type environment we get 3D graphics properly overlaid on top of physical structures like walls or furniture. We can see our own bodies clearly in this mode. So it’s closer to AR than wearing an HMD, yet the environment can be completely fictional, like VR.

In the end, the key difference between AR and VR is really whether information is painted on top of a purely fictional replacement environment or added on top of the real environment or mixed somewhere in-between.

Q. What is Mixed Reality?

Mixed Reality is a concept attributed to Paul Milgram and Fumio Kishino. It refers to the spectrum of computer mediated experiences ranging from those built on real environments at one end, to the entirely virtual ones at the other end. So it technically includes both AR and VR and everything in-between, and is a much more general purpose term. People often use MR to describe experiences that aren’t clearly AR or VR where elements of both are intermixed.

Q. Is Google Glass AR?

It would be more correct to call it a Heads Up Display. Putting the virtual image off-axis (up and to the right) makes it hard to overlay virtual information on top of real objects in 3D such that the information is tightly aligned. Having a display for only one eye makes that even harder. However, it does well for putting a small window of text and/or images in your field of view.

Q. Is Oculus VR?

Yes. But Oculus with a Leap Motion sensor or RGB cameras on the front that insert the real environment inside the virtual one, moves it closer and closer to AR. However, in all cases, there is more to be done than just putting goggles over your eyes and tracking your head before we can spend considerable time in VR.

Q. What is Presence?

Presence is the feeling that you are where your eyes say you are. You hopefully feel it now, wherever you are. If you hold up a magazine photo of the Bahamas, you probably don’t believe you’re there at any level of cognition. With VR, you could feel like you’re somewhere else and your senses would begin to believe it, treat it as real. With a really good book or movie, you may feel like you’re there as well, but those tend not to engage your automatic responses quite as strongly as VR does.

In VR, achieving a feeling of presence requires a set of technical achievements, including sufficient resolution of the displays, low enough latency for head motions and wide enough field of view to fully immerse you in a virtual world and not knock you out of the experience or make you sick.

Q. What else does VR/AR need?

Interactivity is fundamental to reality. If you put your hands through a wall or fall through a chair you just tried to sit on, you might question the reality of your situation — or yourself. This isn’t just about basic collision detection and haptic feedback, but being able to affect the world and have it affect you.

There is a virtuous cycle that happens when we do this right. We feel a sense of presence from the immersive displays and low-latency head tracking. We can see ourselves and hopefully our virtual bodies track our real ones properly, so we align with our natural sense of proprioception.

We can interact with the world by touching it, as best we can, and having it touch us. The world is changed by our presence, and the world changes us by our being there. If a virtual tree falls in the forest and we’re in the way, we duck. If we draw some graffiti on the wall, it remains there.

On the other hand, if we break that cycle, we diminish the reality of either VR or AR. Our brains recoil from being tricked and trying to make sense of something that is unnatural. And the result is a loss of reality, and consumers get annoyed and frustrated as a result.