The Future of Virtual Reality
Part Two of a Conversation with
by Janice J. Heiss
February 25, 2003
Jaron Lanier is well known among developers as the co-inventor of
"virtual reality," a term he coined in the 1980s. Renowned as a
composer, musician, and artist, he has taught at many university computer
science departments around the country, including Yale,
Dartmouth, Columbia and Penn. He recently served as the lead
scientist for the National Tele-Immersion Initiative, which is
devoted, among other things, to using computers to enable people
in different cities to experience the illusion that they are in the
same room at the same time.
In this, the second part of a two-part interview, we asked him to
talk about virtual reality, tele-immersion, and JavaTM technology.
You are known for claiming that virtual reality (which you are
credited with naming) would eventually enhance our lives. How's
About ten years ago, I predicted that virtual reality would be
accessible to consumers by about 2010. I still think that's true.
Virtual reality is a somewhat broad idea, and the definition isn't
fixed. It means different things to different people. As used in industrial
technology, there's no question that virtual reality has already been a
success. You can't buy a car today that wasn't designed using it.
And you can't put gas in that car that wasn't made out of oil that
was discovered using virtual reality through an oil field simulation.
Most new drugs are made in a process assisted by virtual reality. There are many other examples.
"What I envision is not so much a pre-programmed virtual world
that you might play as a game, but rather a virtual world that you
can change from the inside that people use as a form of expression
in which they're creating things together."
- Virtual Reality Pioneer
What most people are curious about, though, isn't so much these
industrial uses; rather, they want to experience some new level of cultural
expression that arises out of virtual worlds. And what we currently
have, in this regard, is the video game world. The difference
between video games and my sense of what virtual reality would
be like relates only partly to the intensity of the experience.
Certainly we'd like to feel like we're inside the virtual world, and
that world would be more vivid and fluid and detailed -- more
computationally intensive -- than what we see today on PlayStation
games. But the main difference isn't any of that stuff.
The main element lacking in video games (compared to what I hope
we'll see in virtual reality) is an expressive power. And so, what I
envision is not so much a pre-programmed virtual world that you
might play as a game, but rather a virtual world that you can
change from the inside; a world that people use as a form of expression, in
which they're creating things together. Just as people make up their
own Web pages, they would make up little realities and visit each
other's realities, or co-create them. And I think that level of activity
would give rise to really, really wonderful new sorts of human
relationships and experiences. I still believe in that.
Jaron Lanier and Sun Microsystems
In February of 1998, Sun acquired the patent portfolio of VPL
Research Inc., Jaron Lanier's former company that pioneered the
development of virtual reality. At the time Lanier remarked, "I'm
delighted that Sun has acquired VPL's assets. Sun's commitment to
open systems and the Java paradigm will provide a superb context
for the formulation of competitive strategies by both VR users and
developers. The next generation of applications will have to deal
with a level of complexity that other leading platforms cannot
address. Virtual reality-based applications will be needed in order
to manage giant databases and networks, advanced medical
imaging, and fast turn-around mechanical design. And all of these
mega-applications will have to support real time collaboration over
the net. Sun is in an ideal position to enable this new level of
And you know, the problem with digital stuff is always the same:
the hardware gets cheaper and becomes more available, but it takes a
really long time to figure out how to write good software. We still
don't have good personal computer software. So, how long will it
take for us to get good virtual reality software? It's not going to
happen overnight. It's a big project.
Tele-Immersion - The Illusion of Being in the Same Room
You're the chief scientist for the National Tele-Immersion
Initiative, which is connected to something called Internet 2. Tell
us about this.
It's a fun project. The Internet 2 project is a large-scale
collaboration that has provided a laboratory platform for people to
do academic research on things like quality-of-service protocols
and new applications that couldn't be done on the regular Internet.
Another way to think about it is that it provides
universities with their own network at a good price. And as a side
effect, it has subsidized the rise of Napster, which is kind of amusing.
Tele-immersion is the lead application for Internet 2, the driver
application to push Internet 2. So, we would try to experiment to
get tele-immersion to work. I was the chief scientist at Advanced
Networks, which was also the engineering office of Internet 2.
I was charged with coming up with some applications that simply
couldn't be done on a lower level of Internet access. So, we had our
own proprietary net, which was called Abilene. And we could
write our own protocols on it and route it as we wanted. We could
do all kinds of things to it. And that created an ideal opportunity
to try out the next level of human experience over the net, which
has been awkwardly called tele-immersion. We need a better name
for it. I don't like the name, but at any rate, tele-immersion is a
form of virtual reality that, instead of putting you in an imaginary
world, connects two people -- who are significantly far apart from each other in the real world -- and gives them the illusion
that they're in the same room.
Tele-immersion is ultra-realistic, full-scale 3D video-conferencing
in which people feel like they are actually together. It's really hard to do
on a lot of levels. The Internet implementation side of it is really
hard, because it requires an extreme level of performance. And
even the slightest latency issues become extremely important. It's also
really difficult to do on a sensing and display level; it's not easy to
sense the world in this sort of three-dimensional way, and it's hard to recreate it realistically.
So, there was a consortium of a dozen universities that worked for
a few years to make a first demo. And the first demos weren't perfect,
but they looked pretty good.
And it's gotten better and better. It's not ready for prime time by
any means, because it's still an extremely expensive laboratory
technology. But I believe that, eventually, you will be able to have
the illusion of being present with another person, to the degree that you
can walk around them and shake their hand. Though this will take
a while, soon you will see a substantial improvement in the sense
of contact you get from video-conferencing and telephones.
And I think it will be worth it. I think it will be a reasonable
alternative to air travel for a lot of things -- for meetings, and for
helping keep families together when people move around. I think it
will be a useful addition to the means we have to connect with
other people, which is really what this is all for. I'm really happy
with how it turned out.
"Tele-immersion is ultra-realistic, full-scale 3D video-conferencing
in which people feel like they are actually together."
- Virtual Reality Pioneer
Consulting with Hollywood
You were a consultant for some of the futuristic aspects of the
movie "Minority Report," which depicted a frightening world in
which advertising signs could recognize you as you walked past them,
and remember what you had previously bought and then make very
personalized sales pitches to you.
It was kind of funny what happened. Because
Hollywood people mostly respond to what they can see, rather than
abstract visions, all the information technology that actually made
it into the movie was very strongly based on demos that had
already been created -- some of them quite old. Some of
the things, such as the interface gloves, go back two decades or
more. There's a literal-mindedness to Hollywood
that's hard to overcome. I presented all sorts of ideas for what
information technology might look like in fifty years, but the least
noble of these were the only ones that stuck.
Nowhere in "Minority Report" do we see people using technology
to interact creatively with each other. We don't see children
inventing their own technological culture, which is a reality today.
Philip K. Dick, who wrote the short story on which the film was
based, didn't live long enough to see this reality. If he had, I
believe he would have written a different kind of story.
The characters in "Minority Report" are either consumers (who are
used by the advertisements, the animated cereal box and so forth) or elite
controllers (the pre-crime officers who get to use a zippy
interface). Three-dimensional displays are used for recorded
images, but not for live contact.
As you can see, I'm a little bit of a Hollywood antagonist. I'll work
with them, but I'm not too impressed.
Musical Instruments as Ideal Interfaces
You once said, "If there's any object in human experience that's
a precedent for what a computer should be, it's a musical
instrument: a device where you can explore a huge range of
possibilities through an interface that connects your mind and your
body, allowing you to be emotionally authentic and expressive."
I still believe that. I think musical instruments are the only good user interfaces that have ever been designed. I have an enormous
collection of them and I'm always trying to learn to play new ones.
I think that the musical instrument is the precedent for how
computers should be designed. So, having said that, somebody
might ask me, "Well, how could you specifically improve a
computer based on your experience with instruments?" And here I
have to say I'm still working on it.
Can you give us some clues?
Essentially, you want to discover your own high-level
variable when you're controlling a complex system. Here's what I
mean by that. If you're working with an instrument, you gradually
train your own body and mind to the instrument. Your own body
and mind are pattern recognition engines and you're always
training yourself to the world and adapting to the world in
wonderful, complex, subtle, and deep ways. So, you start doing
that with the instrument. But each musician finds a different set of
high-level parameters that they're continuing to refine on,
sometimes consciously and sometimes not consciously. And the
areas of action for them that are continuing to change and grow
will vary from musician to musician. So, they become adaptive in
some ways and less adaptive in other ways. And that's how people
become specialized in their connection to the world.
"I think Java technology is fantastic and Java 3DTM deserves a lot
more attention at this point."
- Virtual Reality Pioneer
That's how people become good at things -- and some people are
perhaps inherently good at some things. We have to be able to
recreate that effect with computers. And right now, we can't do it,
except to a limited degree; of course, there are some people who are better at
playing fast video games and that sort of thing. But there are just so
many things wrong, it's hard to know where to begin. The first
thing is that, in humans, the mind and the body aren't really
separate. There's a continuity. And there's a continuity between
step-by-step conscious thinking and physical action and emotion.
And you can think of physical action as a form of intelligence in
which your brain has created patterns that take into account
implicit patterns in your body's physiology. And that's a very
sophisticated form of computation, which shouldn't be dismissed at
Similarly, emotion can be thought of as an extremely sophisticated
form of computation in which you're looking at the very highest
level patterns of life experience. And computers continuously draw
us down to the sort of tweedly tedious level at which these things
that we really excel at can't be relevant. It's extremely
frustrating. Through their physicality, musical instruments allow us
to incorporate those aspects of ourselves that involve high levels of
pattern recognition. And we have to get to that point with
computers some day. I think it's going to be hard. I don't expect
this to happen overnight at all. But that's certainly what I'm going
for. I don't know if I'll see it in my lifetime.
You once said that, while it is commonly thought that small
start-ups come up with new ideas, Sun is noteworthy as a big
company that created the JavaTM programming language. How is
Sun doing at applying Java technology and other ideas to virtual
I think Java technology is fantastic and Java 3DTM deserves a lot
more attention at this point. Having more layers of software
systems distributed is a good thing, and having more open systems
is a good thing. And there's another element, which is really important
to me -- working on biological metaphors. If you look at the
viruses that are floating around and how expensive they are, the
kind of monoculture we have on PCs is a terrible security risk. What you
really want is some balance between sanitization and variety so
that you have robustness. I think Java systems achieve that balance
much more effectively than anchored OS systems. And I think
that's going to be a surprise driving software at some point in the
Jaron Lanier's Home Page
Brief Biography of Jaron Lanier
One Half of a Manifesto
Slashdot Discussion of Jaron Lanier
The Poetry of Programming
Coding from Scratch
Java 3DTM API