Virtual Retinal Display (VRD)
            Technology

We would like to advise you to read the glossary first, as you may encounter some difficulties considering medical terminology.
This presentation is broken into the following sections:



Overview
    The VRD was invented at the University of Washington in the Human Interface Technology Lab (HIT) in 1991. The development began in November 1993. The aim was to produce a full color, wide field-of-view, high resolution, high brightness, low cost virtual display. Microvision Inc. has the exclusive license to commercialize the VRD technology. This technology  has many potential applications, from head-mounted displays (HMDs) for military/aerospace applications to medical society.
    The VRD projects a modulated beam of light (from an electronic source) directly onto the retina of the eye producing a rasterized image. The viewer has the illusion of seeing the source image as if he/she stands two feet away in front of a 14-inch monitor. In reality, the image is on the retina of its eye and not on a screen. The quality of the image he/she sees is excellent with stereo view, full color, wide field of view, no flickering characteristics. Using the VRD technology it is possible to build a display with the  following characteristics:

Glossary

Cortex
      The cerebral cortex, the outermost layer of the cerebral hemispheres of the brain, responsible for all forms of conscious experience.

Retina
    A multi-layered sheet of nerve cells at the back of each eye which converts light into electrical signals that are transmitted to the brain through the optic nerves and tracts. Below is a picture of the anatomy of the retina showing layers, vessels and nerves.

Image courtesy of CIBAVision
 

Photoreceptor
    One of the 125 million nerve cells in the retina of each eye that emit electrical signals when activated by light of a particular wavelength. There are two types of photoreceptor cells in vertebrates: rods, used in low light, and cones, which respond in brighter light and to color.


Image courtesy of Howard Hughes Medical Institute

The intricate layers and connections of nerve cells in the retina were drawn by the famed Spanish anatomist Santiago Ram?n y Cajal around 1900. Rod (a) and cone (b) cells are at the top. Optic nerve fibers leading to the brain may be seen at bottom right.

VRD
    Is a new display technology that scans modulated low energy laser light directly onto the viewer�s retina to create a perception of a virtual image.



Human Visual Pathway

Image courtesy of Howard Hughes Medical Institute

As shown in the above figure the reflected rays of the pencil pass through its lens. The inverted image of the pencil is directly projected onto the retina. The rod and cone cells of the retina are responsible of tranmitting signals to the brain through the optical nerve.
Another important factor in seeing is the stereo view capability. The images presented to the human vision must provide those characteristics that make them appear stereoscopic. That way, the viewer can optically calculate the distance of far seen objects. "The dumber the animal, the smarter its retina," observes Denis Baylor of Stanford Medical School.
Below is a graphical explanation of how the human vision generates stereoscopic images.


Image courtesy of Howard Hughes Medical Institute

The two eyes provide slightly different views of the  same scene. Information from the left visual field goes  to the right side of the retina in both eyes. At the optic
 chiasm, half the nerve fibers from the left eye cross  over to the right hemisphere and the rest stay  uncrossed, so that all the information from the left  visual field ends up in the right hemisphere.  In this way, a given hemisphere gets information from  the opposite half of the visual world�but each  hemisphere gets input from both eyes. Thus, the future VRD product must have stereoscopic capabilities.



How the VRD works?
Before we dive into the VRD technology take a look at ( http://www.cs.nps.navy.mil/people/faculty/capps/iap/class1/disptech/disptech.html  ) the way most of our monitors work.
    In a similar way the source image is projected directly onto the retina instead of the back of the screen. While in the Flat Panel Display pixels are projected from a highly integrated matrix of electronically controlled light valves or light-emitting  elements, the VRD technology eliminates any screen outside of the eye and addresses the retina with a single stream of pixels.
    Unlike CRT monitors, the VRD has no phosphorus persistence but, depends on the light-gathering properties of the photoreceptors and the temporal integration properties of the visual system. Therefore, the interactions between the VRD delivered light and the psychophysical image quality created by the brain are of extreme importance for future enhancements.


Image courtesy of Microvision

As shown above, the source image is directed into the drive electronics device where the green, blue, and red light are modulated. Then each pixel passes through the horizontal and vertical scanners (one moves from left to right and the other from up to down) and produces a rasterized image onto the retina. The retina has no persistence like phosphorus. Moreover, there is no flickering and the produced image is very bright with very high resolution. The time each pixel is projected onto the retina is very small (30-40ns). Furthermore, these devices consume very little amount of energy. They also provide a wider field of view.
A very sophisticated representation of how the beam is projected through the pupil can be found in the HIT Lab of Washington University (VRD animations).

System Modules

    Drive electronics
    The drive electronics control the acousto-optic modulators that encode the image data into the pulse stream. The color combiner multiplexes the individually-modulated red, green, and blue beams to produce a serial stream of pixels, which is launched into a singlemode optical fiber to propagate to the scanner assembly. The drive electronics receive and process an incoming video signal, provide image compensation, and control image display. For VGA projection, the electronics process over 18 Mpix/s. The virtual retinal display is capable of providing UXGA resolution of 1600 x 1200 or 115 Mpix/s.
    Light source
    The light source module contains laser light sources, acousto-optic modulators to create the pulse stream, and a color combiner that multiplexes the pulse streams. To provide sufficient brightness, full-color displays suitable for outdoor, daylight applications incorporate red diode lasers (635nm), green solid-state lasers
(532 nm), and blue solid-state or argon gas lasers (450-470 nm range). Systems designed for indoor use can incorporate LEDs; red, blue, and green devices currently under development for such systems are being tested. Generally, the energy levels are on the order of nanowatts to milliwatts, depending on display requirements. The levels of light involved are well within laser safety standards for viewing, as confirmed by analysis.
    Scanner assembly
    The scanner assembly contains two scanning mirrors. One 24 mm x 6 mm x 6 mm scanning mirror sweeps the beam horizontally at a high frequency, typically between 15.75 kHz and 18.9 kHz for 60-Hz non-interlaced refresh rates. This corresponds to one-half the VESA monitor-timing standard since the retinal scanning display can process and display pixels bidirectionally. A second scanning mirror sweeps the beam of laser light vertically at 60 Hz to complete the raster image.
Microvision is currently developing a microelectromechanical system (MEMS-) based full image scanner capable of bi-directional scanning. Unlike MEMS-based spatial light modulators used for projection applications, the millimeter-sized mirror in the virtual retinal display will provide the scan angles and rates necessary to provide the full raster image. The need to fabricate only one mirror significantly reduces the development effort for higher resolutions and associated yield problems. It also eliminates the potential for "dead" pixels due to inoperative mirror elements.
    Pupil expander
    Nominally the entire image would be contained in an area of 2 mm2. The exit-pupil expander (not shown in Figure 2) is an optical device that increases the natural output angle of the image and enlarges it up to 18 mm on a side for ease of viewing. The raster image created by the horizontal and vertical scanners passes through the pupil expander and on to the viewer optics.
    Viewer optics
    The viewer optics relay the scanned raster image to the oculars worn by the user. The optical system varies according to the application. In the case of military applications such as helmet mounted or head mounted display optics, the system incorporates glass and or plastic components; for medical applications such as image-guided surgery, head-mounted plastic optics are used. In industrial or personal displays, the optics might be a simple plastic lens.



Characteristics
 
Resolution The VRD produces very high resolution images. Its resolution is limited only by diffraction and optical aberrations in the light source and not by how small one can make an individual pixel element in a large array of pixel elements.
Contrast  Ratio The brightness of the VRD can be increased to very high levels or decreased to minimal levels. As a result, its contrast ratio is inherently high and far greater than that of standard flat panel displays or even conventional CRT monitors.
Luminance  Conventional electronic displays do not emit (or transmit) substantial amounts of light energy. As a result, they are primarily used in controlled lighting environments, and it is difficult to see them under bright ambient light conditions such as exist outdoors. In a see-through mode, the VRD can be controlled to allow the user to see an image that matches the brightness levels of the ambient light conditions.
Color Range CRTs and miniature FPDs are able to reflect only a portion of the total palette of colors visible to the human eye, and are limited in the degree of saturation they can achieve. Because red, green and blue light sources used in VRD technology emit highly saturated, pure color, the VRD can produce a range of possible colors and color fidelity superior to any other electronic display technology.
Power Consumption Because they do not efficiently convert electrical energy into light energy, both back-lit FPDs and CRTs draw substantial power and are among the biggest battery consumers in portable devices that use them. As a result, their brightness is relatively low, because most of their energy input is wasted. VRD technology, by contrast, conveys virtually all of its generated light onto the retina, allowing bright displays with minimum power requirements.

Scalability
VRD technology is capable of improving performance more rapidly and with significantly lower research and development and tooling requirements than competing technologies.
Range of Application Because the VRD delivers a truly general-purpose solution, without performance compromises or tradeoffs, it is appropriate for a broader range of applications than competing display technologies. This means that it can potentially achieve greater economies of scale necessary to drive costs downward and increase its appeal versus other display solutions.
Cost Current flat panel displays use exotic and costly production techniques involving complex assemblies and critical tolerances. In contrast, the basic design of the VRD consists of subsystems that are very simple in their design and largely make use of established optical and electronic technologies. Investment in specialized manufacturing equipment is  not required. Ultimately, VRD devices will be mass-produced at low cost.



Why there is a need for VRD?
    We need a cost effective and portable device that will consume very little energy and low bandwidth. We must provide the user (e.g. pilot) a wider view. Moreover - the pilot - must be able to read the plane's intruments even having the sunrays coming directly onto his eyes. This kind of view must be stereo capable and full color. Furthermore, we  need to superimpose information on the the user�s view. Thus, he/she won't have to take their eyes away from the target, while they will be able to read information projected onto their retina.



Who deals with this issue?
    Human Interface Technology Lab at the University of Washington participating in the Integrated Small Precision Optics Manufacturing Technology Consortium (ISPOMT) is developing the display. MicroVision Inc. (Seattle) funds the research and holds an exclusive license to manufacture and distribute the VRD. US Naval Research has granted a research for Interactive VRD (IVRD). The U.S. Army's Aviation Applied Technology Directorate (AATD) has awarded virtual retinal display  manufacturer Microvision Inc. a contract to produce an advanced helmet-mounted display and imaging system for use in the Virtual Cockpit Optimization Program (VCOP). The primary subcontractor in the project will be Boeing Co. research and development arm Boeing Phantom Works. The system will integrate Microvision's virtual retinal display technology (see Retinal Scanning Leads HMD Race) with Boeing software to provide information to rotorcraft pilots.



US NAVY (IVRD) Project
    Participating in the Integrated Small Precision Optics Manufacturing Technology (ISPOMT) consortium, the HIT Lab is designing and developing an interactive VRD for US Navy pilots. The Office of Naval Research is developing the IVRD device under the Grant #N00014-96-0008. With this device the pilot will be able to see better in bright light, will be able to have a stereo display and he could interact through this device by pinpointing on the display with his fingres. This device will be very light and will consume very little energy.  The VRD is the only display technology that has sufficient luminance to be used as an augmented display over the pilot's real world view in bright sunlight.


Image courtesy of HIT Lab, Washington University
 

The goals of the IVRD project are:



Prototypes and Emulators
 
Image courtesy of HIT Lab, Washington University Image courtesy of HIT Lab, Washington University

Prototype Characteristics:     640x480
                                            60 Hz
                                            50 degrees f-o-v
                                            RGB
Emulator: 
The emulator system requires a scanned light sources of the emulator to be modulated by passing the light through a 35mm slide. This provides us with the capability to vary the spot size at the retina, to vary the spacing between 
horizontal lines, to vary the spacing between pixels etc.
More at  HIT Lab



Applications

Head Mounted Displays (HMDs)
    "Helicopter pilots require information to support time-critical (and often life-and-death) decisions," Swanson said. "If that information is presented in a graphical and intuitive fashion, it reduces the pilot's workload and can enhance visibility in degraded conditions. A helmet-mounted display capable of presenting full-color graphical information in both day and night flight operations has been the missing link to creating an effective pilot-data interface. That ultimately could save both lives and money."
    Rick Rutkowski, Microvision's president and chief executive officer, said, "The Army has a powerful vision: the ability to overlay flight reference data, sensor imagery and weapons symbology on [images from] the outside world." Such a versatile display capability is expected to provide a significant performance boost to both aircraft and pilot. "When you can also enable a pilot to see the normally invisible 'bloom' of a radar signature, or to project a 'pathway in the sky' in front of him, and to superimpose wireframe or 3-D imagery onto the terrain, it becomes even more powerful," he said.
    Microvision delivered a monochrome green binocular HMD system to the Army with a total field of view of 52� horizontal by 30� vertical with 1716 horizontal pixels by 960 display lines, and 1470 ft-Lm displayed raster luminance at the eye (see Figure 3). The system weighs approximately 2 lb, including two display engines (one for each eye), the helmet-mounting structure, and the optics to provide the large display format. The system will project the images in a "see through" fashion �the pilot will still be able to see the background scene,  but has the option of focusing in on the presented information.

    According to Rutkowski, the Army's vision of the virtual cockpit also includes a "what you see depends on where you look" concept. As the pilot looks up and out of the cockpit, various types of targeting, navigational or terrain overlays would appear. When pilots look in a downward direction, they may see "virtual" instruments projected onto the eye that literally replace many of the existing dials and multifunction displays that are in cockpits today. Microvision has modularized the system designed to permit delivery of a full-color 1280 x 1024-pixel biocular head-mounted system three months later, as well as an HDTV (1920 x 1080 pixels) HMD to the Air Force by the year 2000 (see Figure 4).
 

    Wearable "augmented reality" displays Incorporated into eyeglasses, goggles or helmets, VRD technology will display an image that doesn't block the user's view but will instead superimpose a high-contrast monochromatic or color image on top of it. This ability can enhance the safety, precision and productivity of professionals performing complex tasks.

Interactive VRD
    Wearable three-dimensional / interactive displays VRD technology can be incorporated into eyeglasses, goggles or helmets to create a  stereoscopic, 3-d effect. These compact, high-resolution displays can further enhance the visual realism of the interactive experience to make the simulated environment more engaging.


Image courtesy of HIT Lab, University of Washington

Hand-held displays
    Hand-held two-dimensional displays VRD technology will be integrated into cellular phones and pagers, allowing users to tap into business networks or the Internet to view e-mail, web pages, faxes and files as if on a full-size desktop monitor. This ability affords greater convenience and utility to users of these devices.

Image courtesy of Microvision

Medical applications (as referenced by Microvision Inc)
    It has been estimated by the Community Services for the Blind and Partially Sighted that 2% of the population in the United States have low vision (20/400 or worse, legal blindness being 20/200). People with low vision are not completely blind, but many are unable to hold a drivers license and have a great deal of difficulty reading or watching television. Possible causes include damage to the retina - central or peripheral field degradation - and damage to the optical pathway - corneal damage/malformation or lens damage. The HIT Lab develops these VRD that can provide more visible images to people with low vision. Microvision capped 1998 by signing a contract with the Wallace-Kettering Neuroscience Institute (Dayton, Ohio) to collaborate on headgear for neurosurgery.
    In its test screenings, "Patients without retinal damage saw moving particles throughout their field of vision, but patients with areas of retinal damage were actually able to paradoxically see their own blind areas and localize them," a Microvision spokesman said.
    "Using a pen-based computer program, patients were asked to trace the borders of any visual disturbances," he said, "in effect creating a digital map of their own retina, which accurately detailed the locations of damaged tissue. The images were then compared with the 'gold standard' � photographs taken from a fundus camera � with outstanding results." A fundus camera captures images of the internal surfaces of hollow organs like the eye.
    The VRD-based workstation can "screen for retinal damage in asymptomatic patients within virtually all of the visually significant retina," William Freeman, professor of ophthalmology and administrative director of the retina service at Shiley, reported at the meeting. "This rapid, non-invasive screening test has the potential to be an excellent addition to the clinical tool set of primary-care health providers to rapidly screen for retinal damage and diseases. It will also allow for community screening of underserved populations to detect vision loss." In the past year, a Shiley spokesman said, the workstation screened 58 patients with a variety of retinal diseases, including AIDS-related CMV retinitis, diabetic retinopathy, ocular melanoma, blood-vessel disorders, retinal detachment and macular degeneration.



The Future of VRD Technology
      Future systems will be even more compact than present versions once the MEMS-based scanners are incorporated. Edge-emitting, super-luminescent light-emitting diodes (SLEDs) and miniature diode lasers under development will allow direct light modulation. In conjunction with application-specific integrated-circuit technology, these devices will permit the direct fabrication of a VRD display engine incorporating the electronics, light sources, and scanning assembly, all in a compact, hand-held, battery-operated package.
      The ultimate goal for the retinal scanning technology is a lightweight eyewear package as below.


Image courtesy of Microvision

The approach can also be adapted to image projection systems. The applications for VRD technology are varied�HUDs, color projections systems for entertainment or flight training simulators, etc. A key area for continued development is an image display system that can augment and enhance a person's task performance.



References
University of Washington HIT Lab (IVRD)
The Urgent need to use both eyes
How we see colors
Visual Pathway
Microvision
Microvision's Award
Microvision and Boeing
Retinal Scanning leads HMD
Ophtalmology