Skip Masthead Links   » search tips  |  Search:     
The Source for Developers
A Sun Developer Network Site
Products and Technologies
Technical Topics
Sun Microsystems
Face to Face with Dr. Ben Bederson

Face to Face

Dr. Ben Bederson, Assistant Professor, Human-Computer Interaction Lab, University of Maryland

interviewed by Jon Byous


[Dr. Ben Bederson] The audience "oohs" and "aahs" as four people move paint cursors independently across a single computer screen, each drawing their part of a child's school house. These four people each have a mouse, and they are driving four simultaneous inputs into one system. Everyone is impressed, even the nine-year-old.

Is there a way that computers can better support people working together, really together? That is the question that Dr. Ben Bederson wrestles with day and night.

His conclusion is that, "people work together best when they are together in the same room, using the same computer."

So Dr. Bederson has devised applications that demonstrate the value of a "zoomable interactive space with information visualization for multiple simultaneous interactive users."

Q How did this idea for your research and development come about?

A People just seem to collaborate, or work together, better when they are in close proximity, seeing each other in person.

Human bodies are extremely powerful input devices. We're more than eyes and ears. We sense the presence of someone, even if we don't see them. We receive a lot of information from small things, like the angle of the listener's neck and position of their eyes.

Humans are very perceptive at minute, subtle differences, especially in face-to-face communication with another person, or working shoulder-to-shoulder.

Q But computers don't support that environment?

A Not yet. But, I can't imagine that ten years from now, computers will be used by only one person at a time. It almost has to be that computers will support multiple simultaneous users. The only question for me is not 'if' but 'when' and 'how.'

We're taking the first steps in answering that.

Q How are you approaching it?

A My research mission is to question the assumptions that computer designers have made implicitly through history in the design of computers. There are two basic assumptions that I am exploring.

The first is the assumption that the correct interface for a computer is a "Windows/Office" style interface. This WIMP (windows/icons/menus/pointers) interface came out of Xerox PARC in the late 70s, it worked, it was successful, and it stuck. But does an interface have to be that way?

The second assumption is that a computer is designed for one person sitting in front of it. In practice, people commonly work together with more than one person in front of a computer at a time. There might be two people debugging, or two people writing something, or one person showing someone else how to use something, or several people being creative together, brainstorming, searching. That's how people work together. And computers don't support that type of collaboration with more than one user.

Of course, there are significant challenges to be solved with two people sitting in front of a single computer. To build a system that supports multiple simultaneous users means solving ergonomic design problems and physical space issues. There's not enough display space. It puts more demands on the computer, and there are command conflicts with multiple simultaneous inputs.

The first way I have addressed these assumptions and problems is to break out of the box. The computer screen is a terrible display device compared to the real world, but we can take advantage of it as a medium by remembering that it has a powerful computer attached to it. So instead of having documents in windows that mimic paper as a direct metaphor, I've taken elements of the user data and laid them out spatially on the screen. This way, we can take advantage of the fact that humans are highly tuned to recognizing motion and understanding visual cues.

The result of this work so far is three related technologies:
  • Jazz is a platform for 2D+zooming object-oriented graphics.
  • MID (Multiple Input Devices) is a toolkit that supports multiple input devices.
  • KidPad is a multi-user storytelling tool for children that builds on Jazz and MID

Q The ability to zoom in and out is an essential feature of your work. Do you use 3D also?

A I opted not to use 3D. For information that is more abstract and semantic, 2D plus zooming may actually be better. And it is definitely less complicated -- both from the implementation side, and from the user's point of view.

With 2D zooming, you can "move" out for an overview of the data and move in for more detail. You still have the simplicity of a WIMP interface, but it doesn't have to be more complex than just clicking on something to go to it.

Of course, there is a place in many applications for WIMP interfaces. But not all of them. The KidPad application doesn't have a single button, pull-down menu, or tool palette. It's all just very smooth interactive graphics in an immersive space. And for the times you do want traditional interfaces, Jazz supports embedding Swing widgets within the zoomable space. So, our goal is to support the best of both worlds, mixing interactive quality graphics with zooming, and traditional interfaces.

Q What is a zoomable user interface best suited for?

A A zoomable interface is very good for the process of storytelling, and making presentations.

A key element of storytelling is spinning a web that shows relationships - being able to relate parts of your content to other parts. So an interface is needed where the contextual relationship between your information parts is an implicit part of the interface itself. And also, storytelling and presenting are tasks where visual aesthetic and entertainment are important components to getting the job done. If you are not engaging your audience, then you are not communicating with your audience. So having an interface mechanism that is engaging is not irrelevant, it's part of the task. It's not a bonus, it is the task.

So all of those requirements have to come together in an interface designed for storytelling and giving presentations, and that's why I have focused on zoomable interfaces.

Q How do you set about building a zoomable interface?

A In a way, a zoomable interface is 2D graphics with this extra parameter of scale. By far, the easiest way to do this is in an object-oriented way.

First, you have one object that corresponds to each graphic element. Then I use 2D affine transforms, already built into the Java 2 platform, as mechanisms to translate, scale, rotate, or shear any object in the space. These objects -- text, images, graphics -- can then be grouped in space in a way you might think of as slides in a presentation.

Then you need a "camera" to move around and look at different places in this space -- to do sophisticated information visualization. And you want to be able to change the "view" of the camera, for example, with zooming lenses, fisheye views, and radar views.

So, we built a platform to do this called Jazz. It is an object-oriented structure -- built entirely on the Java 2 platform -- for building 2D objects with transforms and camera views.

Jazz is the implementation of the Java platform that supports 2D object-oriented zoomable graphics, multiple views, and animation. It is analogous to the Java 3D API, but for 2D graphics. In fact, it uses the Java 2DTM API-based renderer as its functionality base.

Q How does it work?

A The Java 2 platform provides a wonderful renderer and GUI tool kit. But it lacks the data structure for building 2D object-oriented graphics. Jazz is the platform that makes that easy for developers. It is a 2D scenegraph, or hierarchical structure, that sits on top of the Java 2D technology-based renderer.

Using a scenegraph is not the standard way of thinking about 2D graphics, but I think it is important. It is the structure that manages hierarchical relationships among nodes.

Different node types provide functionality, such as grouping, fading in and out, transforming object stickiness, and animation, among others. Then there are visual components that plug into these nodes. They're a very simple class that define a paint method, a bounds method, and a pick method, allowing the object to be plugged into the scenegraph and rendered with all of the power of the renderer.

With this structure of basic visual components, Jazz is very extensible for developers to define their own visual components that will then plug into the scenegraph. In addition, if developers want to add new interactive functionality, developers can extend the node class to define a new node type that will fit into the scenegraph structure.

Q It sounds wildly extensible.

A That's an important goal. Jazz is primarily funded by DARPA for the Command Post of the Future project. We are building a platform that will be used in many different ways. We know from our experience of thinking about these interfaces for a number of years, that extensibility is key. Even though we have ideas about what's interesting, we can't possibly add all of the features that could be used. The best thing that we can do is to provide a structure which makes it easy for other developers to add their own features.

And because the Java 2 platform is so flexible, and allows dynamic loading, you can do this very dynamically. You can load in a module that defines a new node type or a completely different layout manager developed by someone else, or from the Internet.

Q What drove your decision to choose Java 2 technology?

A I started this research at the University of New Mexico with my collaborators several years ago. At that time, I built the initial program in C++, tightly tied it to UNIX® and X, and hooked it up to the Tcl/Tk scripting language.

What we found was that it was very simple to use for mocking things up, or prototyping. But it was very difficult to extend it in any meaningful way. And it was tied to UNIX, which most of our target users didn't have.

When Java 2 technology with the Java 2D API came along, it gave us everything we needed: a clean language, good object oriented design, freedom from low-level details, such as memory management, and access to substantial utility libraries. It also had a high-quality 2D renderer that supported rendering anything in any scale. Finally, it ran under both Microsoft Windows and UNIX.

Q How big is your development team and how long did it take to develop Jazz?

A There are five of us: one staff programmer, three students, and me. We did the redesign and coding of Jazz from its original C++ version using Java 2 technology in just under a year. And that includes the development of MID and KidPad.

Q Is Jazz available to developers now?

A Yes. Jazz is entirely open source so it is freely available on the web site, and we encourage people to use it and tell us what you think, and make contributions to it.

We want people to download the platform, use it, and give us feedback, preferably with code additions! We will continue to offer new releases of Jazz on the web. While Jazz is open source, we defined it in such a way that people are able to develop proprietary applications based on it.

I'm strongly motivated by the Java technology community source model. I wouldn't be giving away open source code if it didn't work with the Java technology community. But it does clearly work. In fact, I'm able to do all kinds of things in building these applications that I couldn't do if I didn't have the Java 2 source code.

Q What is the Single Display Groupware application you are working on?

A At the University of Maryland, we have been working in collaboration with researchers in Sweden and England, funded by the European Union, on a project called "KidStory". Our goal is to support kids with new collaborative storytelling tools. By doing so, hopefully kids can learn communication, collaboration, and literacy skills. We work with children as our design partners which also leads to some fascinating challenges in our research process.

Through this, we've developed an application using Jazz called KidPad, which gives children as young as five years old the ability to have fairly advanced computer skills. They have the ability to do creative authoring, zooming, hyperlinks, scene manipulation, and animation. This is all done in the application in an entirely graphical way with direct manipulation without any pull-down menus or tool palettes.

But the most interesting part is that we do this with several children simultaneously working on one computer. Each computer has several mice plugged into a Universal Serial Bus port, and the software enables the mice to work independently.

When one child joins another, they collaborate, say, by drawing an object together or by drawing separate objects on the same screen simultaneously. When their drawing tools are in close proximity, special properties take over to enhance the collaboration, allowing them to, for example, fill in the space between their "crayon" tools as one fat crayon or squash and stretch objects by working together. Various tools have different properties when they are working with another user's tool.

We had to develop the Multiple Input Device (MID) software ourselves to do this, because most operating systems are based on the assumption that a computer has only one mouse and one keyboard.

MID is a Java technology-based toolkit that parallels the traditional Java technology-based event mechanism but provides access to independent sources. It gives you the option of creating different Java platform modules for different event sources. We have one so far. It supports multiple USB mice on Windows 98.

So if you run your application to use MID events, rather than regular events, every event has an extra method that returns the device that generated each event. Or, you can add a MID event listener that only fires when events come from a specified device. It's very clean. MID is designed to support many different types of event devices so other people can write their own event sources, such as gestures for pen input.

Here again, Java 2 technology was our choice. It has a very nice event listener architecture that we were able to extend to support other event types. And because Jazz is written for the Java 2 platform, we were able to write an application such as KidPad that uses both Jazz and MID.

It's all very flexible.

To contact Dr. Bederson, email him at:

See Also

The Jazz Home Page

The MID Home Page

Dr. Bederson's Home Page

About the Author

Jon Byous has been writing about technology -- from chips to mainframes -- for twenty years, working for Sun, Cisco, Oracle, Netscape, Quantum, and dozens of smaller tech firms. Jon has written over 25 articles about Java technology, a passion of his, for Sun's Java technology web site.