The Reality Files #04

A head-mounted three dimensional display (1968) by Ivan E. Sutherland

This post is part of ‘The Reality Files’, a publication and a series where I read research papers that focus on immersive and multisensory technologies, such as Augmented Reality, Virtual Reality, Extended Reality (‘XR’), and Haptics. I intend to produce a chronological reading of at least 25 papers, from 1960s to the 2020s.

I have tried to choose papers that have been influential or focus on topics that I have identified as important while working with these technologies. I am also aiming at a diversity of authors and topics. A disclaimer regarding this selection is that I do not have access to academic journals behind paywalls, and therefore I have not been able to include some papers I would have liked to. However, I welcome readers recommending and/or sending me papers — thank you!

A head-mounted three dimensional display

“The fundamental idea behind the three-dimensional display is to present the user with a perspective image which changes as he moves.“

This is how Ivan E. Sutherland starts this, maybe lesser known (link to PDF) than his original Ultimate Display paper, but more substantial one that documents the development of what became known as the ‘Sword of Damocles’. It was a seminal piece of hardware in the history of virtual reality and the display and tracking technologies that enable it, as the quote below testifies:

“Because it is very difficult to measure eye rotation, we are fortunate that the per- spective picture presented need not be changed as the user moves his eyes to concentrate on whatever part of the picture he chooses. The perspective picture presented need only be changed when he moves his head. In fact, we measure only the position and orientation of the optical system fastened to the user’s head.”

Sutherland had conducted some preliminary 3D display experiments during late 1966 and early 1967 at the MIT Lincoln Laboratory, and this paper builds on that initial work, albeit with a similar end goal: “Our objective in this project has been to surround the user with displayed three-dimensional information.”

The outcome was a setup that enabled six-degrees of freedom but with certain limitations down to the highly custom-built technology:

“The user is able to move his head three feet off axis in any direction to get a better view of nearby objects. He can turn completely around and can tilt his head up or down thirty or forty degrees. The objects displayed appear to hang in the space all around the user.”

It is difficult to imagine how such a solution, the ergonomics of the device itself notwithstanding, would compare with today’s VR. If that kind of a comparison would be possible, the main takeaway would probably be a realisation of how for granted we take certain aspects of such technologies today.

3D wireframe graphics

Despite the other limitations, Sutherland’s team had an ambition to address occlusion to create a believable experience:

“In order to make truly realistic pictures of solid three-dimensional objects, it is necessary to solve the “hidden line problem.” Although it is easy to compute the perspective positions of all parts of a complex object, it is difficult to compute which portions of one object are hidden by another object.”

The ‘Hidden line’ problem had to do with the wireframe graphics that the system was able to support, i.e. Sutherland wanted to hide the wireframe vertices that were farther in distance from the viewer and thus create a realistic perception of 3D objects and depth. They just did not have enough computing power to achieve what they aspired to. The shortcoming did become apparent in practice, when the team tested the application and reported ambiguous interpretations of the scene layout, especially regarding the depth of objects.

The Special-purpose components

Sutherland spends the first third of the paper describing how the setup works technically as a combination of (among other things) a matrix multiplier, clipping divider (that enables drawing only the wireframe graphics in the user’s field of view) and a line-generator:

“We built the special-purpose digital matrix multiplier and clipping divider to compute the appropriate perspective image dynamically because no available general-purpose computer is fast enough to provide a flicker-free dynamic picture.”

The matrix multiplier was used to convert information dynamically from the fixed room coordinate system to the moving eye coordinate system. “The job of the clipping divider is to accept three-dimensional information in the eye coordinate system and convert it to appropriate two-dimensional endpoints for display.” The below illustration from the paper shows how the various components work in tandem:

Having the custom-built components allowed to leave the computer in the setup to process the tracking information and render the graphics:

“The computer in this system is used only to process the head-position sensor information once per frame, and to contain and manipulate the three-dimensional drawing. No available general-purpose computer would be fasl enough to become intimately involved in the perspective computations required for dynamic perspective display.”

Sutherland describes the optical solution as follows – understandably the field of view was quite narrow:

“The optical system in this headset magnifies the pictures on each of two tiny cathode ray tubes to present a virtual image about eighteen inches in front of each of the user’s eyes. Each virtual image is roughly the size of a conventional CRT display. The user has a 40 degree field of view of the synthetic information displayed on the miniature cathode ray tubes.“

Tracking

While they had to contend with the head position sensor only, the aspiration was for what we today call room-scale VR. The head motion was possible within a volume of about six feet in diameter and three feet high. Sutherland was very aware of the ergonomic limitations of the setup – yet, at least anecdotally I have heard that still in the 1990s proprietary HMD solutions were not much more comfortable:

“The mechanical head position sensor is rather heavy and uncomfortable to use.“

The paper continues with lots of detail about how continuous ultrawave sound was used to measure head movements. An interesting detail is that in the earlier work, the head tracking was inaccurate to the point that after a few minutes of use the cumulative errors started to make the user experience insufferable. These mentions tell us how iterative this kind of ground-breaking R&D tends to be, with baby steps across the various components incrementally improving the overall UX.

Stereoscopic satisfaction

Then, Sutherland explains how the perspective transformation of the 3D objects was achieved. The system discussed was more sophisticated in its wireframe representation of the 3D space, achieving a believable sense of presence in a room with walls marked with N, S, E, W and the ceiling and the floor accordingly.

“The biggest surprise we have had to date is the favorable response of users to good stereo. The two-tube opti- cal system presents independent images to each eye. […] Observers capable of stereo vision uniformly remark on the realism of the resulting images.”

A belated impact

In Sutherland’s acknowledgements it becomes clear how the success of the project was dependent on many specific technologies coming together: the clipping divider for occlusion, and the matrix multiplier for translating the two coordinate systems into a dynamic whole, but also programming solutions to display curved surfaces in stereo. Kent Bye’s interview of Sutherland’s colleague Henry Fuchs is enlightening in this respect:

There is no doubt about how influential The Sword of Damocles was as a research effort. The proprietary enterprise VR systems in the 1970s and 1990s owe a lot to Sutherland’s experiments.

However, Sutherland himself did not focus on VR after the milestone project — perhaps its impact came with a considerable delay. Instead, he continued to work on ground-breaking research on computer graphics and human-computer interaction, which includes the highly influential Sketchpad, ‘A man-machine graphical communication system’.

I welcome readers to recommend and/or send me papers – feel free to comment or send me a private note. Thank you!