VR/AR Introduction and Overview

From EuroVR Knowledge Base

Virtual Reality (VR) and Virtual Environments (VEs)

At present there are no single concise or generally accepted definitions of Virtual Reality (VR) and Virtual Environments (VEs). This is partly due to the continual state of evolution of the many technologies involved and also due to people using the terms to mean a variety of things. However most people would agree that VR refers to:

The combination of systems that are used to create and maintain virtual environments.

In the context of VR, the attributes of virtual environments (VEs) are listed as follows:

computer-generated representations of real or imaginary environments;
experienced as three dimensional via a number of senses - visual, aural and/or tactile;
objects within these environments are independent of the user and can display real world behaviour;
the user or users have autonomous control - the freedom to navigate and interact with objects, using a number of different viewpoints;
interaction occurs in real-time; and
the users experience feelings of presence and/or immersion.

VR is a new level of human computer interaction where, in principle, people are able to perform the activities which they are best suited e.g. logical reasoning, problem-solving, interpreting information in different ways to form different conclusions and so on. Computers are able to perform the activities that they are suited to - storing a huge amount of information, performing calculations at great speed and accuracy and displaying the information in different ways.

The user is said to experience feelings of immersion and/or ‘presence’ in the VE. Slater and Wilbur (1995) distinguish between immersion and presence by defining ‘immersion’ as “the extent to which the computer displays are capable of delivering an inclusive, extensive, surrounding and vivid illusion of reality to the senses of a human participant,” (p. 13) and presence as “a state of consciousness, the (psychological) sense of being in the VE” (p. 14). These two attributes have been the focus of much research (Heeter, 1992; Held and Durlach 1992; Loomis, 1992; Sheridan, 1992; Zeltzer, 1992; Barfield and Weghorst, 1993; Slater et al, 1994; Witmer and Singer, 1994 and Barfield et al, 1995) as it is felt that, along with ‘interactivity’ they distinguish VEs over other similar technologies.

Virtual Reality (VR) research and development is a rapidly evolving field. New display technologies and interaction devices are continually being produced and modified, having an impact on the way in which participants view, navigate and interact within Virtual Environments (VEs). In addition, as VR technology is being used to an increasing extent within industry, it is now possible to assess the impact of introducing such technology into the workplace at an individual and organisational level.

VR offers a number of potential benefits:

VR may be used where manipulation of real-life variables would risk damage to people, equipment or the environment (eg. in hazardous environments)
VR can provide an alternative where manipulation of real-life variables has an unacceptably high associated resource cost (eg. logistics, finance, personnel or national security)
VR allows rapid prototyping and configuration of an environment where perceptual input might need to be changed frequently (eg. reconfiguring interface details or instrument layouts)
VR may be used to enhance or degrade or otherwise alter some aspect of reality. As such VR might be used to impose visual restrictions on the user (eg. smoke effects in a burning compartment or reduced visibility due to a visual defect), or VR might be used to highlight components in the real world which could otherwise be missed (eg. fire extinguishers or escape routes through buildings) VR allows users to experience views of micro or macroscopic entities in a variety of dimensions (eg. using VR to explore the atomic surface of materials or human tissue at a nanoscopic level (Stone, 2001b), in a way that would not be physically or ethically possible in the real world)
VR can be used where real locations are impossible or difficult for users to physically occupy (eg. hazardous environments, simulating outer space, under the sea, etc)

Other definitions

The difficulty in defining VR and VEs has occurred because of a number of problems. Firstly, many different disciplines from art, psychology, design, computer science, engineering, manufacturing, medicine etc., have become very interested in applications using VEs. However these diverse disciplines can cause confusion with terminology as in some cases the same words may have different definitions making it difficult for these disciplines to communicate effectively. The National Research Council (NRC, 1995) provided a good example of this, “computer scientists naturally use the terms input and output in reference to the computer, psychologists use these terms in reference to the human user. Thus, in a virtual environment system, what is output to the psychologist is input to the computer scientist.” (P. 14). However because VR is a computer technology which focuses on allowing a persons ability to interact with information presented by the computer, the future development of VEs relies on collaborative work between the many disciplines.

There is also the problem of misleading reports about the technology. In the attempt to ‘jump on the bandwagon’ a large number of conferences, magazines and publications began in the early 1990’s; companies were advertising their ‘active use’ of VR systems and even a few films were made - “Lawnmower Man”, “Lawnmower Man 2” and “Disclosure”. However, this resulted in false expectations of the technology which caused much disappointment to the point where researchers working with VR systems began using alternative terms like ‘VE systems’ and ‘interactive environments’, in order to avoid the confusion.

Also there is the problem in distinguishing them from other similar technologies. VEs have grown out of many technologies and is part of what people have termed ‘graphic simulation systems’ (Zeltzer, 1992), ‘technologically mediated experiences or synthetic experiences’ (Robinett, 1992), or ‘synthetic environment systems’ (National Research Council, 1995).

Zeltzer (1992) was one of the first authors to attempt a classification of VEs. His definition assumed that any VE has three components. Firstly, a set of models/objects or processes; secondly, a means of modifying the states of these models; and finally, a range of sensory modalities to allow the user to experience the VE. Zeltzer represents these components on a cube with scales relating to ‘autonomy’, ‘interaction’ and ‘presence’,

Zeltzer defines ‘autonomy’ as “a qualitative measure of the ability of a computational model to act and react to simulated events and stimuli, ranging from ‘0’ for the passive geometric model to ‘1’ for the most sophisticated, physically based virtual agent” (p. 127). ‘Interaction’ is, “the degree of access to model parameters at runtime (i.e., the ability to define and modify states of a model with immediate response). The range is from 0 for ‘batch’ processing in which no interaction at runtime is possible, to 1 for comprehensive, real-time access to all model parameters” (p. 127). And ‘presence’ is “our sense of being in and of the world ... engendered by our ability to affect the world through touch, gesture, voice, etc.” Therefore the presence axis provides a crude “measure of the number and fidelity of available sensory input and output channels” (p. 128) which implies that the more the user’s senses are engaged by the environment, the more ‘presence’ they will feel. However the degree of sensory input and output is very much dependent on the application and the design of the VE. This design is guided by ‘selective fidelity’. (Johnston 1987, cited by Zeltzer 1992 and Robinett 1992). Generally it is impossible to reproduce the real world in huge detail and complexity, therefore depending on the application, the ‘sensory’ cues which are necessary for the user to fulfil the application must be carefully identified. Then the design of the environment must match as closely as possible the human perceptual and motor performance required for successfully completing the application. Given the limitations on current technology many trade-offs have to be made and how these effect the degree of presence has been the focus of much research work over the years.

Zeltzer’s cube represents at the point (0,0,0) the early graphic systems e.g. graph plotters and charts, which have no autonomy, interactivity or presence, to the ideal VR system (1,1,1) with full autonomy, interaction and presence. However, Zeltzer suggests that current VR systems are at the (0,1,1) point, that is, a high degree of interactivity and possibly, presence but very limited degree of automation of objects.

Robinett (1992) proposed a taxonomy on ‘synthetic experience’ which he defined as “perceiving a representation or simulacrum of something physically real rather than the thing itself.” (p.230). It considers experiences which rely on some form of technology e.g. computers and digital electronics, to interpret the user’s actions and provide a response. Therefore it includes technologies like the telescope, microscope, television and telephone, as well as, teleoperation, VR and flight simulation. The systems are classified into nine categories each with further sub-divisions. The aim is to provide some clarity on the similarities and differences of the various ‘technologically mediated experiences’. The nine dimensions and their possibilities are shown in Table 2.1 (overleaf). In Robinett’s discussion of this taxonomy, he compares it with Zeltzer’s model and Naimark’s taxonomy on methods for recording and reproducing experience (Naimark, 1991). In general he states that his taxonomy attempts to cover the overall domains of both models for recorded, simulated and transmitted experience. He suggests that it offers a ‘starting-point’ for discussion, and, in particular, explores the potential possibilities for VR systems and their relationship to other systems.

More recently, the National Research Council (U.S. committee consisting of the National Academy of Sciences, the National Academy of Engineering and the Institute of Medicine) produced a report on ‘Virtual Reality - Research and Developments’ (NRC, 1995). They have suggested that VR systems are part of ‘synthetic environment (SE) systems’. That is, a system where “the human operator is transported into a new interactive environment by means of devices that display signals to the operator’s sense organs and devices that sense various actions of the operator” (NRC, 1995; p. 13). Other systems which belong to this group include ‘teleoperation’ and ‘simulator systems’.

The figure shows that in a normal situation the ‘human operator’ and the environment, directly interact with each other. In a teleoperator system, the human operator interacts with the environment via some form of ‘human-machine interface’ and a ‘telerobot’. Sheridan (1992) defines a teleoperator as “a machine that extends a person’s sensing and/or manipulating capability to a location remote from that person” (p.4). He defines a telerobot as “an advanced form of teleoperator the behaviour of which a human operator supervises through a computer intermediary. That is, the operator intermittently communicates to a computer information about goals, constraints, plans, contingencies, assumptions, suggestions and orders relative to a remote task, getting back integrated information about accomplishments, difficulties and concerns and (as requested) raw sensory data. The subordinate telerobot executes the task on the basis of information received from the human operator plus its own artificial sensing and intelligence” (p.4). Therefore these systems are used in areas that may be too remote or dangerous for a human operator, in order to change the state of the environment. These systems have already been used in various applications and being developed for others including space, undersea oil and science, nuclear power plants, toxic waste clean up, construction, agriculture, mining, warehousing and mail delivery, firefighting and lifesaving, policing, military operations, assisted devices for the disabled, telediagnosis, telesurgery and entertainment (Sheridan, 1992; Chapter 2). However, in a VE system, the human operator interacts with the computer environment, so it could be said that the main purpose is to change the state of the human operator or the information stored in the computer. Therefore, the main differences between teleoperators and VEs are the purpose of the systems and the equipment used.

A more difficult distinction to be made is that of VEs and simulation systems. However the National Research Council (NRC, 1995) provide a clear list of where the term VE should be used rather than simulator, as follows: • the system is easily re-configurable by changes in the software • the system can be used to create highly unnatural environments as well as a wide variety of natural ones • the system is highly interactive and adaptive • the system makes use of a wide variety of human sensing modalities and human sensorimotor systems and • the user become highly immersed in the computer-synthesized environment and experiences a strong sense of presence in the artificial environment (p.22).

Furthermore, they suggest that the focus of simulators and VEs are very different. In a simulator, it is usually the equipment e.g. aeroplane, vehicle, etc., which is the focus of the application and so physical mock-ups tend to be used and the ‘simulated environment’ represents distant locations. Currently, the focus of VEs is on the user and so the environment tends to represent the whole situation - near and far objects. Finally, the difference between other computer-based systems is considered mainly on the extent to which the system is 3D, interactive, multi-modal and immersive. These are recognised as the distinguishing features of VR systems.

Additional definitions of VR

A brief history of Virtual Reality (VR)

There are numerous authors which have provided a history of VR (Fisher, 1990; Rheingold, 1991; Ellis, 1991a; Ellis, 1991b; Biocca, F. 1992; Gigante, 1993; Pimental and Teixeira, 1993; Burdea and Coiffet, 1994). A brief summary based on these authors is as follows: VR has developed through a combination of ‘visions’ and ‘enabling’ technologies. In the 1950s and 60s, people were developing the concept of a ‘virtual reality’. The most famous of these ‘visionaries’ was Morton Heilig, a cinematographer in California who is recognised as the person who inspired much of the early industry with his ‘Sensorama’ machine (Heilig, 1992). This consisted of a motorcycle that a person sat on and watched video scenes of Brooklyn in the 1950s. It included sights, sounds, vibration of the cycle for feel, and even smells. However, it was not computer-based and not interactive.

It is generally accepted that VR, as recognised today, was begun by Ivan Sutherland, a researcher at Harvard University, with his paper “The Ultimate Display” (Sutherland, 1965). This reported the first ever head- mounted-device (HMD) or headset which could show computer-generated three dimensional images of cubes. At the same time, another researcher at the University of Wisconsin, Myron Krueger, was experimenting with what is now called, ‘artificial reality,’ (see section 2.4). His GLOWFLOW project involved a computer-controlled light and sound environment which responded to the behaviour of the users (Krueger, 1991). However their systems could not be made commercially with the technology which was then available. Also, VR was not considered a ‘scientific-enough’ subject, according to the American academic community and so information was difficult to disseminate.

During this time, there was a rapid development of ‘enabling’ technologies driven by various sources. These are technologies which ‘enable’ other technologies to exist. Small electromagnetic cathode ray tubes (CRTs) were produced by companies such as Thorn, Thomas and Hughes with military funding. A strong market for consumer electronics resulted in the production of small flat-panel displays and liquid crystal displays (LCD). These innovations allowed for smaller, lighter and cheaper display devices at a higher quality. Also, the personal computing revolution in the ‘70s and ‘80s made available fast, cheap, digital image generation, as well as high speed graphics workstations which could produce images faster and with resolutions of a higher degree. Tracking systems, which could translate the movement and orientation of a person in the real world into a computer environment were being developed (primarily by Honeywell and Polhemus again for military use). It was the convergence of all of these technologies which allowed affordable systems for applications.

In 1985, Michael McGreevy from NASA/Ames Aerospace, Human Factors Research Division, held an event to show his new ‘affordable’ head-mounted-device (HMD), made from currently available technologies. It used a motorcycle helmet with two LCD screens from a couple of mini televisions and a magnetic tracking device which connected the user to the computers which generated the images. It was the first suggestion that the technology was now affordable.

In the same year, Myron Krueger opened his ‘VIDEOPLACE’ in Connecticut Museum of Natural History. This used a combination of video camera, computer graphics and gesture/position-sensing technologies. The system allowed multiple users to interact with each other even though they were in different rooms.

In other places in the U.S., researchers were exploring glove technologies. Thomas Zimmerman is recognised as the inventor of the VR glove, (although in 1981, a researcher called Gary Grimes, who was working for Bell Laboratories, had patented a glove-based computer interface device. This glove used small switches at each finger joint to allow the user to interact with computer images. However Bell did not pursue the work (Rheingold, 1991)). Zimmerman’s glove was light-weight and used thin, pliable, hollow plastic tubes which conducted light. He developed it so that he could play true ‘air guitar’. When Zimmerman met Jaron Lanier (who was then famous for programming a video game called “Moondust” for Atari), they founded VPL Research Inc.

The real breakthrough for VR came when Scott Fisher of NASA/Ames commissioned VPL to develop a glove for their VR system. This reached the attention of the media with the result of a lot of ‘hype’ which raised the awareness of VR in the public domain. However, even though it is generally agreed that VR largely began in the U.S., the world’s first commercial VR system was launched in 1991 in the U.K. at Wembley, London, by Jonathan Waldern, a researcher at the former Leicester Polytechnic. He first designed the system in 1984 after reading a paper by Jim Clark, a student of Sutherland’s in Utah. By 1988, he and some friends had built the first VR arcade system in his garage and after the launch it was placed in London’s Covent Garden‘s Rock Garden Club and ‘W’ industries had begun the VR ‘revolution’.

From 1991 until now - 1998, VR systems and VEs have been progressing their way along the learning and development curve. The technology is still placing limitations on what has been envisaged for VEs. However there is also still a need to understand how VEs can be used effectively. Research work in the area is now prolific around the world with Japan now becoming one of the major players along with the United States, the U.K. and Germany. A list of the research work being carried out around the world can be found in Appendix I. There are 20 countries actively exploring VEs, with numerous research sites which can be accessed via the internet. Information is now more readily available, companies are releasing their internal studies and researchers and industrialists are beginning to form collaborations over many different projects.

Types of VR systems

Desktop VR

Desktop VR tends to be the cheapest form of VR system (see Figure 2.6. overleaf) as it consists mainly of a standard affordable computer which most people can now buy for home use. For this reason it is referred to as a ‘low-end’ system. The minimum specification of the computer in order to run the specialised VR software available (see section 2.4.2.2) is constantly changing. Three years ago it was possible to run a VE on a 33 MHz 386 PC with 8 MB RAM with a special graphics accelerator card and sound card. However, today, it is not worth purchasing a 386 PC even for the home market, as it would be difficult to maintain the parts, as many are no longer available and current software, not only VR software, would not be able to run effectively on it. Most computer vendors are now offering the home market, as standard, a Pentium 233 PC with 32 MB RAM and full multi-media capability, at an ever-decreasing price, making desktop VR more accessible to the home market, as well as industry. The usual visual display used by desktop VR is the computer monitor, generally SVGA (Super Video Graphics Adaptor) with anything up to 16 million colours and either 14”, 15”, 17”, 19” or 21 inches in size. However, desktop VR can support other visual devices like ‘shutter glasses’ and ‘stereo screens’ (see section 2.4.2.5). The sound cards in the computer are usually adequate for providing sound in the VE and are delivered through speakers. The input devices used to interact with the VE and often associated with this system are the keyboard, mouse, joystick or spacemouse (a six-degrees-of-freedom device - see section 2.4.2.7) although in some systems touch-screens are also used.

The main disadvantage of desktop VR is that the relatively small amount of processing power (compared with the more powerful computers used in other VR systems) places limits on the capabilities of the VE (this is discussed in section 2.4.2.1). Also, as the visual display is often just the computer monitor, these systems have been criticised for not utilising the full potential of the three-dimensional and ‘presence’ qualities of VEs. This is because the images are still essentially two-dimensional (unless the user has shutter glasses or stereo screens which can assist them to have stereo vision, discussed in section 2.4.2.5). Also, the VE does not fill the user’s complete field-of-view and, therefore, it is still possible to get distracted by objects in the peripheral view which can diminish feelings of presence. However, this is still the most popular choice of system because the initial investment cost is minimal. Furthermore there are applications being developed using desktop VR to show that with careful consideration of the capabilities and limitations, effective VEs are possible. This is also a popular system because it has less of the possible side-effects which are associated with some of the other VR systems. Applications being explored are wide and varied, therefore in the short-term at least, desktop VR appears to be the way forward for many VE applications.

Projected VR

Projected VR systems use large projection displays, either on one surface (which can have a viewing angle of up to 120º across), or multiple projection displays to create a room or 'CAVE’ (Cave Automatic Virtual Environment, see Figure 2.7). They run on high-end graphics workstations such as the Silicon Graphics range. The user can have shutter glasses in order to see the projections in three-dimensions and either some form of handheld device or a chair with hand devices attached are used to navigate around and interact with the VE. The advantage of these systems is that they increase the quality of immersion and presence, as compared with just a computer screen because the display fills the user’s entire field-of-view, providing the illusion that they are ‘in’ the environment. However even though many people can see the VE, still only one user can control the navigation. Also they can be very costly in terms of equipment and development time and they require a lot of space. The types of applications currently being explored using this type of system have mostly been impressive walkthroughs of buildings, art galleries and museums (e.g. Shaw, 1994).

Artificial Reality

An artificial reality system has been defined by Myron Krueger, as a system which “perceives a participant’s action in terms of the body’s relationship to a graphic world and generates responses that maintain the illusion that his actions are taking place within that world.” (Krueger, 1991; p. 268). Krueger is recognised as the pioneer of this particular technology and there is no other researcher who has carried out more work into its applications. Comprehensive information about this system and its applications can be found in ‘Artificial Reality’ (Krueger, 1983) and ‘Artificial Reality II’ (Krueger, 1991). Generally, the system consists of videos and computers which merge a video image of the user with computer graphics. The user is able to watch a monitor which shows their body interacting with the objects (or people) in the VE (see Figure 2.8). This technology is already frequently used by the television media to provide interesting backdrops for the news, weather and other shows. It is also used by the film industry to merge computer-animated characters with real characters.

Mixed Reality

Augmented Reality

Augmented reality systems (see Figure 2.9) have been described, by Pimental and Teixeira (1992) as the “use of transparent glasses onto which data, diagrams, animation or video can be projected to aid people who need to be simultaneously in the real world and also be able to access additional data to do their jobs” (p. 11). The example application given by Pimental and Teixeira (1992) is of Boeing’s exploration into the technology for aircraft engine mechanics. They are developing a system which allows the mechanics to access diagrams, parts lists and text while they work on a real engine, with the aim to be able to eventually overlay an entire structural diagram onto the engine to give the mechanic a type of ‘x-ray’ vision. Other application areas actively being explored are in the medical field by surgeons, using information from CAT (Computer-aided tomography) scans and x-rays to overlay onto real patients in order to examine the most appropriate route for surgery (Truppe et al, 1996).

Headset systems

Finally, headset systems are probably those most associated with VR technology (see Figure 2.10, overleaf). They consist mainly of a head-mounted display/device (HMD) which the user wears to receive visual and auditory information from the VE. The user interacts with the VE using some form of hand-held controller, like a wand, joystick or dataglove, and trackers on the HMD and input devices allow the computer to constantly update the position of the user. The advantage of this type of system is that it completely blocks any external influences from the real world by enclosing the user’s visual and auditory senses with a HMD. This creates a strong sense of immersion and presence, as the user is unaware of any other environment except the VE. However, for this very reason, the system also has its disadvantages. Researchers in the area believe that the closer that the VE gets to resembling reality the more cases of ‘simulator-type sickness’ may occur (Kennedy, 1991; Levison and Pew, 1993). Simulator sickness has been defined as ”a feeling of discomfort that arises from performing tasks in the simulator, where such discomfort is not elicited when the same tasks are performed operationally. This discomfort may include nausea and disorientation that occur while the simulated tasks are being performed, plus adverse symptoms that persist (or become initially apparent) after the person has left the simulator.” (Levison and Pew, 1993; p. 70). It is also sometimes referred to as ‘cybersickness’ and is believed to be partly caused by ‘sensory cue conflict’, that is, when a person’s senses are receiving conflicting information. In a VE, the user may be ‘moving’ and receiving all the usual information via their visual and auditory senses to confirm this. However, in the real environment they are stationary and their body systems are sending contradictory information. It is felt that this conflict is enough to induce ‘sickness’.

However, much research is on-going in this area in order to identify the factors which may cause this effect and ways of predicting an individuals’ susceptibility to simulator sickness (Hettinger et al, 1990; Kennedy et al, 1992; McCauley and Sharkey, 1992; Kennedy et al, 1993; Oman, 1993; Regan and Ramsey, 1994a; Regan and Ramsey, 1994b; Cobb et al, 1995). Therefore, this type of system has a number of problems which need to be addressed before it should be widely implemented.