Haptic Interaction

From EuroVR Knowledge Base

Research Area

The term "haptics" derives from the Greek word "αφή" (noun, pronounced: hapthei) or "άπτεσθαι"(verb, pronounced: haptesthai), which generally refer to the sense of touch. The haptic sense is usually divided into two main distinct sensory modalities (although such a clear separation cannot be absolutely established on a physiological basis, due to the variety and complexity of sensory interactions):

1. the kinaesthetic sense (motion and force sensing), which includes perception of muscular effort as well as proprioception, and
2. the tactile sense, which provides cutaneous information, related to contact between the skin of the human body and the external environment (pressure, vibration, temperature etc.), thus enabling the perception of physical properties such as the surface characteristics of touched objects (texture etc.).

Undoubtedly, the most prominent characteristic of Virtual Reality systems concerns multimodal (real-time) sensory-motor interaction between the human operator (user) and the virtual (computer simulated and animated) environment. Such a natural and intuitive human/computer interaction should involve all the sensory channels of the human being, that is, not only vision but also all other sensory-motor modalities. The haptic sense is probably the most "powerful" of senses used practically in all everyday human activities to interact with the physical environment surrounding us and to perform daily tasks. Particularly, the human hand, with its exceptional dexterity and sensory-motor capacities, constitutes the most versatile "tool employed by the human being to explore the physical world and interact with it, in order to acquire useful multidimensional sensory information. Moreover, the close interconnection between haptic perception and physical action classifies the haptic sense as the most important "active sense, as it relates particularly to complex manipulative and exploratory procedures. The goal of efficiently integrating such sensory-motor functionalities and skills within a VR system poses extraordinary challenges for researchers and engineers in the field. Providing the user of a virtual environment with "haptic feedback" requires physical interaction and coupling with specially designed devices that can artificially create a variety of touch-related sensations, usually characterised as: (a) force/kinaesthetic display (corresponding to the application of forces and motion constraints on the human body) and (b) tactile display (stimulating the surface of the skin and enclosing various cutaneous sensations, particularly texture, temperature or contact). The study and integration of haptic interaction modalities in Virtual Reality systems consists in providing the user of a virtual environment with all these sensations that are involved when exploring and manipulating virtual objects. Studying all these issues forms the basic focus of the research field described in general under the term "haptic interaction". The development of haptic interfaces originates from teleoperation. Indeed, the first force-feedback arms were developed for teleoperation tasks within hazardous environments. However, the field of haptics is now rapidly evolving, offering tremendous new potential in terms of human-machine interaction paradigms, and promising to open vast new directions of novel application domains. These applications can be classified in the following broad areas (or application sectors):

• Education and Training Applications (e.g. interactive haptic simulation of physical phenomena for educational purposes, novel computer-assisted teaching and learning scenarios, etc.)
• Industry: New Interactive Design Tools and Virtual Prototyping Applications, Maintenance (Simulation and Training) Applications, Teleoperation / Telerobotics and Nano / Micro and Tele-Manipulation Applications
• Medical Applications:
• Interactive Computer-Assisted Medical Education, Clinical/Surgical Training and Skill Assessment, Assistance Tools for Planning of Medical Procedure, Telesurgery / Telediagnosis and Rehabilitation
• Scientific Visualization – Haptic Exploration of Complex Scientific Data
• Gaming – Entertainment Industry (interactive video games, theme parks, etc.)
• Arts and Creation (virtual sculpture, virtual musical instruments, etc.)

Thus, the field of haptic interaction has become one of the most active of the moment in Virtual Reality, and it concerns an increasing number of researchers and companies worldwide. The Haptic Interaction WG has mainly identified five major Fields of Research (RF), which will lead the activity of the working group through all its existence. These five fields cover the wide range of scientific problems, which occur when integrating haptic feedback within a virtual environment (see Figure 1). These five research fields are:

• RF1 - Haptic Hardware. This field concerns the building of haptic devices. It involves mechanics, electronics, automation and control, the integration smart materials and technology, etc.
• RF2 - Haptic Software. This field is also known as “Computer Haptics” (the “haptic rendering software”, or the computer-based contribution in the generation of a haptic feedback to the user). It involves algorithmic components to design the haptic “core” library, including collision detection, texture mapping, etc.
• RF3 - Haptic Interaction Techniques. This field deals with Computer-Human Interaction (CHI) based on haptics. It includes Ergonomics and evaluation methods. It aims at providing guidelines for the design of haptic interfaces.
• RF4 - Haptic Perception. This field deals with the problem of haptic perception as seen from the side of the user, basically the psychological (psychophysical) and physiological issues. This field provides important human-related haptic information that are helpful to well “size” and design haptic interfacing technology.
• RF5 - Haptic Applications. This field deals with the use and applications of haptics. It is mainly concerned with the transfer of research results to actual industry and society applications. It is concerned with the risks and safety of the haptic interaction, the industrial potential and the market analysis, the standards definition, etc.
  
  

  

  Fig. 1 - Research fields decomposition
  
  

The aim of the Haptic Interaction WG and of its joint program of work is to promote and develop research and technology in each one of these five broad areas.

Research Needs

Nowadays, the use of haptic devices still remains restricted to research laboratories. It is obvious that many problems related to the use of haptic devices derive from this minority character. Finding a killer application that spreads out the use of haptic devices as consumer goods is an important milestone in the development of this technology.

As a consequence, the development of haptic applications is scarcely motivated by a real need, but it is more often driven by the availability of the technology. A debate is needed about this issue, because this problem is a symptom of a potential weakness. It is time to search how to use haptics to solve real problems, and stop searching problems that match haptics-based solutions.

Thus, this section is devoted to specific topics necessary to overcome bottlenecks and allow the development of haptic technologies and applications.

Technological Needs

Haptic rendering has computational requirements much harder than visual and auditory display. A stable haptic system displaying smooth and realistic forces and torques requires haptic update rates of 1 kHz or more, and current haptic virtual objects models are quite simplistic when they are compared to the static and dynamic behaviour of objects in the real world.

More than ever, the central problem of the use of haptics in large-scale applications is the level of complexity, which can be addressed with state-of-the-art solutions today. It is commonly accepted that haptic interaction calls for very fast response time in collision detection and contact simulation, which are very computation-intensive processes. However, the size of processed data has to be increased by a factor of 10 or more probably, in order to achieved a good usability in daily work.

There is, therefore, a clear need to elicit computationally efficient models and interaction techniques that result in real-time haptic displays that match the human perceptual capabilities in accuracy and resolution. Another problem is the synchronization of the visual, auditory and haptic displays. This can be problematic, because each modality requires different types of approximations to simulate the same physical phenomenon.

It seems completely necessary to use multiple processors with shared memory and/or multi-threading and haptic rendering hardware implementation for acceleration in a way similar to OpenGL. A promising field is haptics over a network (like internet) which is already being researched and will allow some collaborative applications. Techniques for reducing the transmission delays and haptics facilities in the standardized protocols for distributed VEs are open issues.

Regarding machine haptics the stronger need probably concerns the finding of actuators specifically designed for haptic purposes. An important point is the reduction of size of the actuators which remains still one of the major problems that limit the portability of haptic devices. In this sense, there is a great expectation in the development of new tactile devices based on the recent technology of microactuators.

We see haptic interaction as a very promising field of study concerning interaction with numerical (CAD) models. It is important to mention that this kind of interaction is not to be limited to VR environments (Virtual Reality Centers, etc.), but it can also be considered on "standard" workstations (PC, etc.), with adequate interfaces. The number of applications seems to be very high when considering the domain of technical tasks (assembly/disassembly, etc.) in design and training. This is particularly obvious in aeronautics industrywhere such tasks (maintenance, manufacturing…) are met very often. A survey performed in DASSAULT AVIATION facilities in 2003, showed that about two thirds of the people questioned about the information they would like to have in the domain of touch, mentioned first (in priority) a feedback information related to the force induced by the manipulation of objects. The French PERF-RV platform has led to significant results and advances in this field of study with for example the prototype coupling the VIRTUOSE 6D (Haption) interface with CATIA V5. The prototype demonstrated that a seamless integration of haptics with the digital mock-up was possible, using existing state-of-the-art technologies. This kind of interface seemed very helpful when moving objects and facing (or trying to avoid ) collisions. The user could clearly and rapidly investigate solutions for performing given tasks (for example, moving an object to extract it from given complex volume). As an example, current expectations of aeronautics in this domain are on the one hand to be able to perform simulations with larger and larger models (reducing computation times for haptic interaction) and on the other hand to expand the functionality of the coupling (we can mention for example the possibility to handle different objects in sequence…). It seems to that progresses performed in these two directions will allow a very significant increase in the number of scenarios that can be studied.

Standardization

A key issue concerns the lack of standard interfaces between the several software components needed for haptic interaction: collision detection, contact simulation, data management, force control, etc. As a consequence, all attempts to assemble the best off-the-shelf components together run into hard integration problems. A large scale endeavour of a critical mass of groups towards setting a software standard for haptics, like OpenGL for graphics, has not been tried yet. At the current stage, it seems to be easier for many groups to develop a proprietary library instead of adopting an existing framework. Many of the proposed frameworks are platform- and/or hardware-dependent.

There are currently many different haptic Software Development Kits (SDK) or Application Program Interfaces (API) developed by commercial companies. The best known is probably the GHOST SDK created by SensAble technologies for their Phantom device. GHOST was the first of such libraries to be largely distributed and with a certain commercial success.

From a standardization perspective, these proprietary APIs are some problems: they generally support only one device type, they can be very expensive and they are not open source and not expandable. If one wants to design and test new devices and algorithms the use of such APIs is thus very limited.

There are some exceptions like the Reachin API from Reachin Technologies, one of the first haptic/graphic API that was independent from a specific haptic device or the e-Touch from Novint technologies, the first to launch a set of open-module haptic/graphic libraries. However, there is still a real need for haptic libraries with open source and hardware-independent code allowing anyone who works with haptic devices to re-use a basic core of well-known algorithms. It would be interesting to define a high level haptic/graphics language with functions for collision algorithms, simulation, deformable objects manipulation, input/output access, communications, etc.

Research on haptic perception

Although haptic interaction with objects in our environment and with our environment plays a fundamental role for human perception, the sense of touch was by far less intensively inves¬tigated than for example vision. This might be due to the diversity of physical properties that are involved in providing haptic information. Thus, there is a need today for more scientific research on haptic perception.

The haptic community is still concerned with basic topics, such as under¬standing the primitives of haptic and tactile perception such as temperature, force, frequency, etc. The knowledge of these principles in haptic perception could then feed back in the devel¬opment of new hardware. In return, this new hardware could be restricted to stimulate the part of the haptic sense, which actually provides the best information for the final percept. The idea behind this is that we are able to exploit the limitations of the human haptic system, as we already do for the visual modality. In addition, having a deeper understanding of the human haptic system characteristics and limitations, as well as of the human perceptual processes, will make us able to develop more effective Guidelines for Evaluation and Testing of haptic devices and applications.

The state¬-of-¬the-¬art to stimulate haptic perception is mostly based on electromagnetic actuators. Commercially available devices are too often restricted to highly specific applications such as vibration feedback in game consoles and force feedback in virtual training scenarios for surgeons. Having the appropriate display technology available is the first key to any basic scientific research on perception. For example in vision, improved monitor and display technologies provided the basis for a successful research on the visual system.

New Concepts

Although there are some well established haptic device concepts, much more effort should be done at this level. More imaginative solutions should be found. For instance, haptic perception can be achieved or complemented using other sensory stimuli (concept of “pseudo-haptic” feedback, use of haptic illusions). Moreover, tactile (exteroceptive) receptors are not properly exploited. A lot of research and technical advances are needed in this field.

From our point of view, the best way to achieve this goal and find new interaction concepts is to gather researchers and developers coming from different fields, including engineering, psychology, ergonomics, physiology, and even art. Teamwork seems essential to elicit significant advances in haptics.

Unfortunately, it seems that the universal device does not exist yet, as it is the case for other sensory modalities. Haptic interfaces can rarely be used in an application different from the one they were designed for. It is probably impossible to find such a universal device but regrouping similar tasks could lead to a set of versatile haptic displays. To do so, a categorization of haptic interaction paradigms should be carried out more systematically.

Current Situation

Haptic devices entered the commercial market in 1993, with the founding of the companies Sensable and Immersion in USA. In more than ten years, very few applications have been a real success, with volumes exceeding a mere hundred. Indeed, apart from force-feedback joysticks and pads used in computer games, only Sensable achieves some kind of success, with their software suite for intuitive design FreeForm™. Some SMEs, dedicated to the development of haptic devices and applications, have recently appeared on the VR market in Europe, such as Haption in France or Reachin in Sweden. However, these companies remain isolated.
Nevertheless, all experts agree that haptics, and especially force feedback, will play an important role in the man-machine interfaces of the future. But the time needed for reaching such a goal remains unclear. Several factors can be put forward for explaining the lack of large-scale applications today:

1. Despite years of dedicated research, haptic technology is still in the cradle. Plenty of reliable commercial devices are available, but prices are high and there is a lack of solid benchmarking. Moreover, most available devices are not well adapted to large-scale applications.
   
2. Lots of research and development work is done all around the world, but large firms are very little involved, apart from the game industry. As a consequence, development projects are small and isolated, and do not reach the critical mass.
   

Today, only eight companies propose haptic devices on the market, and six of them are still in the “start-up” stage:

• MPB Technologies (http://www.mpb-technologies.ca/), a Canadian company mostly active in the research and telecommunications domain, manufactures a product developed by McGill University.
  
• CyVerse (http://www.cyverse.co.jp/), a Japanese company founded in 1995, manufactures cable-driven haptic devices after a design from the Sato Laboratory.
  
• FCS Robotics (http://www.fcs-robotics.com/), a Dutch company founded in 1995 as a spin-off of the Fokker group, produces devices for medical and rehabilitation applications.
  
• Force Dimension (http://www.forcedimension.com), a Swiss company founded in 2001 as a spin-off of EPFL, is mostly active in the research market.
  
• Haption (http://www.haption.com), a French company founded in 2001 as a spin-off of CEA (French agency for nuclear energy), focuses on research and industrial applications.
  
• Quanser (http://www.quanser.com/), a Canadian company focusing on research and educational applications, started manufacturing their own haptic devices in 2004.
  
• Novint (http://www.novint.com/), a US company focusing on affordable, 3-dof haptic devices for the consumer market.
  

Focusing on one particular application, namely assembly simulation for the transport industry, two successive initiatives have placed Europe on the leading edge:

• iViP (integrierte Virtuelle Produktentstehung), a German R&D project, which started in 1998 for a period of four years, included industry partners such as BMW, Volkswagen, Daimler Chrysler, Siemens, Bosch, etc.
  
• Perf-RV (Plate-forme Française de Réalité Virtuelle - http://www.perfrv.org/), a French project over the period 2000-04, in which the transport industry was represented by Renault, PSA Peugeot Citroen, EADS, Dassault Aviation, Alstom, etc.
  

Especially Perf-RV has achieved very good results in the simulation of assembly tasks with force-feedback, and several industrial partners have now entered the phase of technology assessment for operational use (e.g. PSA and EADS). It shall be noted that, of all topics covered by the project, force-feedback was the area where the major advances were achieved.
Since a large number of former Perf-RV partners integrated the Intuition network of excellence, we can assume that Intuition is now on the front line with respect to haptics for assembly simulation. Of course, the same is not true for other applications like medical training or intuitive design.
Haptic interaction has stimulated a strong effort of research for the past five years in Europe. Here follows the list of past or ongoing projects funded by the European community, which involve haptic feedback and haptic devices.

• DIVIPRO:
  European project IST-1999-11421 (http://aig.cs.man.ac.uk/divipro/ ). The main objective of the DIVIPRO project (Distributed Virtual Prototyping) was to generate a virtual environment that integrated multi-sensory interaction (visual, audio and touch senses), real-world simulation utilities, such as collision detection, gravitational force, constraints recognition and flexible elements and a distributed environment that would allow multiple users, in the same or different locations, to work concurrently with the same virtual model. This environment could be used by different teams, such as design, assembly, maintenance and post-sales teams, throughout the life cycle of a product. The final system is a Distributed Virtual Prototyping Environment, which allows users in the same or remote locations to create and analyse new products without using a physical model, mock-ups or prototypes, but by means of realistic navigation and visualisation of the virtual model, using touch sensing for shape inspection and geometry manipulation through haptic devices.
  

• MUVII:
  European Project IST-2000-28463 (http://muvii.hpclab.ceid.upatras.gr/enter.html ). The initial idea was to develop wearable and cheap devices to be used in immersive theatres or in virtual museums. The haptic developments were made by CEA. The project came up with two interesting concepts. First, a simple 2dof haptic device that is wearable, and can be produced cheap wise. The main problem remaining was the association of contents with the haptic effects of the device. The second device was a device for 2 or 3 fingers which still need further developments before it could be commercialized.
  

• TOUCH-HAPSYS:
  European Project IST-2001-38040 (http://www.touch-hapsys.org/ ). This project envisages establishing a new generation of high-fidelity haptic display technologies. The newly developed systems will not only cover haptic interaction but also attempt to complement haptic information by visual and auditory input. On one side, the consortium will explore and develop new technologies, which will be used to significantly improve haptic displays. On the other side the psychophysical basis of human haptic perception will be investigated. One goal is to exploit haptic illusions to overcome fundamental technological limitations. Four demonstrators covering typical application scenarios with a critical technological challenge will be developed: Haptic interaction with biological tissues, haptic texture rendering and recognition, the simulation of rigid objects with clearly defined, sharp edges, and multi-modal volumetric exploration systems.
  

• CREATE:
  European Project IST-2001-34231 (http://www.cs.ucl.ac.uk/research/vr/Projects/Create/ ). The global scope of the CREATE project (Constructivist Mixed Reality for Design, Education, and Cultural Heritage) is to develop a mixed reality framework that will enable highly interactive real-time construction and manipulation of realistic, virtual worlds based on real sources. This framework will be tested and applied to cultural heritage content in an educational context, as well as to the design and review of architectural/urban planning settings. This project will use a haptic device in edutainment activities. Using a haptic device from PERCRO, the goal is that of taking pieces from the ground and reconstructing a temple.
  

• GRAB:
  European Project IST-2000-26151 (http://www.grab-eu.com/ ). The main aim of the GRAB project (Computer GRaphics Access for Blind people through a haptic virtual environment) is to eliminate some of the barriers that impeded blind people to have access to the computer and its applications, allowing them to have access to the 3D graphic computer world through the sense of touch and with audio help, by means of a new Haptic and Audio Virtual Environment (HAVE). The new HAVE allows its user to feel with his/her fingers the shape of the virtual 3D objects. This is achieved using a new 3D force-feedback Haptic Interface specifically developed in this project to touch 3D virtual objects both with the thumb and the index fingertips or both index fingertips while moving the hands in a desktop workspace. During the haptic exploration, the user can also receive audio feedback, both speech and non-speech. Thew new HAVE provides an integrated platform for the design and development of audio-haptic applications in different fields (architecture, art, aeronautics, medicine,..) but the project was focused on three applications for visual impaired people: an adventure game, a city map explorer and a chart explorer.
  

• PURE-FORM:
  European Project IST-2000-29580 (http://www.pureform.org ). The main objective of the Pure-Form project is the realization of the “Museum of Pure Form”, a virtual reality system where the user can interact, through the senses of touch and sight, with digital models of 3D art forms and sculptures. A selected set of sculptures belonging to the collection of partner museums are digitally acquired to create a digital database of artworks copies. The haptic interface devised for the MPF system is an exoskeleton for force feedback on the upper limb integrated with a haptic interface for fingers. The haptic interface for fingers is able to exert an arbitrary force of direction and breadth on the fingers tips. In particular, the forces generated during the contact with sculptures are transmitted to the operator by means of two thimbles placed on the tips of forefinger and thumb. Haptic Interface system has been first validated in a CAVE structure and then installed in temporary exhibitions of partner museums, where the visitors could haptically explore the digital sculptures immersed in the Virtual Environment.
  

• VIRTUAL:
  Project n. 1999-11030, funded by the European Community under the ‘Competitive and Sustainable Growth’ Program (1998-2002). The main objective of the VIRTUAL project (Virtual reality systems for perceived ergonomic quality testing of driving task and design) is to develop an integrated platform, based on Virtual Reality (VR) technology, including systems for testing and related experimental procedures, to be used for better studying and improving the ergonomic design of the vehicles, in the process of making the driving tasks simpler and more comfortable and then of increasing safety. The driving idea is to make available testing systems where a person can interact with a virtual representation of a vehicle and perform realistic evaluation tests on relevant aspects of the ergonomic quality of the vehicles, such as internal and external visibility, reach and operation of the main controls such as steering wheel and gear shift, perception of lateral and rear visibility during maneuvering. In the final level of the project, the physical response of the controls are reproduced by a generic force effectors, a 5 DOF exoskeleton, able to give the person the sense of reaching and operating a real control (existing and defined only in the virtual environment) coherent with its characteristics as represented in the virtual environment.
  

• ENACTIVE:
  European Network of Excellence IST-2002-002114 (http://www.enactivenetwork.com ) The general objective of the proposed Network is the creation of a multidisciplinary organisation with the aim of structuring the research, at a European level, on a new generation of human-computer interfaces, or Enactive Interfaces, such as they are shortly described here under. Enactive Interfaces are related to a fundamental “interaction” concept which is not exploited by most of the existing technologies. As stated by the famous cognitive psychologist Jerome Bruner, the traditional interaction with the information mediated by a computer is mostly based on symbolic or iconic knowledge, and not on enactive knowledge. While in the symbolic way of learning knowledge is stored as words, mathematical symbols or other symbol systems, in the iconic stage knowledge is stored in the form of visual images, such as diagrams and illustrations that can accompany verbal information or stay on their own.

State of the Art

The State of the Art of Haptic Hardware can be found here.

Links

This article is managed by the Haptic Interaction WG.
You might want to look at the Haptic Interaction Research Roadmap.