Category:Evaluation and Testing
From EuroVR Knowledge Base
Contents |
Brief overview of area
It is equally important that evaluation work is carried out in end-user companies in order to examine the varied needs of the actual end-users and monitor the success of the transfer of VR technology into real working industrial applications. This allows us to identify key usability issues and further user requirements, from which we can work to bridge any gaps between company expectations, the actual performance of the VR/VE system and any technical constraints.
Evaluation of a part or the whole VR/VE system will indicate whether it performs as it was intended, and also whether participant performance is at an adequate or expected level. For example, evaluation can show whether a VE designed to train participants to learn how to complete a specific task actually achieves its goals. Performance on such a task could be compared to traditional methods of training to show whether VR/VE has any added value. Sometimes there is no difference between traditional and VR/VE training, however, participants may find one training method more acceptable than another.
User tests are also often conducted to examine which factors are important in order to perform a particular task and whether certain display systems meet these needs better than others.
Not only does evaluation provide feedback for the evaluator or the company in question, but it also provides important feedback to the participant in terms of their progress on a particular task. It can help to improve task performance and increase participant motivation. Evaluation can highlight any strengths and weaknesses of a system, for example, in terms of usability of the VE/devices/interface/interaction technique, and can also assess the suitability of VE participation for particular users (e.g. some users may consistently report sickness symptoms and do not habituate to VE use), as well as the health and safety of a particular VR/VE system.
Evaluation also allows comparison across different systems or components of a VR/VE system and can assess whether any changes made to a system produce the required outcome. End-user behaviour can be observed and assessed on predetermined criteria and any negative symptoms of VR/VE use can be monitored.
During evaluation industrial VR/VE users may consider a number of different factors such as cost effectiveness, precision, functionality and quality of the technology they are using. For example, when VR/VE is being used for design purposes a number of performance measures could indicate how successful the use of technology has been in achieving the company’s overall aims, e.g. reducing the number of physical prototypes being built (cost), and accelerating the design process (time).
For many users it is important to assess that the technology being used actually meets, and is acceptable for, the industrial/academic user’s application needs. There are problems with defining some of the measurable performance indicators in VEs, for example, usability and presence. Researchers often view usability and presence as incorporating different elements; thus there are a range of tools available which inevitably measure different proposed elements of these concepts.
Vision and potential scenarios of use
Application of training for medical emergencies
Doctor Zivago wants to train new doctors in medical emergencies. Currently, he uses a mannequin as the patient and actors surrounding the scene in order to elicit the stress of the situation. He wants to evaluation if this procedure could be implemented in VR/VE with virtual actors instead of actual ones. His main interest is that the VR system could elicit the same stress to the trainee. The evaluation should focus on this issue. Doctor Zivago must evaluate the level of presence of the trainee in order to asses if the system could substitute the current procedure. Dr Zivago must also evaluate a series system, in order to choose the most appropriate one, that will enable the trainee to have the desired levels on immersion, free of movement, sound, smell, altogether realism. Dr Zivago must also choose from a variety of different patient models (both maneken, behavioural model, 3D representation). He should also evaluate the different agents will be present in the simulation. Dr. Zivago would like to determine if the results of the training can be extrapolated to the real world emergencies situations. (validity of the results).
Path Evaluation in an radioactive environment
Mike would like to use a VE, representation of their Nuclear plant, to determine which is the best way to perform an maintenance intervention. He needs to define (to include) several intelligent avatars representing operators. He also needs to assign tasks to each avatar, that involves navigation, interaction with objects in environment. The paths that operators take along in the environment, should be analyzed to determine the amount of radiation received by each operator. Several alternatives can be planned and analyzed to find out the safest one. He needs to find an easy way to manipulate the avatars since he does not have many experience in interaction with 3D environments. (evaluating different ways of interaction) He needs to insert into the VE data about radiation sources that can be used for the radiation dose calculation. The radiation fields should be visualized 3D; A possible result could be some figures of the amount of radiation of each avatar received, in order to select the best scenario.
Interaction during design review
VR is used for design review. Power wall is used to present 3D interactive models of new products. Users will be everyone involved in the design process and review, including casual VR users (e.g. managers). They want to interact with the product to: • Move around • Explore different viewpoints – find correct position for the task (not have to search for it) • Comparing kinematics of seat/driving position • Test functionalities (opening doors, ashtray, sunroof, etc)
John has to evaluate a new interaction device, using an intelligent speaking interactive virtual agent (= the VR system). This new system allows a user to get direct feedback from the virtual agent. For example, when inside the vehicle, the user can reach out and touch the seats and dashboard and gets tactile feedback (resistance and textures).
How can we evaluate whether the new system is better than previous systems?
Evaluation criteria: • Comparison with previous systems – cost effectiveness, added value • Usability for different user groups • Acceptance (psychological and emotional) • Technical accuracy
Useful links
To keep up to date in this area you may like to join the Evaluation and Testing working group INTUITION WG2.8
References
Gotts, G. and Cox, T. (1988). Stress and arousal checklist: A manual for its administration, scoring and interpretation. Swinburne Press.
Kennedy, R.S. Lane, N.E., Lilienthal, M.G., Berbaum, K.S., and Hettinger, L.J. (1992) Profile Analysis of Simulator Sickness Symptoms: Application to Virtual Environment Systems. Presence 1(3) 295-301.
Kennedy, R.S., Lane, N.E., Berbaum, K.S., and Lilienthal, M.G. (1993). Simulator Sickness Questionnaire: An enhanced method for quantifying simulator sickness. The International Journal of Aviation Psychology, 3(3), 203-220.
Nichols, S.C. (1999). Virtual Reality Induced Symptoms and Effects: Methodological and Theoretical Issues. PhD Thesis, University of Nottingham.
Nichols, S., and Patel, H. (2002). Health and safety implications of virtual reality: a review of empirical evidence. Applied Ergonomics, 33, 251-271.
Pages in category "Evaluation and Testing"
The following 6 pages are in this category, out of 6 total.
