Evaluation Methods Usage in Projects

From EuroVR Knowledge Base

This article presents a set of lists of projects with common evaluation methodologies and which INTUITION Partners are or have been involved in. This list has been created according to the data extracted from INTUITION Projects uploaded to the Knowledge Base. The scope of this article is not to gather all the evaluation methodologies identified at the different Projects in the database, but rather offer information on which evaluation methodologies are more commonly used across Projects.

From the data gathered, the following common evaluation methodologies have been identified: Questionnaires, Interviews, Laboratory Experiments, Experts Review, On-Site Experiments and Observation. Some feedback on these methods is provided.

Contents

Questionnaires

  • View of the Future - We used a large number of questionnaires to investigate usability, presence and effects as a result of using VR technologies. Some questionnaires were from validated sources, while others we had to develop for the project based on existing research and the needs of the user companies. A battery of tests were prodiced as part of a manual which also included objective tests.  The debate continues regarding the use of subjective versus objective tests, but for best results the combination of both shold be considered.   More research is needed in methods which are user-friendly and robust.
  • SIMILAR
  • ENORASI
  • PERF RV
  • IRMA - A large set of questionnaires were used pre-test, during test and post-test with a group of real users performing tasks in a virtual environment using a variety of display devices (projection, monitor, headset) and interaction devices (mouse, joystick, glove).  The purpose was to investigate user feedback on usability, utility, presence and effects.  The questionnaires used had been developed over years of experience and based on existing research by the University of Nottingham, in conjunction with some noted work (e.g. SSQ). 
  • INSCAPE
  • EXTRAS
  • LAHYSTOTRAIN
  • EVICS
  • PHASE
  • CREATE
  • ALBATROS - In order to measure Presence, two different types of questionnaires were used as part of the laboratory experiments conducted: Presence questionnaire (PQ) and Slater-Usoh-Steed Questionnaire (SUS).
  • INTENTIO - A set of questionnaires were developed in order to assess the effects of pre-test training with BCI on users in VE, as well as to identify which aspects of the VE should be modified to improve such effects. Anxiety and attention measures were also performed in the form of questionnaires.
  • VIZCON
  • AMUSEMENT


Interviews


Laboratory experiments

  • View of the Future - A number of lab experiments were performed by human factors experts and the development team to examine usability issues such as posture, comfort, fatigue, etc., of different VR set-ups and interaction devices. Based on this work recommendations were provided to assist the design process.
  • AVPRC
  • PERF RV
  • CADPIPE
  • PC-VR
  • GRACE
  • Helma
  • ALMA
  • EVICS
  • PHASE
  • New body postures
  • ALBATROS - An experimental study was performed where participants were divided into two groups: experts and non-experts. The task given to the partipants was to make a diagnostic analysis on a virtual patient in a VE that simulated an emergency situation. Measures included two types of presence questionnaries and psycophisiological measures (Dermal response system)
  • INTENTIO - The purpose of the experiments conducted was to assess the percentage of success at classification tasks, that is to say, how well the machine performed at identifying the commands prompted by the participants by means of the BCI. The experiments also allowed to study the effects of training in both VE and other environments.
  • VIZCON
  • 4D LIVE
  • VidKamPaik
  • ARARAT
  • VBE2
  • PROGNOS
  • AMUSEMENT
  • 3D MAASTO


Expert review


On-site experiments

  • View of the Future - In this project we were fortunate enough to have good support from our user companies. We were given access to designers and developers on-site which greatly helped the development and implementation of the project results. We had real users providing real feedback. A combination of questionnaires and interviews were used.
  • PERF RV
  • IRMA
  • COHERENT
  • LAHYSTOTRAIN
  • Helma
  • PHASE


Observation

Retrieved from "http://kb.eurovr-association.org/index.php/Evaluation_Methods_Usage_in_Projects"
Log in / Create account  
Views