Category:Entertainment and Culture

From EuroVR Knowledge Base

Brief overview of area

insert reference

Entertainment and culture are application domains that can largely benefit from the use of VR/VE technologies. There is a clear additional value in applying VR/VE technologies to these domains. Also, in many cases, there are evident business models, behind this application of VR/VE technologies. Therefore, it is not surprising that entertainment and culture are among the first domains to adopt VR/VEs.

Wider use of VR technologies has been enabled by several factors: significant progress in hardware performance including 3D accelerators available in every contemporary graphics card, increasing availability of automatic 3D modelling tools and consequently also 3D models, and development of standards, which enable platform-independent representation of VR contents. Also, development of broadband multi-purpose communication networks is a big opportunity for popularisation of VR-based multimedia services.

Furthermore, wide availability of 3D computer games and movies based on 3D computer graphics results in increasing familiarity of users with 3D technologies and at the same time is raising user expectations. Young generations of people familiar with interactive 3D games and movies based on 3D graphics look for similar experiences in other domains. This may be a good opportunity for many institutions to increase attractiveness of their services.

At the same time, we can observe convergence of technologies in the fields of virtual reality, 3D games and next generation interactive 3D television. Games start to employ virtual reality technologies, many virtual reality projects are based on game engines, VR technology is used in television production. In this roadmap, we do not make distinctions between these fields; instead we look at the research field holistically.

Five main application areas are covered by this roadmap: cultural heritage, fine arts, next-generation interactive TV, 3D games and entertainment/ edutainment parks.

In cultural heritage, VR is being used mainly to enable people to access cultural resources that would otherwise be inaccessible to them. This includes remote access to museum exhibitions – including both digitised artefacts and reconstructions of real places. This also includes access to collections that cannot be exhibited in a traditional way because of the lack of space, high-cost or fragility of artefacts. An important element is possibility to create virtual exhibitions that draw from multiple sources, often distributed around the world. Virtual reality offers one more important advantage to the cultural heritage field – it enables virtual reconstruction of objects or places that have been partially or entirely destroyed or lost.

Virtual reality is also a new media for fine arts. Current technology is reaching the level in which equipment is robust and software is stable enough to be used also by people other than engineers, e.g. artists. This new media is very intriguing from a fine arts point of view since it can raise new forms of arts. For example, immersive drawing opens a totally new field to visual arts when painters can draw freely in 3D space, not only projections to 2D canvas. Also sculptors can work with intangible masses, such as triangle meshes and particles generated with 3D computer graphics. Dance, music, and theatre have been applying more and more VR technology, e.g., virtual set-ups, motion capture, and interaction devices to make art performances more interesting and expressive.

Next generation interactive TV is another promising application area for VR technology. The virtual studio has already found its way into television production. VR promises an increase in the quality and flexibility of television productions, at the same time reducing associated costs. Also, entirely new forms of expression are possible for television, enabling high dynamism of digital contents and active participation of viewers.

A field strongly related to VR is 3D games. For many years, the 3D game industry has been one of the main drivers of VR due to its large audience and its strong economical basis. One of the most significant examples is the emergence of fast and cheap 3D graphics cards (GPU) over the last several years. Latest 3D games use cutting edge software technologies, such as real-time 3D and sound rendering and artificial intelligence engines, which are also the technological basis for VR applications. Currently, 3D games are a tremendous support for dissemination of VR/VE applications and for experimenting with new technologies.

Another field where VR technology can play an important role is edutainment (edu-cation + enter-tainment). 3D interactive games can be used as a playground to educate people about our history, arts, science, etc. This is sometimes referred to as ‘serious games’. The most advanced systems employing VR edutainment have appeared in big amusement parks in form of entertainment shows with a highly educational content in the form of collective or individual games or immersive experiences in real or synthetic environments. Relevant examples of this kind of products are the developments of the Foundation of the Hellenic World, which allow people to participate in ancient Olympics games so as to learn about them in a fun and engaging way. Other interesting examples are some of the developments of the Belgium company De Pinxi, where one can find different exhibitions for Theme Parks around the world, some examples of their productions are: Human Body Travel – a collective game to learn about the human body by travelling inside it, or The Mexican Adventure (A taste of death) a game to learn about the ancient Mexican culture around the dead, etc. Famous museums such as the Smithsonian Planetarium are using VR and edutainment to educate their visitors about astronomy and the secrets of the universe.

The application of VR to edutainment is becoming more and more important also for the academic community. Examples include the annual conference on Edutainment and a special issue, on "VR based Edutainment”, of the Springer journal on Virtual Reality.

Vision and potential scenarios of use

Culture

Cultural Heritage Site

A visitor arrives at an interactive cultural heritage site. The site covers ruins of an ancient city and it features a VR reconstruction of the city and its inhabitants. At the entrance, the visitor is provided with lightweight high-resolution see-through glasses (that enable him/her to see the original ruins and a reconstructed 3D environment) as well as miniature headphones able to reproduce high-quality 3D sound. The level of transparency of the glasses may be easily adjusted to enable more natural or more virtual view. The visitor puts on the glasses and the headphones and the ruins are complemented with a detailed 3D reconstruction of missing or destroyed architectural objects, animated people wearing ancient dresses and speaking and behaving in a natural way, as well as flora and fauna typical to this place. A virtual guide in the form of a flying angel appears in front of the visitor. The guide asks a series of questions to adjust the visit depending on the visitor’s background and personal interests. The visitor may answer the questions either by speaking or may activate hand tracking enabling him/her to use a contextual 3D menu. After the initial setup is completed, the guide starts to explain the history of the ancient city indicating to the visitor where to go and what to observe. The animated objects of the VR environment (people, animals) avoid visitors as if they belonged to the virtual world. During the whole visit the visitor may ask questions to the guide, he may also skip some parts of the itinerary or request in-depth explanations. The guide is equipped with an advanced AI engine that allows it to react automatically when the visitor shows symptoms of fatigue or lack of interest in specific details. At any time, the visitor may switch off the guide and use only the 3D menu to access information. Also, particular elements of the 3D reconstruction may be switched on or off, while the whole VR environment is still fully functional. While leaving the site, the visitor is offered a photograph “taken at the site”, showing him/her surrounded by the ancient environment – buildings, trees, people and animals.

Virtual Museum

The European Museum of History and Culture has been established in 2012. It features a unique collection of European ‘digital’ artefacts, digitized documents and recordings. All European countries continuously contribute to the museum. The museum is available only as a web service. The museum can be accessed in a number of ways. There are several immersive VR installations both in Europe and on other continents that allow visitors to access the museum (as well as other museums) in a highly immersive and interactive way. These installations provide high-quality 3D real-time visual and aural renderings of the virtual environment. Special haptic devices allow visitors to touch and manipulate high-resolution models of artefacts. Visitors may plan a visit to the museum before entering the VR installation using a simple web interface by either selecting an interactive scenario focusing on a specific element of culture or history, or selecting a number of 3D reconstructions of places to visit, or a number of 3D objects to examine. Importantly, the museum may be also accessed using low-end devices such as personal computers, interactive TV sets, or even small portable devices, such as camera phones. This enables the use of the museum’s resources at schools, at home or while travelling. The contents is adapted automatically to reflect the capabilities of the access device, e.g. performance, display resolution, interaction devices.

Interactive 3D TV

The i3TV company provides high-quality interactive real-time 3D TV contents. Users may receive the contents with 3D multimedia entertainment installations at home, personal computers, and portable devices. The company provides several channels of live transmission as well as on-line access to repository of archival contents and supporting material. The broadcasted contents are a mixture of audiovisual streams and synthetic 3D elements. As an example, during an athletic event transmitted by i3TV a user may request detailed animated 3D reconstruction of a successful high-jump attempt. A user may slow down or stop the animation at any time, see it from different angles, replay or even compare two attempts in the same 3D environment. The second attempt may be either one of currently provided attempts or an archival attempt taken from a library of resources. After finishing analysis of the attempt, the user may return to watching the live contents.

Entertainment

Interactive Theatre

A bird catcher comes to the stage while the previous scene is ending. The virtual set-up, built with several immersive projection screens, change their images. A misty forest with animated trees is seen by the audience. The bird catching scene starts and the actor walks to the middle of the stage. The acoustics of the whole theatre change to be more reverberant since the previous scene happened in an open field. The walls of the theatre are covered by sound absorbing material and virtual acoustics are provided with microphones and multiple loudspeakers, all mounted on the walls.

The bird catcher tells the audience a story about birds that he is catching. He asks the audience to be as silent as possible so as not to frighten the shy birds. As spectators hold their breath, computer generated birds appear on the screens on the stage. The bird catcher tries to throw a catching net (tracked by the virtual set-up system) on the birds, but they are too fast for the catcher and they fly away screaming. The scene continues and suddenly a giant dragon sound is heard from the back left corner of the audience area and a warm and humid air flow is covering the audience area...

Virtual Studio Production

EduTV is a new TV station specializing in production of interactive educational TV contents. The TV contents are broadcasted via satellites and are also available on the Internet – both live and through an on-line archive. The TV content production pipeline in EduTV is purely digital using the virtual studio technique. A presenter is acting in an empty or almost empty space with a specific-colour background – bluebox. Using the chroma-keying technique, the background is replaced by naturally-looking computer-generated 3D environment. The environment is much cheaper to create and modify than traditional physical setup. It provides very good visual quality and can be manipulated during live broadcast.

Before the live production an operator adjusts the environment to the current requirements – the presenter will be accompanied by a guest – so additional space must be added. The operator knows the subject of today’s program and he/she is preparing supporting material in the form of 3D models of objects. During the live production process, the operator controls the state of the virtual scenery, can modify and animate elements of the scenery (e.g., a virtual display showing video), can add and remove objects to/from the environment, etc. The presenter is currently explaining differences between two species of birds, so the operator adds corresponding animated models to the scenery. The birds fly around the studio and finally sit at a virtual table. The presenter takes them both into his hands and manipulates to indicate differences. Then, the presenter asks questions to the viewers. Viewers may immediately respond using interaction devices in their set-top-boxes or computers. Statistics of the answers are immediately visualized in the studio. Viewers can also participate in the production by voting, asking or answering questions, manipulating objects, etc.

3D Games

Roger and John have met at a new Entertainment Centre to play with a novel version of the good old Donkey Kong game. Roger starts the game in front of a wall-size 3D screen where he sees barrels rolling towards him. When photorealistic barrels reach him, he has to jump over them. Simultaneously, he has to avoid hitting virtual roof elements to proceed in the game. A few cameras mounted to the wall interpret his movements. If too many barrels are in his way he can shout loud magic words to blow up the barrels so he can pass safely. By mistake he jumps too low and hits the barrel and his shoes give him tactile feedback. As Roger failed, it is John's turn to play.

After one and a half hours of playing Roger and John are exhausted and feel too tired for more of this type of gaming. However, they feel great since there is no need to go jogging anymore today; since this novel 3D game is very good for sport also. So, they can now relax and play their virtual air guitar - karaoke game...

Ubiquitous Gaming

A young person appears at the end of the street. Handling a small electronic device, similar to popular gamepads, in his hand he looks around cautiously through an eye mounted see-through display. The display – wirelessly connected to a portable palm computer equipped with a GPS system and the newest version of the "Alien Attack: Overkill" game – shows a small virtual creature superimposed on the view of the real street. Suddenly, the small creature jumps squirting a poisonous lethal substance directly at the young player. Instantaneously, the player dodges and manipulates the gamepad with build-in positioning and tracking sensor. With long trained skill he points a cross-hair cursor directly at the running Alien and presses fire button right on time. At this same moment the virtual Alien disintegrates in a small but spectacular explosion. Proud Earth defender scores another not so easy hit and is being promoted to the rank of captain.

Several minutes later, two women and one man meet on the nearby square crowded with people. Each of the trio approaches the square from a different direction, guided by virtual signs visible in their head-mounted see-through displays. Some of the directing aids have been visible in the form of virtual arrows superimposed on real buildings; some of them have been given in the form of logical quests to be resolved. Their mobile devices attached through a wireless network to a central database keep tracking their positions through a combined GPS-GSM-WiFi location based service. They do not know each other. The only sign identifying them in the crowd is a small blue pulsing triangle over their heads. As soon as they come near, their devices retrieve a quest from the central database – to solve it, they should work as a team putting into correct place fragments of a logical puzzle collected individually through the few last hours of the game. Although one of players was a newbie in this game, collaborative work helped her to find a solution that has been sent back to the central game engine. Solution was correct and the entire group has been awarded: virtual signs directed them to the adjacent restaurant where they have been treated to a free lunch and a small hard wood pedestal with a coded inscription as a trophy founded by the restaurant – who is the sponsor of the game. Of course, the coded inscription is in the form of a small tag with several symbols. While incomprehensible for incidental viewer, the tag observed through the see-through displays, reveal a much bigger virtual gold cup.

Tourism

A number of tourists are visiting a city. The tourists are equipped with lightweight see-through AR displays connected to mobile devices (e.g. smart-phones), which based on information about the user location (GPS) and photographs/video of natural and architectural objects taken by the mobile device, can provide on request additional visual or narrative information about the places or buildings. The information is superimposed on the actual view. The information can be filtered or customized for specific users needs. The system can also perform a function of a virtual guide. Based on information provided by the user (time, budget, interests), a sightseeing path is proposed taking into account the most important or interesting places in the city. The system can also give users directions guiding them to the nearest hotel, parking, gas station, shop, etc. The system enables communication between users. The system tracks positions of registered users, and based on the users’ profiles can match users or groups of users (e.g., tourists from the same country or tourists interested in meeting other tourists) and display their location or guiding information. The system can also be used for "keeping in touch" between users by identifying location of people belonging to a group (family, friends, etc.).

Content Creation

Mike is currently working on a new video game as a freelance 3D artist. The deadline to deliver 3D data is very soon and the work to be done seems to be huge: he has to provide a detailed 3D model of a large futuristic city, which looks like Paris. Fortunately, he has just received the new version of Easy-3D-Content, which is really amazing software.

A first module generates 3D environments (buildings, roads, parks, etc.) from a satellite map retrieved from Internet. The generation can be automatic using a procedural technique, but to get the futuristic style, Mike has to provide some samples to the software. In a first step, he is creating a few buildings using a classical modeller. Then, he provides those models to the software, which deduce a shape grammar from them. When the grammar is ready, the tool creates thousands of buildings automatically, in a few minutes only. The style is always the same, but all the buildings are different.

Another module makes it possible to easily incorporate data directly extracted from pictures. Data are either geometric (3D reconstruction from several points of view) or photometric (textures, materials).

To complete its work, Mike has to lay out some particular places which are important in the game scenario. He is selecting some relevant 3D objects in a large community database. All objects have parametric description, so it is rather easy to customize them. Then, using a high-level description, he is describing the semantic links between objects. Its description looks like this: "put a table in the living room and three chairs around", "fill the shelves with books", “move the window in front of the door”, etc. The environment is built in real-time. When he drags any object to adjust its position, relations are maintained. For instance, when the table is moved, all the chairs are automatically relocated.

Useful links