Interactive Storytelling in VirtualEnvironments: Building the “Holodeck”Marc CAVAZZA (1), Ruth AYLETT (2) Kerstin DAUTENHAHN (3),Clive FENCOTT (1) and Fred CHARLES (1)(1) University of Teesside, Middlesbrough, TS1 3BA, UK{m.o.cavazza, p.c.fencott, f.charles}@tees.ac.uk(2) University of Salford, Salford, M5 4WT, [email protected](2) University of Hertfordshire, Hatfield, AL10 9AB, [email protected]. In this paper, we introduce an ongoing research project, which aims at developing an immersivestorytelling environment. In such a system, which is inspired from the “Holodeck” device, human users will beable to interact with artificial actors through speech, while sharing the same physical environment. Once on theset, the users will be able to participate in story generation by influencing virtual actors or altering the commonenvironment. An essential aspect is that user involvement will not have to be permanent, and that the users willbe allowed to leave the set to watch the story unfold in their absence and to return at a later stage. We discusssome fundamental AI problems with which the implementation will be faced and relate them to narrativetheories and models of storytelling. More specifically, we describe how users can interfere with the virtualactor’s plans to influence story generation in terms of interaction1. IntroductionOur “Holodeck” project aims at developing an immersive story generation system supportedby an Intelligent Virtual Environment. The system would appear to the user as an immersivevirtual environment enabling physical interaction (i.e. through a CAVE or other immersiveprojection display). In this environment, the user would be able to interact with a cast ofvirtual actors (VA’s) sharing the same environment and addressing them in natural language.An essential feature is that the user involvement should not be a permanent one. Instead, theuser should be allowed to participate in the action, helping to create the story and leave thestage to watch the story unfold as a consequence of his/her intervention. Another essentialaspect is that the user would still be able to get back into the story at some suitable time, oreven to influence it from his spectator position, for instance by interfering with the emotionalstate or the behaviour of VA’s. This new form of entertainment would implement in its ownway the convergence of VR, cinema and computer games as described for instance byAltman and Nakatsu [1] (see Figure 1), and has obviously been influenced by [2]. Therelations between VR and storytelling are complex and even though our main purpose is todevelop an interactive storytelling system, these relations are worth exploring into somemore details. Story-telling and narratives are more than an enjoyable form of entertainment.Humans are social animals, a child needs a social environment in order to grow up andbecome a social being who is able to survive in human society. It has been argued thatnarratives are the fundamental structure by which humans organise their experiences andmemories, in particular of the social world. New events are interpreted in terms of oldstories, we know what we tell and we tell what we know (Schank & Abelson in [3]; [4]).Figure 1. Convergence in VR and Storytelling (courtesy of Dr. Ryohei Nakatsu).Story-telling plays an important role in the development of a child’s understanding of theworld and its relationship to the world [5]. From an evolutionary perspective, it is argued thatstories have been the most efficient means of understanding and communicating about thesocial world (Read & Miller in [3]; [6]). Narratives are more than representations of reality,human knowledge about ‘reality’ is constructed around stories. On the level of theindividual, stories are important in order to construct and re-construct one’s autobiographical‘self’, or ‘centre of narrative gravity’ [7]. According to Bruner [4] narrative operates as aninstrument of mind in the construction of reality. Thus, story-telling in an intelligent virtualenvironment (IVE) is more than an entertaining add-on, it is a important requirement in orderto create a believable (virtual) world, in order to allow a human to be socially immersed andbeing able to understand this world. Believable virtual environments need to be socialenvironments in order to support a user’s social presence [8]. An actor in the “Holodeck”scenario needs to be social, it needs to know about other agents and their relationships in thevirtual environment, in particular it needs to know about the human. Additionally, abelievable social world is a world which presents itself in terms of narrative, so that it canmeet a user’s cognitive needs to interpret, understand and interact with the world in terms ofstories [9]. Basic social requirements of such narrative agents are 1) the ability to recogniseand identify other agents, 2) the ability to establish and memorise direct (one-to-one)relationships with other agents, and 3) the ability to monitor and memorise third-partyrelationships, namely relationships between other agents in the shared environments [6]. Generally, story-telling and narrative is understood in term of human language. Here,virtual characters are able to interact with a human by means of spoken or written language(e.g. [10]. However, in addition to such ‘stories-in-words’ a narrative can also be expressedas ‘stories-in-actions’. Thus, in the “Holodeck” scenario the believability of the narrativevirtual actors will depend on their ability to communicate with each other and the human byusing language (in particular in situations where the human is a spectator or ‘on-stage’ withthe other agents), but narrative can also be expressed in non-verbal behaviour and the wayactions are organised and expressed in a consistent fashion [11]. Where narrative is created ’on the fly’ bottom-up from interaction, as we are proposing,rather than as is more usually the case, top-down from a script, there may be no singlenarrative thread. Different narratives might be extracted from the same mesh of events andinteractions from the perspective of different characters. The focus of this work is to createan environment in which events and interactions allow the user to ‘storify’ his or herexperience, whether derived from participation or from observation of other characters.Figure 2. User Interfering with VA’s Plans. However, a