Staging

Language in virtual worlds

In the multimedia project Staging, CST works together with a number of other Danish research institutions on the topic of interaction with and within virtual worlds. The goal is to allow the user to speak as well as to use other modalities (for instance gestures via a data glove) in order to interact with autonomous agents in a virtual farm. To integrate verbal input with input from other modalities so as to allow as natural as possible a form of communication between user and machine, is a complex and innovative task. The system's language engineering component must be able to translate verbal messages and matching gestures to abstract meaning representations the agents can react to - of course in a coherent dialogue.

Dialogfragment in Stagings farmscenario

Enlarge

Video

The video demonstrates communication with a virtual agent. The software integrated three modalities: speech, pointing, and hand movements. The software consisted of modules for speech recognition, for speech generation, for modality integration, text parsing and semantic analysis, and for communication management. Communication between these modules was enabled by CORBA, which made it possible to run CPU-intensive tasks on dedicated servers. Most of the software was written in C++, but the communication manager was written in Bracmat. The front-end was always running on a Windows 98 PC, while modality integration, parsing, semantic analysis and communication management most often were done on a SUN SPARC station. Some of the hardware drivers did not survive the upgrade to Windows 2000, so the system is currently defunct.

This video was recorded by Patrizia Paggio and Bart Jongejan, Centre for Language Technology, Copenhagen, Denmark, on March 4, 2002.

Literature

Patrizia Paggio and Bart Jongejan. "Multimodal communication in the virtual farm of the staging Project." Proceedings of International Workshop on Information Presentation and Natural Multimodal Dialogue, Verona. 2001.

Patrizia Paggio and Bart Jongejan. Multimodal Communication in Virtual Environments. Multimodal Intelligent Information Presentation. Kluwer Academic Publishers, 2005. s. 27-47

CST Contact

Patrizia Paggio patrizia @ cst.dk