Melanie Baljko
melanie@cs.toronto.edu
http://www.cs.toronto.edu/~melanie/

Computational Simulations of Mediated Face-to-Face Multimodal
Communication

Melanie Baljko
University of Toronto

Several empirical studies have provided descriptions and analyses of
the communicative actions that individuals perform in communicative
exchanges in which one of the interlocutors has a communication
disorder and uses a \em{voice-output communication aid} (VOCA).  When
provided with various input actions or sequences of input actions,
these communication devices produce spoken utterances through speech
synthesis.  These studies have identified the various ways such
so-called "aided-communicators" employ --- in isolation or in
combination --- their various \em{modes of communication}, such as
gesture, vocalization (non-speech sounds), facial expression, eye
gaze, torso position, and speech (i.e., the synthesized speech which
is provided by the communication aid).

In this presentation, I will describe the factors thought to affect
the strategies employed by aided communicators, such as his or her
perceptions of the communicative environment, of the other
interlocutor(s), and of the condition of his or her own embodiment.
These communicative strategies demonstrate the ability of
communicators --- even though highly constrained due to physical
disability --- to adapt, both from one communicative exchange to the
next and during a particular communicative exchange.  I will describe
a formal model that was developed to account for these phenomena, its
computational implementation, and the methodology that was developed
for its evaluation.  Preliminary analyses show that the data derived
from the computational simulations resemble empirical observations.  I
will also discuss how, although initially developed for use in the
design of communication aids, the formal model has potential
applications in other application domains, such as in multimodal
interface managers and in the architectures of embodied communicative
agents.