We are developing a prototype storytelling robot for use with children in rehabilitation. Children can remotely control a large furry robot by using a variety of body sensors adapted to their disability or rehabilitation goal. In doing so, they can teach the robot to act out emotions (e.g. sad, happy, excited) and then write stories using the storytelling software and include those emotions in the story. The story can then be "played" by the remote controlled robot, which acts out the story and the emotions. We believe that this robot can motivate the children and help them reach their therapy goals through therapeutic play, either by exercising muscles or joints (e.g. for physically challenges children) or by reflecting on the expression of emotions (e.g. for autistic children). We use an innovative design methodology involving children as design partners.

Video games have been used in the past for a variety of therapeutic and rehabilitation situation (Griffiths 1997) and several studies - mainly set in a clinical environment - have shown how those games can be used as a mean to achieve desired functional goals (e.g. Vilozni, et al. 1994). The rich literature that describes the use of puppetry as a therapeutic, educational, and social tool (e.g., White, J.& Allers, C. T.,1994, Vidler, 1972; Currant, 1985; Carter, 1987; Caputo, 1993) also shows the benefits of technology. For example researchers have found that the use of puppets as a mediating technology can produce more expressive communication patterns among children who are in crisis or have on-going emotional, behavioral or academic challenges. Technology can be used by a child to confront what may be too painful or difficult to express otherwise. By telling a "story" with these props, children can "play out" their feelings just as an adult may "talk out" their difficulties with a therapist (Cochran, 1996).

Here we introduce a storytelling robot technology for therapeutic play with the hope that it will provide a richer environment useful for children with a variety of challenges, as well as the long term motivation needed to reach children's therapeutic goals.

The University of Maryland Human-Computer Interaction Laboratory (HCIL) has done extensive work in interaction design and evaluation with a unique program focusing on the design of children's technology (Druin, 1999a; Druin, 1999b). One of the more unique projects to come out of this work has been a prototype robot for storytelling (Druin, et al., 1999; Montemayor et al., 2000), which demonstrated the benefits of working with children as active partners in the design of new technology. This storytelling robot was originally designed for all children, without consideration for special challenges. Children using the storytelling software could write stories and select emotions from a list of pre-programmed emotions (e.g. happy, sad, surprised). The story could then be "played" by the remote controlled robot, which told the story (playing recorded or synthesized speech) and acted out the emotions at the appropriate time in the story.

The collaboration with AnthroTronix is broadening the scope of our storytelling robot research to emphasize designing tools for the purpose of children's rehabilitation.

AnthroTronix is a startup company; part of the Rehabilitation Engineering Research Center led by The Catholic University of America and the National Rehabilitation Hospital. AnthroTronix designs and develops robotics and virtual reality interfaces to support children’s rehabilitation. The company has developed a system that augments human function by providing personal augmentation devices (PADs), controlled by physiological signals, for the purpose of navigating and manipulating the external environment under their control. The system utilizes a variety of sensors connected to detect input (or "gestures") from the user. Software is then used to assess and process this input, and to map the input to appropriate outputs. The information fed back to the user depends on the intended outcome. For example, the systems can be used to train the user to learn input-output mappings, to enhance performance, and to assess performance.

This paper first reports on a pilot study using a video game with AnthroTronix assistive technology, which illustrates the potential effectiveness of therapeutic play and some of its limitations. Then we describe how we extended the storytelling robot to enable children to directly control the robot, to "teach" the robot how to act out the emotions. This was accomplished using AnthroTronix assitive technology, allowing for the easy adaptation of sensors to support children's abilities or challenges. We believe that a robot can provide a strong motivation for children to reach their therapy goals by exercising the appropriate physical ability or emotional expression.

A study was conducted by Rinaldi with children who have mild to moderate cerebral palsy (Rinaldi, 2000). Four children were monitored using a SuperNintendo video game system. The game was used with adaptive inputs, which challenged the participants’ isolated motor control and range of motion. For example, instead of pressing a button to make a game character jump over an obstacle, children had to bend their arm beyond a certain threshold. The input sensors were designed to provide alternative switch access by allowing the switch threshold to be adjusted as the child improves or changes.

The games were used the children at home during a 12-week period. Therapists gave the children exercises and the use of the game was recorded. Strength and range of motion were recorded at 6 weeks interval along with other qualitative information. The findings indicate an overall improvement in range of motion with game use but no changes in strength. The parents indicated that the response to this type of therapy was positive. Everyone enjoyed having the system at home and felt it was safe and easy to use. This case study highlights the benefit of therapeutic interventions that are administered in a challenging, familiar and fun environment that is integrated into a child's daily activities.

Feedback from children and parents made it clear that using the Nintendo game broke the routine therapy exercises but also suggests that the initial excitement might fade after a few weeks. This of course highlights the need for longer studies but it was conjectured that two factors played a role in the decline of interest after 12 weeks. First, an arm movement is slower than the press of a button, which resulted in poor performance in the game, a great disappointment for the children. Second, it was conjectured that the repetitiveness and the complete lack of creativity in the Nintendo activity might be a hurdle in any attempt to alleviate the boredom of daily therapeutic exercise.

This leads us to believe that a creative storytelling robot might provide a long-term motivation that is needed to help children in rehabilitation.

This story was entitled Michelle:

The name, concept, and the development of the PETS prototypes came about because of children. We develop new technologies for children, with children in an "intergenerational design team." This current team consists of children (ages 7-11 years old) and adult professionals, with experience in computer science, education, art, and robotics. Our research goals include creating new technologies as well as understanding better ways to give children a voice in the technology development process. An outcome of our research has been to develop the methodology of "cooperative inquiry," research methods that support the partnership of children and adults in developing new technologies (Druin, 1999a).

For this new part of the project our design team is also comprised of rehabilitation staff. The early phase of the design has been conducted solely with our existing team of child design partners. Children with disabilities will join the team later as informants during the refinement of the design and of course as testers of the refined prototypes. One of our goals is to design a product that will have the potential of being both a commercial toy for all children and a rehabilitation tool for children with special challenges through the addition sensors and the development of the rehab methodology.

From the design sessions with our child design partners emerged several findings:

First it was made clear that the storytelling robot was a very powerful tool to engage children in discussing emotions, their expression or the risks of misinterpreting expressions of emotions, leading us to believe that PETS could be used as a rehabilitation tool for children with autism.

Figure 2

Sensors mounted in arm bands and in a hat allow a user to directly control the robot.

The movements can be recorded and saved to be used later in stories.

Second, despite the great excitement created by the robot when used by the children, one of the most obvious limitations of the initial prototype was that the emotions—or any other movement of the robot—had to be programmed and "pre-stored" in the software in order to be used by the children for their stories. Originally, software was created to control the robot directly via a graphical user interface. But it soon became obvious that a more natural way to operate the robot was to enable children to teach the robot through their own body motions.

A first design session was organized during which children tested a car remotely controlled with an instrumented glove. The children all enjoyed the direct control of the car (comparing it to the buttons or joysticks), and liked the absence of wires between the car and the glove. On the other hand, they also found the glove uncomfortable but most importantly found the proposed mapping between the body movement and the car movement very confusing. They had to bend their wrist up or down to go forward or backward, and spin their wrist right of left to turn, but constantly confused and combined the movements which send the car in seemingly random direction. This session clearly highlighted the need for a direct correspondence between body motion and the motion of the robot. Ideas proposed by the team included:

• the need to use wireless control to avoid tangles of cables between the sensors and the computer;
• embedding the sensors in agreeable soft accessories such as bracelets or hair bands;
• and decorating those accessories with elements that resembled the object or robot part being controlled in order to clarify the connection between sensors and actuators (e.g. a miniature hand might decorate the sensor that controls the hand of the robot, a wheel might decorate the sensor chosen to turning the car)

Based on these ideas, a new prototype interface was developed with sensor controls of the storytelling robot. Sensors (accelerometers) were imbedded in two armbands - to control the 2 arms of the robot- and in a hat to control the head. An additional sensor (built in a shoe sole) allows the entire robot to spin when foot pressure is applied to the front of the shoe (this is clearly not the best way to control a spin but additional refinements will be made to this in the near future).

Early feedback from about 3 hours of trials suggest that our design team children are able to control the robot and that excitement of using the robot for storytelling was heightened by the increased sense of control over the robot. An unexpected observation is that both children and adults’ attention is strongly attracted to the robot - and not to the person controlling the robot. After a few adjustments and corrections of the control system, children were able to control individual movements of the robot, while the randomness of the movements of the robot—when the children was not precisely controlling the robot—gave it a compelling liveliness.

The controls can be recalibrated to match the amplitude of the child’s movements and can also be reassigned. For example, the arm bands can be inverted to control the arm side directly (kid left arm to robot left arm) or in a mirror fashion (kid left arm to robot right arm) which seemed a better initial setup. Of course the sensors could also be used on other body parts to match specific rehabilitation needs (e.g., it could be placed on a foot to exercise an ankle while controlling the robot.)

Our original prototype is very rudimentary. More controls need to be added and more flexibility is needed in matching sensors and robot movements before we can successfully work with children with disabilities. Wireless connections from the computer to the sensors are needed, the size of the robot needs to be reduced to facilitate a robot’s motion on a table or floor, but we can start develop scenarios of use. Below are 2 examples and we are now working with professional rehabilitation experts to develop other strategies and games.

Max has cerebral palsy. To learn to feed himself he needs to do a lot of supination and pronation exercises to strengthen his arms. Today’s standard of care makes him repeat the same exercise 100 times. Five-year old Max finds the exercises extremely boring and often refuses to finish them. But today the therapist tells him "Do you have a pet? Let's tell a story about it together... and the robot will act out the story for you." As the child start talking about the story the therapist equips the child with an elbow pad embedded with input devices and shows the gestures that will make the robot act "happy" of "confused" when appropriate in the story. For example, arm extensions could be used to make the robot move around the space, and arm rotation to turn it around. Max is now engaged in a creative endeavor and is motivated to do the exercises. After much practice and many revisions the story is recorded on video to take home. Next time Max will learn to teach the robots new tricks, associated with new movements; eventually he could control the robot remotely from home and tell the story to the children in the waiting room.

Sue is greeted by his therapist, "Sue, are you interested in telling a story today?" Together they pull out the robot toolkit. The therapist asks, "What character do you want to tell a story with today?" And Sue answers, "Benny," as she pulls out the bird wings, purple horns, and flying saucer. Slowly Sue builds her stuffed robot caracter. Then she moves to the computer and begins to write a story, "My name is Benny and I am SAD." She presses the "tell me a story" button and the robot tells the story and acts sad. The therapist asks Sue, "You know, I'm not sure the robot looks really sad. Can you teach the robot to really look sad?" Then she takes out their robot sensors and they attach them to Sue’s arm and leg. Sue then moves toward the wall. The Therapist asks, "Is that the only way to look sad?" Together they talk about what it means to be sad and try different movements and expressions. Late in the session, the therapist asks, "Sue, can we work on making Benny look happy? I think I'm tired of sad." Sue then goes to the computer and writes a story about Benny, "My name is Benny and I am HAPPY. Benny likes his therapist."

We anticipate producing a robot toolkit that includes a base robot, accessories for customizing the robot (paws, feathers, sound), a rich set of input devices to control the robot adapted to the range of abilities of children in rehabilitation. Software will also be needed to monitor usage. After the usual cycle of formative evaluation and iterative design, we will develop a methodology for rehabilitation staff to provide therapeutic play, including training, examples of use, preparation, and storytelling techniques. Measuring the effectiveness of such therapeutic play with thorough scientific evaluations is our main challenge for the coming year.

We have built a working prototype, scenarios have been developed to illustrate how the prototype could be used and we are ready for testing with challenged children. We believe that our innovative design methodology involving participatory design and children as design partners will lead to a product that will truly excite children's interest as well as support the rehabilitation experience.

We want to thank the Maryland Industrial Partnership program for partial support for this research.

Caputo, R. A. (1993). Using puppets with students with emotional and behavioral disorders. Intervention in School and Clinic, 29 (1), pp. 26-30.

Carter, S. (1987). Use of puppets to treat traumatic grief: A case study. Elementary School Guidance and Counseling, 21(3), pp. 210-215.

Cochran, J. L. (1996). Using play and art therapy to help culturally diverse students overcome barriers to school success. School Counselor, 43, pp. 287-298.

Currant, N. (1985). The expansive educational value of puppets. Academic Therapy, 21(1), pp. 55-60.

Druin, A. (ed.) (1999b). The design of children’s technology. San Francisco, CA: Morgan Kaufmann

Druin, A., Montemayor, J., Hendler, J., McAlister, B., Boltman, A., Fiterman, E., Plaisant, A., Kruskal, A., Olsen, H., Revett, I., Schwenn, T.P., Sumida, L. and Wagner, R., (1999). Designing PETS: A Personal Electronic Teller of Stories Proceedings of Conference on Human Factors in Computing Systems (CHI'99), Pittsburgh, PA, USA, May 15-20, ACM, New York, 326-329

Griffiths, M (1997). Video games and clinical practice: issues, uses and treatments. Br J Clin Psychol Nov;36, 639-41

Herman-Hilker, S L; Hilker, D F; Levine, S P. (1995). Achievement of physical therapy goals as a result of assistive technology use. RESNA, pp. 59-61

Rinaldi, M. C. (2000). Using Adaptive Video Game Technology as Therapeutic Tool, Master of Biomedical Engineering Thesis, The Catholic University of America School of Engineering.

Vidler, V. (1972). Use of puppets to reach the emotionally disturbed. Instructor, 81(9).

White, J. & Allers, C. T. (1994). Play therapy with abused children: A review of the literature. Journal of Counseling and Development, 72, pp. 390-394.