Polisocial Krog

In recent years, we have witnessed a rapid growth of interactive applications to support interventions for children with intellectual developmental disorder (IDD). Our research explores novel interactive learning technologies for this target group. We adopt a game-based approach and blend two interaction paradigms: full-body interaction and interaction with mobile robots. Full-body interaction is a general term for a wide set of technologies that enable the use of body as interaction device to control multimedia contents or digitally enriched physical artifacts. In our research, full-body interaction refers to the capability of interacting with virtual worlds on large screens using body movements or mid-air gestures at the distance. Mobile robots are motion-enabled objects instrumented with sensors and actuators. Sensors detect user’s actions (e.g., speech, physical manipulation, movements), while actuators provide multisensory stimuli, e.g., light effects, vibrations, or movements.

An exploratory study has been performed at a local therapeutic center to test the effectiveness of our design solutions. The study involved 22 low and medium functioning IDD children and their 11 therapists. We are currently analyzing the results and we can only anticipate a general consideration. All therapists agree that the integration of robotic interaction and full-body interaction with large screen opens new extraordinary opportunities in the IDD context. It elicits operational behaviors, social interaction and emotional responses that normally do not occur using other methods, or that require a much longer time to be achieved. For example, an autistic child explicitly called a mate to play with Teo and the screen together: it was the first time he expressed the willingness to play in social mode. A girl with severe Hyperactivity Disorder relaxed in few minutes after meeting Teo, and she was able to concentrate on, and perform, a learning task (the “Colors” game) during the same session.

Our future work will revise our current prototypes in light of an accurate analysis of the results of this preliminary study, and will test more systematically the benefits of our approach in 2 additional therapeutic centers involved in the KROG project.

Hide & Seek

Hide-and-seek is a children’s game in which a number of players conceal themselves in the environment, to be found by one or more seekers. After reaching the counted number, the screen (Kinect) attempts to locate all concealed players.

Traffic Light

The robot calls and shows out “Green light!” or “Red Light! The screen shows a car moving when is red and zebra crossing when is green. As soon as it is green, the player walks towards it otherwise he must freeze in place.


The screen acts out as the “Curator”. Everyone else (robot and child) playing stands at the far end (distance depends upon playing area selected). The object of the game is for a “Statue” to tag the Curator, thereby resetting the game.

Witch Says Color

The virtual avatar goes on a colored circle and says the color name.There are two variants: a) the robot move to the circle and the child imitates it; b) the child and the robot run to the physical circle on the floor.


The screen shows a colored figure (fruit, animal…). The robot gives the child some buttons to click on. The child has to choose from a restricted gamut of choices. The game proceeds giving therapists the possibility to select what figure will be shown.


The child sees hisself inside the screen as a story character as well as the robot. They, together, cooperating or non, proceed into the story and listen to the instructions (e.g. say: “Hello”, wave your hand…).





Politecnico di Milano

Funded by