Skip to main content

Research Repository

Advanced Search

An RGB-D based social behavior interpretation system for a humanoid social robot

Zaraki, Abolfazl; Giuliani, Manuel; Dehkordi, Maryam Banitalebi; Mazzei, Daniele; D'ursi, Annamaria; Rossi, Danilo De


Abolfazl Zaraki

Manuel Giuliani
Co- Director Bristol Robotics Laboratory

Maryam Banitalebi Dehkordi

Daniele Mazzei

Annamaria D'ursi

Danilo De Rossi


We used a new method called “Ghost-in-the-Machine” (GiM) to investigate social interactions with a robotic bartender taking orders for drinks and serving them. Using the GiM paradigm allowed us to identify how human participants recognize the intentions of customers on the basis of the output of the robotic recognizers. Specifically, we measured which recognizer modalities (e.g., speech, the distance to the bar) were relevant at different stages of the interaction. This provided insights into human social behavior necessary for the development of socially competent robots. When initiating the drink-order interaction, the most important recognizers were those based on computer vision. When drink orders were being placed, however, the most important information source was the speech recognition. Interestingly, the participants used only a subset of the available information, focussing only on a few relevant recognizers while ignoring others. This reduced the risk of acting on erroneous sensor data and enabled them to complete service interactions more swiftly than a robot using all available sensor data. We also investigated socially appropriate response strategies. In their responses, the participants preferred to use the same modality as the customer’s requests, e.g., they tended to respond verbally to verbal requests. Also, they added redundancy to their responses, for instance by using echo questions. We argue that incorporating the social strategies discovered with the GiM paradigm in multimodal grammars of human–robot interactions improves the robustness and the ease-of-use of these interactions, and therefore provides a smoother user experience.


Zaraki, A., Giuliani, M., Dehkordi, M. B., Mazzei, D., D'ursi, A., & Rossi, D. D. (2014, October). An RGB-D based social behavior interpretation system for a humanoid social robot. Paper presented at 2nd RSI International Conference on Robotics and Mechatronics (ICRoM 2014), Tehran, Iran

Presentation Conference Type Conference Paper (unpublished)
Conference Name 2nd RSI International Conference on Robotics and Mechatronics (ICRoM 2014)
Conference Location Tehran, Iran
Start Date Oct 15, 2014
End Date Oct 17, 2014
Acceptance Date Oct 15, 2014
Publication Date Dec 18, 2014
Journal IEEE Transactions on Human-Machine Systems
Peer Reviewed Peer Reviewed
Volume 2
Pages 157-168
Series Title Lecture Notes in Computer Science
Series Number 44
Keywords active vision, context-dependent social gaze behavior, human-robot interaction, scene analysis, social attention
Public URL
Publisher URL
Additional Information Additional Information : Best Paper Award, Best Presentation Award
Title of Conference or Conference Proceedings : 2014 Second RSI/ISM International Conference on Robotics and Mechatronics (ICRoM)