Skip to main content

Research Repository

Advanced Search

Head and shoulders: Automatic error detection in human-robot interaction

Trung, Pauline; Giuliani, Manuel; Miksch, Michael; Stollnberger, Gerald; Stadler, Susanne; Mirnig, Nicole; Tscheligi, Manfred


Pauline Trung

Michael Miksch

Gerald Stollnberger

Susanne Stadler

Nicole Mirnig

Manfred Tscheligi


We describe a novel method for automatic detection of errors in human-robot interactions. Our approach is to detect errors based on the classification of head and shoulder movements of humans who are interacting with erroneous robots. We conducted a user study in which participants interacted with a robot that we programmed to make two types of errors: social norm violations and technical failures. During the interaction, we recorded the behavior of the participants with a Kinect v1 RGB-D camera. Overall, we recorded a data corpus of 237,998 frames at 25 frames per second; 83.48% frames showed no error situation; 16.52% showed an error situation. Furthermore, we computed six different feature sets to represent the movements of the participants and temporal aspects of their movements. Using this data we trained a rule learner, a Naive Bayes classifier, and a k-nearest neighbor classifier and evaluated the classifiers with 10-fold cross validation and leave-one-out cross validation. The results of this evaluation suggest the following: (1) The detection of an error situation works well, when the robot has seen the human before; (2) Rule learner and k-nearest neighbor classifiers work well for automated error detection when the robot is interacting with a known human; (3) For unknown humans, the Naive Bayes classifier performed the best; (4) The classification of social norm violations does perform the worst; (5) There was no big performance difference between using the original data and normalized feature sets that represent the relative position of the participants.


Trung, P., Giuliani, M., Miksch, M., Stollnberger, G., Stadler, S., Mirnig, N., & Tscheligi, M. (2017, November). Head and shoulders: Automatic error detection in human-robot interaction. Paper presented at 19th ACM International Conference on Multimodal Interaction

Presentation Conference Type Conference Paper (unpublished)
Conference Name 19th ACM International Conference on Multimodal Interaction
Start Date Nov 13, 2017
End Date Nov 17, 2017
Acceptance Date Nov 14, 2017
Publication Date Nov 14, 2017
Journal Proceedings of International Conference on Multimodal Interaction
Peer Reviewed Peer Reviewed
ISBN 9781450355438
Keywords human-robot interaction, error detection, error situation, human activity recognition, RBG-D camera, faulty robot
Related Public URLs
Additional Information Title of Conference or Conference Proceedings : 19th ACM International Conference on Multimodal Interaction