Head and shoulders: Automatic error detection in human-robot interaction

Trung, Pauline; Giuliani, Manuel; Miksch, Michael; Stollnberger, Gerald; Stadler, Susanne; Mirnig, Nicole; Tscheligi, Manfred

Abstract

We describe a novel method for automatic detection of errors in human-robot interactions. Our approach is to detect errors based on the classification of head and shoulder movements of humans who are interacting with erroneous robots. We conducted a user study in which participants interacted with a robot that we programmed to make two types of errors: social norm violations and technical failures. During the interaction, we recorded the behavior of the participants with a Kinect v1 RGB-D camera. Overall, we recorded a data corpus of 237,998 frames at 25 frames per second; 83.48% frames showed no error situation; 16.52% showed an error situation. Furthermore, we computed six different feature sets to represent the movements of the participants and temporal aspects of their movements. Using this data we trained a rule learner, a Naive Bayes classifier, and a k-nearest neighbor classifier and evaluated the classifiers with 10-fold cross validation and leave-one-out cross validation. The results of this evaluation suggest the following: (1) The detection of an error situation works well, when the robot has seen the human before; (2) Rule learner and k-nearest neighbor classifiers work well for automated error detection when the robot is interacting with a known human; (3) For unknown humans, the Naive Bayes classifier performed the best; (4) The classification of social norm violations does perform the worst; (5) There was no big performance difference between using the original data and normalized feature sets that represent the relative position of the participants.

Citation

Trung, P., Giuliani, M., Miksch, M., Stollnberger, G., Stadler, S., Mirnig, N., & Tscheligi, M. (2017, November). Head and shoulders: Automatic error detection in human-robot interaction. Paper presented at 19th ACM International Conference on Multimodal Interaction

• Presentation Conference Type: Conference Paper (unpublished)
• Conference Name: 19th ACM International Conference on Multimodal Interaction
• Start Date: Nov 13, 2017
• End Date: Nov 17, 2017
• Acceptance Date: Nov 14, 2017
• Publication Date: Nov 14, 2017
• Journal: Proceedings of International Conference on Multimodal Interaction
• Peer Reviewed: Peer Reviewed
• ISBN: 9781450355438
• Keywords: human-robot interaction, error detection, error situation, human activity recognition, RBG-D camera, faulty robot
• Related Public URLs: https://icmi.acm.org/2017/
• Additional Information: Title of Conference or Conference Proceedings : 19th ACM International Conference on Multimodal Interaction