Comparing objective and subjective measures of usability in a human-robot dialogue system

Foster, Mary Ellen; Giuliani, Manuel; Knoll, Alois

Abstract

We present a human-robot dialogue system that enables a robot to work together with a human user to build wooden construction toys. We then describe a study in which na¨ıve subjects interacted with this system under a range of conditions and then completed a user-satisfaction questionnaire.The results of this study provide a wide range of subjective and objective measures of the quality of the interactions. To assess which aspects of the interaction had the greatest impact on the users’ opinions of the system, we used a method based on the PARADISE evaluation framework (Walker et al., 1997) to derive a performance function from our data. The major contributors to user satisfaction were the number of repetition requests (which had a negative effect on satisfaction), the dialogue length, and the users’ recall of the system instructions (both of which contributed positively).

• Presentation Conference Type: Conference Paper (unpublished)
• Conference Name: Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing (ACL-IJCNLP 2009)
• Start Date: Aug 2, 2009
• End Date: Aug 7, 2009
• Acceptance Date: Aug 2, 2009
• Publication Date: Aug 2, 2009
• Peer Reviewed: Peer Reviewed
• Keywords: comparing, objective, subjective, measures, usability, human-robot, dialogue, system
• Public URL: https://uwe-repository.worktribe.com/output/993811
• Publisher URL: http://dl.acm.org/citation.cfm?id=1690270
• Additional Information: Title of Conference or Conference Proceedings : Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP