Inter-annotator agreement using the Conversation Analysis Modelling Schema, for dialogue

Duran, Nathan; Battle, Steve; Smith, Jim

Abstract

We present the Conversation Analysis Modeling Schema (CAMS), a novel dialogue labeling schema that combines the Conversation Analysis concept of Adjacency Pairs, with Dialogue Acts. The aim is to capture both the semantic and syntactic structure of dialogue, in a format that is independent of the domain or topic, and which facilitates the computational modeling of dialogue. A labeling task undertaken by novice annotators is used to evaluate its efficacy on a selection of task-oriented and non-task-oriented dialogs, and to measure inter-annotator agreement. To deepen the “human-factors” analysis we also record and examine users’ self-reported confidence scores and average utterance annotation times. Inter-annotator agreement is shown to be higher for task-oriented dialogs than non-task-oriented, though the structure of the dialogue itself has a more significant impact. We further examine the assumptions around expected agreement for two weighted agreement coefficients, Alpha and Beta, and show that annotators assign labels using similar probability distributions, small variations can result in large differences in agreement values between biased and unbiased measures.

• Journal Article Type: Article
• Acceptance Date: Dec 15, 2021
• Online Publication Date: Jan 17, 2022
• Publication Date: 2022-10
• Deposit Date: Mar 2, 2022
• Publicly Available Date: Mar 8, 2022
• Journal: Communication Methods and Measures
• Print ISSN: 1931-2458
• Electronic ISSN: 1931-2466
• Publisher: Taylor & Francis (Routledge)
• Peer Reviewed: Peer Reviewed
• Volume: 16
• Issue: 3
• Pages: 182-214
• DOI: https://doi.org/10.1080/19312458.2021.2020229
• Keywords: Conversation Analysis, Inter-annotator Agreement, Annotation Schema, Dialogue
• Public URL: https://uwe-repository.worktribe.com/output/8584971
• Additional Information: Peer Review Statement: The publishing and review policy for this title is described in its Aims & Scope.; Aim & Scope: http://www.tandfonline.com/action/journalInformation?show=aimsScope&journalCode=hcms20; Published: 2022-01-17