Are we capturing individual differences? Evaluating the test–retest reliability of experimental tasks used to measure social cognitive abilities

Pennington, Charlotte R.; Birch-Hurst, Kayley; Ploszajski, Matthew; Clark, Kait; Hedge, Craig; Shaw, Daniel J.

Abstract

Social cognitive skills are crucial for positive interpersonal relationships, health, and wellbeing and encompass both automatic and reflexive processes. To assess this myriad of skills, researchers have developed numerous experimental tasks that measure automatic imitation, emotion recognition, empathy, perspective taking, and intergroup bias and have used these to reveal important individual differences in social cognition. However, the very reason these tasks produce robust experimental effects – low between-participant variability – can make their use as correlational tools problematic. We performed an evaluation of test–retest reliability for common experimental tasks that measure social cognition. One-hundred and fifty participants completed the race-Implicit Association Test (r-IAT), Stimulus–Response Compatibility (SRC) task, Emotional Go/No-Go (eGNG) task, Dot Perspective-Taking (DPT) task, and State Affective Empathy (SAE) task, as well as the Interpersonal Reactivity Index (IRI) and indices of Explicit Bias (EB) across two sessions within 3 weeks. Estimates of test–retest reliability varied considerably between tasks and their indices: the eGNG task had good reliability (ICC = 0.63–0.69); the SAE task had moderate-to-good reliability (ICC = 0.56–0.77); the r-IAT had moderate reliability (ICC = 0.49); the DPT task had poor-to-good reliability (ICC = 0.24–0.60); and the SRC task had poor reliability (ICC = 0.09–0.29). The IRI had good-to-excellent reliability (ICC = 0.76–0.83) and EB had good reliability (ICC = 0.70–0.77). Experimental tasks of social cognition are used routinely to assess individual differences, but their suitability for this is rarely evaluated. Researchers investigating individual differences must assess the test–retest reliability of their measures.

• Journal Article Type: Article
• Acceptance Date: Jan 13, 2025
• Online Publication Date: Jan 31, 2025
• Publication Date: Jan 31, 2025
• Deposit Date: Feb 4, 2025
• Publicly Available Date: Feb 4, 2025
• Journal: Behavior Research Methods
• Print ISSN: 1554-351X
• Electronic ISSN: 1554-3528
• Publisher: Springer (part of Springer Nature)
• Peer Reviewed: Peer Reviewed
• Volume: 57
• Issue: 2
• Article Number: 82
• DOI: https://doi.org/10.3758/s13428-025-02606-5
• Public URL: https://uwe-repository.worktribe.com/output/13714879
• Publisher URL: https://link.springer.com/article/10.3758/s13428-025-02606-5
• Additional Information: Accepted: 13 January 2025; First Online: 31 January 2025; : ; : This study received ethical approval from the Institutional Review Board at the University of the West of England (REF: HAS.18.07.21).; : All participants provided written informed consent to participate.; : All participants provided written informed consent for their data to be reported in aggregate in scientific publications and conference proceedings.; : None.