Development of a gamified Simon task for machine learning and eye-tracking analysis of cognitive performance in early dementia diagnosis

Wang, Xuli; Field, Sofia; Conway, Myra; Smith, Melvyn; Zhang, Wenhao; Zook, Nancy

Abstract

Introduction

Technologies leveraging machine learning are enabling the creation of efficient methods for the early detection of age-related neurodegeneration outside of a lab environment. Combining cognitive assessments with eye movement analysis has the potential to sensitively differentiate typical ageing and early stages of dementia. This study aimed to develop and evaluate a gamified Simon task, measuring executive control of automatic/congruent and controlled/incongruent responses. Machine learning was developed to detect subtle congruity-related differences in gameplay and eye movement data.

Method

Forty participants (Mage=20.62, SD=2.07) completed seventy-two trials (50% spatially congruent; 50% low/high speed) in both low and high cognitive loads whilst eye movements were recorded using an off-the-shelf camera. The game, played on a digital tablet, involved sorting fruits into baskets as moving images ‘ripened’ from green to red or yellow using spatially congruent and incongruent features. Reaction time (RT) differences by congruence was assessed. A time-series transformer model was developed to automatically recognise the congruity of each three-second trial purely based on eye movements.

Results

In the low-load conditions, RT differences were not significant for congruity; however, in the high-load, fast condition, the incongruent RTs were significantly higher than congruent, indicating a ‘Simon Effect’ (F = 9.11, p<.01). Utilising eye movement data, with an 80:20 training-to-testing split, the time-series transformer model achieved a classification accuracy of 85.1% for recognising the two congruity levels.

Conclusions

This study validated the presence of the ‘Simon Effect’ in a gamified task. We show, for the first time, that machine learning can detect congruity from just three seconds of eye movements. Future work will continue to co-design the game with older adults and focus on subtle cognitive and eye movement variability using neuropsychological assessments alongside the game to inform machine learning for early, pre-clinical identification of atypical changes associated with dementia.  

• Presentation Conference Type: Poster
• Conference Name: Alzheimer's Research UK Research Conference 2025
• Start Date: Feb 25, 2025
• End Date: Feb 26, 2025
• Acceptance Date: Nov 29, 2024
• Deposit Date: Jun 16, 2025
• Peer Reviewed: Peer Reviewed
• Public URL: https://uwe-repository.worktribe.com/output/14565002
• Additional Information: Authors:; Xuli Wang1, Sofia Field2, Myra Conway3, Melvyn Smith 1, Wenhao Zhang 1, Nancy Zook2 ; ; 1Centre for Machine Vision, Bristol Robotics Laboratory, UWE Bristol ; ; 2 Psychological Sciences Research Group, UWE Bristol ; ; 3 College of Health, Psychology and Social Care, University of Derby