A modified LSTM model for Chinese sign language recognition using leap motion

Wu, Bixiao; Lu, Zhenyu; Yang, Chenguang

Abstract

At present, there are about 70 million deaf people using sign language in the world, but for most normal people, it is difficult to understand the meaning of the sign language expression. Therefore, it is of great importance to explore the ways of recognising the sign language. In this paper, we propose a dynamic sign language recognition method based on the modified long short-term memory (LSTM) model. Firstly, we use Leap Motion to collect the features of Chinese Sign Language (CSL). LSTM has a good effect in processing time series data, but the parameters of its hidden layer are shared, making it important information lost when dealing with long time series. The attention mechanism can give different attention weights to different features according to the correlation between the input data and output data, so as to enhance the model's attention to key information. Therefore, we combine LSTM with attention mechanism for dynamic sign language recognition. Experimental results show that the recognition accuracy of the modified LSTM model is 99.55%, which is higher than that of LSTM model. Finally, we developed a sign language human-computer interaction system, which verifies the real-time performance and effectiveness of the method proposed in this paper.

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
• Start Date: Oct 9, 2022
• End Date: Oct 12, 2022
• Acceptance Date: Nov 9, 2022
• Publication Date: Nov 18, 2022
• Deposit Date: Dec 2, 2022
• Publicly Available Date: Nov 19, 2024
• Publisher: Institute of Electrical and Electronics Engineers (IEEE)
• Volume: 2022-October
• Pages: 1612-1617
• Series ISSN: 2577-1655
• Book Title: 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
• DOI: https://doi.org/10.1109/SMC53654.2022.9945287
• Keywords: Human computer interaction, Dynamics, Time series analysis, Neural networks, Gesture recognition, Assistive technologies, Real-time systems, Sign language recognition, LSTM, Attention mechanism, Leap Motion
• Public URL: https://uwe-repository.worktribe.com/output/10198165
• Publisher URL: https://ieeexplore.ieee.org/document/9945287
• Related Public URLs: https://ieeexplore.ieee.org/xpl/conhome/9945068/proceeding