A multimodal teleoperation interface for human-robot collaboration

Si, Weiyong; Zhong, Tianjian; Wang, Ning; Yang, Chenguang

Abstract

Human-robot collaboration provides an effective approach to combine human intelligence and the autonomy of robots, which can improve the safety and efficiency of the robot. However, developing an intuitive and immersive human-robot interface with multimodal feedback for human-robot interaction and collaboration is still challenging. In this paper, we developed a multimodal-based human-robot interface to involve humans in the loop. The Unity-based virtual reality (VR) environment, including the virtual robot manipulator and its working environment, was developed to simulate the real working environment of robots. We integrated the digital twin mechanism with the VR environment development, which provides a corresponding model with the physical task. The virtual environment could visualize the visual and haptic feedback through the multimodal sensors in the robot, which provides an immersive and friendly teleoperating environment for human operators. We conduct user study experiments based on NASA Task Load Index, through a physical contact scanning task. The result shows that the proposed multimodal interface improved by 31.8% in terms of the cognitive and physical workload, comparing with the commercial teleportation device Touch X.

• Presentation Conference Type: Conference Paper (published)
• Conference Name: Proceedings - 2023 IEEE International Conference on Mechatronics, ICM 2023
• Start Date: Mar 15, 2023
• End Date: Mar 17, 2023
• Acceptance Date: Jan 12, 2023
• Online Publication Date: Mar 15, 2023
• Publication Date: Mar 15, 2023
• Deposit Date: May 9, 2023
• Publicly Available Date: Mar 16, 2025
• Publisher: Institute of Electrical and Electronics Engineers (IEEE)
• Book Title: 2023 IEEE International Conference on Mechatronics (ICM)
• ISBN: 978-1-6654-6662-2
• DOI: https://doi.org/10.1109/ICM54990.2023.10102060
• Keywords: Immersive teleoperation, Human-in-the-loop, Human-robot interface, Visualization, Solid modeling, Multimodal sensors, NASA, Collaboration, Virtual environments, Teleportation
• Public URL: https://uwe-repository.worktribe.com/output/10723700
• Publisher URL: https://ieeexplore.ieee.org/document/10102060