A vision-guided deep learning framework for dexterous robotic grasping using Gaussian processes and transformers

Kadalagere Sampath, Suhas; Wang, Ning; Yang, Chenguang; Wu, Howard; Liu, Cunjia; Pearson, Martin

Abstract

Robotic manipulation of objects with diverse shapes, sizes, and properties, especially deformable ones, remains a significant challenge in automation, necessitating human-like dexterity through the integration of perception, learning, and control. This study enhances a previous framework combining YOLOv8 for object detection and LSTM networks for adaptive grasping by introducing Gaussian Processes (GPs) for robust grasp predictions and Transformer models for efficient multi-modal sensory data integration. A Random Forest classifier also selects optimal grasp configurations based on object-specific features like geometry and stability. The proposed grasping framework achieved a 95.6% grasp success rate using Transformer-based force modulation, surpassing LSTM (91.3%) and GP (91.3%) models. Evaluation of a diverse dataset showed significant improvements in grasp force modulation, adaptability, and robustness for two- and three-finger grasps. However, limitations were observed in five-finger grasps for certain objects, and some classification failures occurred in the vision system. Overall, this combination of vision-based detection and advanced learning techniques offers a scalable solution for flexible robotic manipulation.

• Journal Article Type: Article
• Acceptance Date: Feb 24, 2025
• Online Publication Date: Feb 28, 2025
• Publication Date: Feb 28, 2025
• Deposit Date: Feb 28, 2025
• Publicly Available Date: Mar 4, 2025
• Journal: Applied Sciences
• Electronic ISSN: 2076-3417
• Publisher: MDPI
• Peer Reviewed: Peer Reviewed
• Volume: 15
• Issue: 5
• Article Number: 2615
• DOI: https://doi.org/10.3390/app15052615
• Keywords: dexterous robotic grasping; adaptive grasping; deep learning in robotics; transformer networks; Gaussian processes; vision-based force modulation
• Public URL: https://uwe-repository.worktribe.com/output/13826594