Real-time style transfer for videos to enhance the realism of simulation of laparoscopic surgeries

Shao, Marine; Clark, James; Huson, David; Hardeberg, Jon

Abstract

Surgical simulation has repeatedly proven its potential, but is limited by the lack of realism to the surgical experience. For laparoscopic surgery simulation, an image processing method known as style transfer can improve the realism by transferring the style of a picture from surgery onto the video of the simulator using a stylisation network. In this article, we propose an adjustable style transfer algorithm for videos to improve the realism of silicone based models. The results show that our method can successfully implement multiple stylisations in real-time onto the video of the simulator while maintaining temporal consistency and overall smoothness of the video. Comparing to other style-transfer methods, our technique can offer multiple stylisations and, except from the method of Johnson et al. [6], achieves better realness score when evaluated by surgeons. This method can also be trained on a generic database in only 3h43 contrary to the other image processing techniques such as image translation which require specific training datasets.

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 2022 10th European Workshop on Visual Information Processing (EUVIP)
• Start Date: Sep 11, 2022
• End Date: Sep 14, 2022
• Acceptance Date: Jul 11, 2022
• Publication Date: Oct 20, 2022
• Deposit Date: Dec 1, 2022
• Publicly Available Date: Oct 21, 2024
• Publisher: Institute of Electrical and Electronics Engineers (IEEE)
• Volume: 2022-September
• Series ISSN: 2471-8963
• Book Title: 2022 10th European Workshop on Visual Information Processing (EUVIP)
• DOI: https://doi.org/10.1109/EUVIP53989.2022.9922706
• Keywords: Surgical simulation, Laparoscopy, Style transfer, Virtual Reality, Training, Visualization , Minimally invasive surgery, Databases, Image processing, Instrument, Europe
• Public URL: https://uwe-repository.worktribe.com/output/10148493
• Publisher URL: https://ieeexplore.ieee.org/document/9922706
• Related Public URLs: https://ieeexplore.ieee.org/xpl/conhome/9921420/proceeding; ; https://euvip2022.org/