Interactive Surgical Robot Control Using Mixed Reality
Part Of
2025 International Automatic Control Conference, CACS 2025
ISBN (of the container)
9798331596408
ISBN
9798331596408
Date Issued
2025
Author(s)
Abstract
This paper proposes an MR-based interactive control framework for surgical robotics, specifically designed for ophthalmic scenarios. Leveraging Microsoft HoloLens 2 and MRTK3, the system enables users to intuitively operate a mobile robotic platform and a 6-degree-of-freedom robotic arm via natural hand gestures, integrating forward/inverse kinematics, trajectory planning, and Unity-ROS real-time communication. Users can define 3D target points and designate movement paths within the MR environment using customized gestures. The system then calculates joint angles through inverse kinematics and transmits the commands to the physical robot via a communication module. By incorporating spatial mesh generation and QR code-based CAD model registration, precise alignment between virtual and physical components is achieved. Interactive gesture control is also implemented to facilitate re-al-time path designation and execution. The system sup-ports both manual and autonomous control modes, with A∗ path planning and Damped Least Squares applied for navigation and end-effector positioning. Additional modules, including autonomous path planning, CAD model visualization, spatial understanding, and UI interactions, form a comprehensive and immersive surgical simulation and training platform.
Event(s)
2025 International Automatic Control Conference, CACS 2025, 5 November 2025 - 8 November 2025, Hsinchu
Publisher
Institute of Electrical and Electronics Engineers Inc.
Type
conference paper