EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting

Changjing Liu*, Yiming Huang*, Long Bai, Beilei Cui, Hongliang Ren**
The Chinese University of Hong Kong
early accepted by MICCAI 2026
*Indicates Equal Contribution
**Indicates corresponding author
Paper Supplementary Code arXiv

Description

Abstract

In robot-assisted minimally invasive surgery, high-fidelity dynamic endoscopic scene reconstruction and simulation are crucial to enhancing downstream tasks and advancing surgical outcomes. However, existing methods primarily focus on visual reconstruction, lacking physics-based descriptions of the scene required for realistic simulation. We propose a unified framework that achieves physics-aware reconstruction and physical simulation of endoscopic scenes through Multi-modal Large Language Models (MLLMs)-guided Gaussian Splatting. Our approach utilizes 4D Gaussian Splatting (4DGS) integrated with pre-trained segmentation and depth estimation to represent deformable tissues and tools. To achieve automatic inference of physical properties, we introduce an object-wise material field that initializes material parameters via MLLM and refines them through a differentiable Material Point Method (MPM) under joint supervision from rendered images and optical flow.

Pipeline

Description

BibTeX

@misc{liu2026endogsimphysicsaware4ddynamic,
        title={EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting}, 
        author={Changjing Liu and Yiming Huang and Long Bai and Beilei Cui and Hongliang Ren},
        year={2026},
        eprint={2605.16022},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2605.16022}, 
  }