MICCAI 2026 MICCAI 2026 (early acceptance)

EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting

Changjing Liu*, Yiming Huang*, Long Bai, Beilei Cui, Hongliang Ren
*Equal Contribution  |  Corresponding author
The Chinese University of Hong Kong
The Chinese University of Hong Kong
Paper Code arXiv Demo
TL;DR From surgical video to physics-based simulation.
  • Unified physics-aware reconstruction and simulation from endoscopic video.
  • Object-wise material field with MLLM initialization and MPM-based refinement.
  • State-of-the-art physical accuracy and visual realism on public and in-house data.
EndoGSim teaser

Abstract

In robot-assisted minimally invasive surgery, high-fidelity dynamic endoscopic scene reconstruction and simulation are crucial to enhancing downstream tasks and advancing surgical outcomes. However, existing methods primarily focus on visual reconstruction, lacking physics-based descriptions of the scene required for realistic simulation. We propose a unified framework that achieves physics-aware reconstruction and physical simulation of endoscopic scenes through Multi-modal Large Language Models (MLLMs)-guided Gaussian Splatting. Our approach utilizes 4D Gaussian Splatting (4DGS) integrated with pre-trained segmentation and depth estimation to represent deformable tissues and tools. To achieve automatic inference of physical properties, we introduce an object-wise material field that initializes material parameters via MLLM and refines them through a differentiable Material Point Method (MPM) under joint supervision from rendered images and optical flow.

Interactive Demo

Drag on the image to apply force. First load may take ~30s on ZeroGPU. Open full demo

Comparison with Prior Work

Table 1: Comparison of methods by video input, auto initialization, object-wise material field, and physical simulation.

PhysGen Physics3D PhysGaussian GIC PhysFlow Ours
Video Input ✗ ✗ ✗ ✓ ✓ ✓
Auto Initialization ✓ ✗ ✗ ✗ ✓ ✓
Material Field ✗ ✗ ✗ ✗ ✗ ✓
Physical Simulation ✗ ✓ ✓ ✓ ✓ ✓

Pipeline

EndoGSim pipeline overview

Results

EndoGSim qualitative comparison across surgical scenes
Qualitative comparison on EndoNeRF and PorcineEndo datasets.
EndoGSim optical flow comparison
Rendered images and optical flow errors compared with baselines.

BibTeX

@article{liu2026endogsim,
        title={EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting},
        author={Liu, Changjing and Huang, Yiming and Bai, Long and Cui, Beilei and Ren, Hongliang},
        year={2026},
        eprint={2605.16022},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2605.16022}, 
  }