Spatial information guided Convolution for Real-Time RGBD Semantic Segmentation

Lin-Zhuo Chen, Zheng Lin, Ziqin Wang, Yong-Liang Yang and Ming-Ming Cheng
⭐ Project Home »
[PDF ] [Code ]


3D spatial information is known to be beneficial to the semantic segmentation task. Most existing methods take 3D spatial data as an additional input, leading to a two-stream segmentation network that processes RGB and 3D spatial information separately. This solution greatly increases the inference time and severely limits its scope for real-time applications. To solve this problem, we propose Spatial information guided Convolution (S-Conv), which allows efficient RGB feature and 3D spatial information integration. S-Conv is competent to infer the sampling offset of the convolution kernel guided by the 3D spatial information, helping the convolutional layer adjust the receptive field and adapt to geometric transformations. S-Conv also incorporates geometric information into the feature learning process by generating spatially adaptive convolutional weights. The capability of perceiving geometry is largely enhanced without much affecting the amount of parameters and computational cost. We further embed S-Conv into a semantic segmentation network, called Spatial information Guided convolutional Network (SGNet), resulting in real-time inference and state-of-the-art performance on NYUDv2 and SUNRGBD datasets.

Architecture of S-Conv

Architecture of S-Conv


Following results tested under NVIDIA 1080TI GPU, Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz:

Results on NYUDv2 Dataset




Results on SUNRGBD Dataset


Some follow-up work

Global-Local Propagation Network for RGB-D Semantic Segmentation. Chen Sihan, Xinxin Zhu, Wei Liu, Xingjian He, and Jing Liu. arXiv preprint arXiv:2101.10801 (2021).

Modality-Guided Subnetwork for Salient Object Detection. Wu, Zongwei, Guillaume Allibert, Christophe Stolz, Chao Ma, and Cédric Demonceaux. arXiv preprint arXiv:2110.04904 (2021).


Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation. Lin-Zhuo Chen, Zheng Lin, Ziqin Wang, Yong-Liang Yang, Ming-Ming Cheng*. IEEE TIP, 2021.

If you find our work is helpful, please cite

  author={Lin-Zhuo Chen and Zheng Lin and Ziqin Wang and Yong-Liang Yang and Ming-Ming Cheng},   
  journal={IEEE Transactions on Image Processing},    
  title={Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation}, 

If you have any questions, feel free to contact me via linzhuochen🥳foxmail😲com