feature fusion module is proposed to fuse different layer features of the backbone network. A joint instance semantic segmentation module is developed to transform semantic features into instance embedding space, and further fused with instance features to facilitate instance segmentation.
In this paper, we propose a novel joint instance and semantic segmentation
approach, which is called JSNet, in order to address the instance and semantic
segmentation of 3D point clouds simultaneously. Firstly, we build an effective
backbone network to extract robust features from the raw point clouds.
Secondly, to obtain more discriminative features, a point cloud feature fusion
module is proposed to fuse the different layer features of the backbone
network. Furthermore, a joint instance semantic segmentation module is
developed to transform semantic features into instance embedding space, and
then the transformed features are further fused with instance features to
facilitate instance segmentation. Meanwhile, this module also aggregates
instance features into semantic feature space to promote semantic segmentation.
Finally, the instance predictions are generated by applying a simple mean-shift
clustering on instance embeddings. As a result, we evaluate the proposed JSNet
on a large-scale 3D indoor point cloud dataset S3DIS and a part dataset
ShapeNet, and compare it with existing approaches. Experimental results
demonstrate our approach outperforms the state-of-the-art method in 3D instance
segmentation with a significant improvement in 3D semantic prediction and our
method is also beneficial for part segmentation. The source code for this work
is available at https://github.com/dlinzhao/JSNet.