SemReg: Semantics-Constrained Point Cloud Registration

The problem

Point cloud registration finds the rotation and translation that line up two partially overlapping 3D scans of the same scene, so that they can be merged into one. Recent learned methods describe every point by the shape around it, let the two scans exchange information through cross-attention, and then match points. Shape alone is often ambiguous, though: partial scans of a bed and a table can look geometrically alike. Cross-attention also lets every region of one scan interact with every region of the other, including pairs that are unlikely to overlap, such as a chair in one scan and a table in the other. These superfluous interactions lead to wrong matches.

Indoor scans captured with RGB-D cameras come with colour images, and semantic understanding is far more mature for 2D images than for 3D geometry. An earlier cross-modal method, PCR-CG, added image colour to the 3D features, but it uses only colour, not the semantic content of the images, and its features remain inconsistent.

How it works

SemReg is a 2D–3D cross-modal network. It keeps a standard 3D backbone for shape features and adds a semantic signal taken from the RGB frames, which it uses in three places: inside attention, at matching time, and in the training loss.

1. Two scans with their colour frames. The input is a pair of partially overlapping point clouds, each with its RGB images. A KPConv backbone downsamples each cloud and extracts local shape features.
2. Semantic features from the images. A pretrained DINOv2 network, kept frozen, extracts features from the RGB images. Using depth and the camera parameters, these are projected onto the 3D points and added to the shape features in self-attention.
3. A semantic prior guides cross-attention. For each source point, SemReg finds the target points whose semantic features closely resemble its own and places a Gaussian over 3D distance around each one. The mixture of these Gaussians, the Gaussian Mixture Semantic Prior (GMSP), weights cross-attention so that features interact mainly between semantically similar regions.
4. Matching within similar semantics. The Semantics Constrained Matching module sets the score of a candidate pair to zero when the semantic prior says the two points belong to unrelated regions. During training, a Semantics Aware Focal Loss supervises only pairs that are semantically related.
5. Pose from the surviving matches. The final rotation and translation are estimated from the correspondences with RANSAC, and can be refined further with ICP.

Results

The evaluation uses 3DMatch, an indoor RGB-D benchmark, and its low-overlap split 3DLoMatch (scan pairs with 10–30 % overlap). Registration recall is the share of scan pairs whose estimated transformation is accurate to within 0.2 m (RMSE). SemReg leads on both splits, ahead of SIRA-PCR on 3DMatch and GeoTransformer on 3DLoMatch, and in the paper’s ICP-refinement comparison its margin over the next-best method grows to 1.5 and 2.9 points. The semantic components matter most when overlap is small: they account for 9.4 points of recall on 3DLoMatch, against 2.6 points on 3DMatch. In one extreme case with 12.53 % overlap, where the only shared object is a chair, the semantic prior still locates the chair in the target scan and the registration succeeds, while GeoTransformer fails.

SemReg matches at a coarse level, with 0.1K–1K correspondences per pair against 6K–7K for dense methods such as GeoTransformer. Its inlier ratio (the share of correspondences within 0.1 m under the ground-truth transformation) is therefore lower than that of some competitors, although its registration recall is higher. Results on the outdoor KITTI dataset are reported in the supplementary material. The method needs RGB images aligned with the scan, so it does not apply directly to standalone point clouds; the authors suggest rendering images from the point cloud as a future workaround.