Dynamic correlation network and structure-aware matching for robust point cloud registration

The problem

Point cloud registration finds the rotation and translation that align two partially overlapping 3D scans of the same scene, for example LiDAR or RGB-D scans captured from different viewpoints. Learned methods describe each point by features aggregated from its neighbourhood and then match points across the scans. When the relative rotation between the scans is large, these features, and the correlations between them, can change with the orientation, and the matches become unstable.

The common remedy is to train with randomly rotated scans, but augmentation cannot cover the continuous space of 3D rotations, SO(3), so orientations not seen in training still degrade performance. Rotation-equivariant networks such as PARE-Net help, yet they still combine neighbouring features through a fixed multi-layer perceptron (MLP) that applies the same correlation model to every neighbourhood. And even when two superpoints (coarsely sampled points, each standing for a local patch of the scan) are correctly matched, the individual points inside them can still be paired wrongly.

How it works

The method follows the coarse-to-fine pattern: a backbone extracts coarse superpoints (a sparse set of points, each standing for a local patch of the scan) and dense point features, superpoints are matched first, and points are then matched only inside matched superpoint patches. It changes two parts of this pipeline. A Dynamic Correlation Network (DCN) inside the PAREConv-FPN backbone decides how to combine neighbouring features from rotation-invariant cues, and a Structure-Aware Point Matching (SAPM) module refines the point matches inside each patch.

1. Rotated input. The two scans arrive at an arbitrary relative rotation; the network has to estimate the rigid transform between them even when that orientation was not seen during training.
2. A rotation-invariant cue. For each point and its nearest neighbours, a vector-neuron encoder builds features from the relative offsets, the mean offset of the neighbourhood and their cross product. These features rotate with the scan; their magnitudes do not, so the magnitudes serve as a rotation-invariant descriptor.
3. Dynamic kernels. A channel reallocation network turns that descriptor into softmax scores over a small bank of learnable kernels (four in the experiments), and each neighbour’s kernel is the score-weighted sum of the bank. Because the scores depend only on invariant quantities, kernel selection does not change with rotation, while the aggregated features remain rotation-equivariant.
4. Context inside each patch. Each matched superpoint pair defines two local patches. SAPM refines every point’s descriptor with linear self-attention over its own patch, at a cost linear in the patch size; points that lie closer to a different superpoint are masked out.
5. Consistent matches and the pose. A row-wise and a column-wise masked softmax over the patch similarity matrix are fused with per-point confidence scores. Entries that are in the top k in both directions become dense matches. Each patch yields a pose hypothesis by SVD, and a feature-based hypothesis proposer keeps the best one, without RANSAC.

Results

3DMatch and 3DLoMatch are indoor benchmarks with overlaps above 30 % and between 10 % and 30 %; KITTI consists of outdoor driving scans. On 3DMatch the method is close to the best RANSAC-based method, BUFFER-X (95.6 %), and slightly ahead of DFAT and PARE-Net (both 94.9 %); with RANSAC in place of its own estimator it still reaches 95.2 % and 79.3 %. When arbitrary rotations are applied to 3DLoMatch, methods that rely mainly on rotation augmentation, such as FCGF and Predator, lose transformation recall, whereas methods with rotation-invariant or equivariant designs, this one included, stay stable. On KITTI, CAST has a lower translation error (2.5 cm), while this method has the lowest rotation error. In the KITTI ablation, DCN and SAPM each reduce the errors on their own, and together they lower the rotation error from 0.239° to 0.211° and the translation error from 5.5 cm to 4.8 cm (Table 6). The model has 4.04 M parameters and takes 100.29 ms on an RTX 4090, faster than CAST (129.80 ms) and PARE-Net (122.80 ms) (Table 5).

The authors note two limitations: when the overlap is critically small, coarse superpoint matches can be wrong, and highly repetitive structures can produce ambiguous descriptors that mislead hypothesis selection.