DeSC: Learning Deep Semantic Descriptor for NeRF Registration

The problem

A Neural Radiance Field (NeRF) stores a scene inside a neural network. Large scenes are often split into several NeRFs, and images captured without reliable global camera poses produce NeRFs that each live in their own coordinate frame. To combine them, one has to estimate the rigid rotation and translation between two independently trained NeRFs: NeRF registration.

Existing methods rely on either geometry or rendered colour. DReg-NeRF samples a point cloud from each NeRF and regresses correspondences, but these point clouds are noisy and contain artifacts from inaccurate density estimates, especially when a NeRF is under-trained. VF-NeRF optimises with a photometric loss, which makes it vulnerable to textureless and highly symmetric cases. Both ignore the view-independent embedding that a NeRF computes internally for every 3D point, which carries both geometry and appearance. Using it directly is not straightforward either: the two NeRFs are trained separately, so their embedding spaces do not match.

How it works

DeSC treats each NeRF as a source of point features, not only of geometry. It samples a point cloud from each NeRF, extracts geometric features with a KPConv backbone, and adds a Deep Semantic Aggregation Module (DSAM) that turns the NeRF’s own embeddings into descriptors that agree across the two NeRFs. A Transformer and a coarse-to-fine matcher then find correspondences, and the pose is solved by SVD.

1. Sample each NeRF. DeSC takes the centres of the non-empty voxels in each NeRF’s occupancy grid and keeps those whose density exceeds a threshold σt. KP-FCNN downsamples this point cloud hierarchically and extracts multi-level geometric features.
2. Query embeddings around each sparse point. For every sparse point (a point of the coarsest downsampled level), a patch is formed from its k nearest dense points (from the next, denser level) within a radius τ. Each patch point queries the NeRF for its view-independent embedding, which is enriched with the point’s colour and density through a sinusoidal encoding, so that the features carry photometric information shared by both NeRFs.
3. Aggregate with a texture-aware graph convolution. A graph convolution runs over each patch. Its edge weights emphasise points that are close to each other but have contrasting features, capturing local high-frequency texture. The output is fused with the geometric features to give the deep semantic descriptor.
4. Add context and match. Geometric self-attention and cross-attention let each point cloud gather context from itself and from the other. Coarse correspondences are selected from the resulting features and refined on dense points with an optimal transport layer.
5. Solve the pose. Outliers are removed, and SVD gives the rotation and translation that align the two NeRFs. During training, a density-aware photometric consistency loss also asks matched patches to have similar density-weighted colour.

Results

The object-level benchmark is built from Objaverse: for each object, 120 rendered images are split into two blocks, each block is given a random transformation, and a separate NeRF is trained on each; 44 objects unseen during training are used for testing. Registration recall counts the pairs whose estimated transformation has an RMSE below 0.2 m. DeSC is compared with the NeRF registration methods DReg-NeRF and VF-NeRF, and with eight point cloud registration methods run on points sampled from the NeRFs. With uniform noise of 10 % of the scene size added, its recall is still 95.74 % (Table II).

On the real-scene datasets DTU, LLFF and Mip-NeRF360, each scene is split into image subsets with 30 %, 50 % and 80 % overlap. DeSC has the highest registration recall in all nine settings, and the margin is largest at 30 % overlap, although VF-NeRF reaches a slightly lower rotation error in some settings. In the ablation, adding the deep semantic aggregation, the photometric encoding, the texture-aware edge weights and the photometric consistency loss in turn raises Objaverse recall step by step, from 93.2 % with none of them to 93.9 %, 95.1 %, 96.9 % and 97.7 %.

The authors state two limitations. DeSC relies on NeRF’s implicit embeddings, so it does not apply directly to explicit representations such as 3D Gaussian Splatting, voxel grids or hash-based methods. Objects that are highly symmetric in both shape and texture, such as a traffic cone, yield symmetric features and can be matched wrongly, even when the rendered alignment looks plausible.