SFRF6D: Selective Fewer-Reference Fusion for 6D Pose Estimation

The problem

6D pose estimation recovers an object’s 3D rotation and 3D translation, the information a robot needs for precise grasping and assembly. Most existing methods depend on a CAD model of the object or on a large collection of reference images. Model-based methods match image features to the CAD model, which limits them in real settings where no model exists. Model-free methods such as Gen6D, OnePose and FoundationPose instead build a multi-view representation from many reference images or a video, and their cross-view alignment becomes hard to keep stable under severe occlusion or large viewpoint changes. Methods that need only a few references lose much of their matching ability when the reference views barely overlap.

SFRF6D sets out to estimate the pose with no CAD model and only a few RGB-D reference images whose poses are known, while staying robust when the object is heavily occluded.

How it works

SFRF6D first rebuilds a rough 3D shape of the object from the reference images and aligns it coarsely with the query. It then renders reference views from that coarse pose and matches them with the query through attention that is steered by 3D geometry, so that visible regions count and occluded ones do not. Finally it refines the pose and chooses among pose hypotheses by rendering and comparing.

1. A few references, one query. The input is a small set of RGB-D reference images with ground-truth poses (four in the main experiments) and an RGB-D query image. A query mask segments the object.

2. Coarse 3D shape. The reference view whose viewpoint is closest to the query becomes the anchor; a ResNet18 extracts its features and its depth map gives an initial object point cloud. The other references are transformed into the same world frame with their camera intrinsics and known poses, filling in the shape, which is then normalised to the range [−1, 1].

3. Coarse alignment. An oriented bounding box gives the object’s centre (the mean of its points) and its rotation (the principal axes from PCA of the point covariance). Because this is still rough, rotation and scale are tuned by maximising the overlap (IoU), which gives the coarse alignment.

4. Geometry-guided sparse attention. Reference point clouds are rendered from the query’s coarse pose. Normalised direction vectors between reference and query points build a top-K mask for cross-attention, with Gumbel-Softmax used for soft top-K sampling; pairs that fail to match are masked out, so attention concentrates on visible, relevant regions. A multi-scale fusion module weighs global against local features.

5. Refine and select. Refine: the object scale and pose are optimised jointly: the pose by rendering the shape and minimising its projection error against the anchor image, and the scale by sampling scales within a range and choosing by IoU against the reconstructed shape. The relative pose to the query is then computed.

   Select: each pose hypothesis is rendered and compared with the query to give a feature vector; a self-attention module scores all hypotheses together, and top-K filtering selects the final pose.

Results

The comparison uses three BOP benchmarks: LINEMOD (LM), Occlusion LINEMOD (LMO), where object visibility can drop below 40 %, and YCB-Video (YCBV). Accuracy is the ADD(-S) score at a threshold of 0.1 times the object diameter. Among the methods that do not use a CAD model, SFRF6D is the most accurate on all three datasets (92.1 %, 67.3 % and 90.2 %) and the fastest. FoundationPose, which uses the CAD model, remains more accurate, with a mean of 89.7 % at 1.0 s; the authors describe SFRF6D’s precision as slightly lower, in exchange for needing no CAD model and less computation.

The ablation on LMO (Table 2) separates the parts. Removing the view selector, which chooses among the rendered views, and using all of them keeps accuracy close (66.2 %) but raises the time from 0.94 s to 1.32 s. The number of references matters most at the low end: one reference gives 57.3 %, two give 63.4 %, four give 67.3 % and eight give 68.1 % at 1.03 s, so the authors settle on four as the balance between accuracy and cost. With a single reference the reconstructed shape depends heavily on that one view, and features cannot be aligned robustly when occlusion or the viewpoint difference is large. The authors list pretrained foundation models, dynamic multi-object scenes and lighter models for mobile and edge platforms as future work.