GCMA6D: Graph Convolution and Cross-Modality Attention Fusion for 6D Pose Estimation

The problem

6D pose estimation recovers the 3D rotation and translation of an object relative to the camera, a step that robotic manipulation and augmented reality depend on. It remains hard for low-texture objects and under partial occlusion. Methods that use RGB images alone lack depth information, which limits their robustness in complex environments. RGB-D methods such as DenseFusion fuse colour and point-cloud features pixel by pixel, but they describe the geometry with PointNet, which may miss fine local detail. Methods that add local feature modelling, such as PVN3D, raise the computational cost, and the size and inference latency of large foundation models hinder real-time deployment.

GCMA6D builds on the pixel-wise fusion of DenseFusion, the baseline of its ablation study, and strengthens each part of it: graph convolution models local geometry, Large Kernel Attention sharpens the image features, and a cross-modality attention module lets colour and geometry inform each other before they are fused.

How it works

The network takes a single RGB-D image. It segments the object, extracts colour features from the image crop and geometric features from the object’s point cloud, fuses them with attention, and regresses the pose from every pixel of the object.

1. Segment and crop. The semantic segmentation network from PoseCNN produces a mask for each object. The mask crops the colour image and cuts the object out of the depth map, which the camera intrinsics convert into a 3D point cloud. This suppresses background interference before any features are computed.
2. Colour features with Large Kernel Attention. A ResNet18 backbone and a pyramid pooling module extract multi-scale features from the image crop. Large Kernel Attention, a depth-wise convolution, a depth-wise dilated convolution and a 1×1 convolution, builds an attention map that is multiplied with the features, so the network captures local texture and edges as well as longer-range context.
3. Geometric features with 3DGCN and PointNet. A 3D graph convolutional network applies learnable 3D graph-convolution kernels to the neighbourhood of each point, followed by graph max pooling, to model local geometric structure. A PointNet branch processes the points with fully connected layers and obtains a global shape feature through max pooling.
4. Cross-modality attention. The CMA-Net module takes the colour features (F1), the 3DGCN features (F2) and the PointNet features (F3). One dot-product attention uses F1, F2, F3 as query, key and value to enhance the colour features; a second uses F2, F1, F3 to enhance the point features. A Squeeze-and-Excitation module reweights the channels of both outputs and of F3, and the three are concatenated.
5. Dense fusion and pose. The fused features of each pixel pass through shared MLPs and average pooling to form a global feature, which is appended to every pixel’s feature. Each pixel then predicts a rotation, a translation and a confidence; the most confident prediction is kept. Following DenseFusion, a small network then refines it iteratively by predicting residual poses.

Results

The evaluation uses two public datasets. LineMOD contains low-texture objects against complex backgrounds under varying illumination; 15 % of the images are used for training and 85 % for testing, and a pose counts as correct when its ADD distance (ADD-S for symmetric objects) is below 10 % of the object’s diameter. In Table 2, GCMA6D has the highest mean accuracy of the seven methods compared, ahead of TMFNet (96.8 %) and QaQ (95.3 %), and reaches 100 % on the symmetric eggbox and glue. The ablation study on LineMOD shows that the modules complement each other: any two together score 96.8–97.4 %, and all three give 97.7 %.

YCB-Video has 21 objects with varying levels of occlusion; testing uses 2,949 key frames from 12 videos. Accuracy is measured as the area under the ADD(-S) accuracy curve for thresholds up to 10 cm, and as the share of poses with an error below 2 cm. GCMA6D’s 92.9 % AUC is above SaMfNet (92.6 %), G2L-Net (92.4 %), MSCNet (91.5 %) and DenseFusion (91.2 %). The authors note lower accuracy on the large and extra-large clamps: the two look very similar, so PoseCNN’s segmentation struggles to produce precise masks for them. They also name highly symmetric or heavily occluded objects as a remaining limitation, and plan to improve the processing of background-masked regions.