Kinematic target surface sensing based on improved deep optical flow tracking

The problem

Phase-shifting profilometry (PSP) measures 3D shape with one projector and one camera. The projector casts at least three phase-shifted sinusoidal fringe patterns onto the object, the camera records how the object’s height distorts them, and the phase recovered from the set of images gives the surface. Because several images are combined, the influence of ambient light and surface reflectivity is removed, which gives PSP high accuracy and robustness. However, PSP requires the object to stay still while the fringe images are captured: if it moves between captures, the images no longer line up and the phase calculation picks up errors.

If the object’s movement is known, as a rotation matrix and translation vector, the motion can be built into the phase calculation and the error removed. The difficulty is measuring that movement from images covered in fringes. Earlier work tracked markers placed on the object, and later work used SIFT features. The periodic fringes are “noise” to SIFT and cause mismatches, so the SIFT-based methods project the fringes in one colour channel of a colour camera and filter that channel out first, which needs colour equipment with a higher cost and lower frame rate than monochrome cameras. Another method tracks the object in the fringe images directly, but it is limited to 2D translation and cannot follow a rotating object.

How it works

The method tracks the object in the fringe images themselves, with an affordable monochrome camera, and handles any 2D movement, rotation included. A modified RAFT optical-flow network estimates where every object pixel has moved, a selection step keeps only the accurate estimates, and the resulting motion is fed into the phase calculation.

1. Motion breaks PSP. The fringes stay fixed while the object rotates and translates between captures, so each pixel sees a different part of the surface in each image. The paper models this by evaluating the reference phase at the moved coordinates in every fringe image.
2. Correlation by NCC instead of dot product. A feature encoder maps two consecutive fringe images to feature maps at 1/8 resolution, and a context encoder processes the first image. The original RAFT compares all pairs of features by a dot product; under fringe illumination the light intensity around a feature point changes as it moves through the fringes. The improved network uses normalised cross-correlation (NCC), which normalises local lighting variations, to build the 4D correlation volume and a four-level correlation pyramid.
3. Iterative flow updates. Starting from zero flow, a GRU-based update operator looks up the correlation pyramid and refines the flow over several iterations. The 1/8-resolution result is upsampled to full resolution, giving a displacement for every pixel.
4. Keep only accurate points. RANSAC first removes obvious outliers. Then a rotation matrix and translation vector are fitted, each point is moved by them, and points that land more than a threshold (such as 0.5 pixel) from their tracked position are discarded. The fit and the check are repeated until every remaining point meets the threshold.
5. Reconstruct with the motion. The final rotation matrix and translation vector describe the object motion, and the correct phase of the moving object is computed from them, giving its surface.

Results

The system uses a 1280 × 720 projector and a camera of the same resolution, with 4-step PSP. For training, 2000 pairs of fringe images before and after movement were captured and split 8:1:1 into training, validation and test sets. Labels came from images without fringes taken at the same positions, where SIFT gives the rotation and translation, from which each pixel’s motion is derived. The training loss converged to 0.25 pixels after 900 epochs. Counting a point as correct when its estimated displacement is within half a pixel of the SIFT-based ground truth, the NCC-enhanced network yields more correct points on the fringe images than the original RAFT (Fig. 10).

The same trained network was applied to four further objects (Fig. 13), and each was reconstructed while moving. The RMS errors of the proposed method are slightly above those of the SIFT-based method for every object, which the authors describe as comparable, while removing the need for colour equipment. The method covers movement in a 2D plane; reconstructing objects with complex 3D movement is left to future work.