Deep transformer-enhanced dynamic structured light for three-dimensional moving object imaging

The problem

Phase-shifting profilometry (PSP) projects a sequence of sinusoidal fringe patterns, each shifted by a known amount, and recovers the 3D shape of a surface from the phase of the captured images. It is robust to ambient light and to changes in surface reflectivity, but it needs at least three images, and the object must stay still while they are taken. If the object moves, each surface point is lit by different projector pixels and seen by different camera pixels in each frame, so the phase steps between frames are no longer the intended ones. The result is ripples, phase-unwrapping errors or a failed reconstruction.

Faster projectors and cameras reduce the problem but raise hardware cost, and the effect of motion still remains. Most error-estimation methods handle motion only along the depth direction and do not address the pixel mismatch caused by motion in other directions. Tracking-based compensation models such as Lu’s work well within the X–Y plane but handle Z-direction motion poorly, and methods that estimate a single global translation miss local, pixel-level motion differences. When an object moves in an arbitrary direction, every surface point has its own displacement and therefore its own phase error.

How it works

The method keeps standard four-step phase shifting and adds two things: a learned tracker that finds where each pixel of one fringe image went in the next ones, and a pixel-wise model of the phase error that remains after that tracking. Both run inside a short sliding window of five frames.

1. Model the motion-induced phase error. The projector cycles four-step fringes (π/2 apart) while the camera captures the moving object. The motion of each surface point is split into an in-plane displacement, which moves the point to another pixel (Δu, Δv), and a displacement along Z, which adds a phase error. The total phase error is the sum of the two.
2. Track pixels with a transformer. The image sequence is split into windows of five frames, and the middle frame is the key frame. A COTR (COrrespondence TRansformer) network, initialised with its original pre-trained weights and fine-tuned on fringe images, matches query pixels in the key frame to the other four frames. A shared CNN backbone extracts features from both images, a transformer encoder–decoder relates them, and an MLP head predicts each match.
3. Align the frames and solve for the Z error. The matches define a pixel-wise mapping that aligns all five frames so the same surface point is followed through the window. Within the short window, the Z-direction velocity is treated as constant, so each frame adds the same extra phase error ε, and frame n of the window (n = 1 to 5, key frame n = 3) carries a Z part of (n − 3)ε. Combining the intensities of the five frames gives three equations in the object phase Φ and ε. For small ε they reduce to a linear equation K₁ε + K₀ = 0, which is solved at every pixel, and the corrected phase Φ follows.
4. Unwrap and reconstruct. The corrected wrapped phase is unwrapped with a three-frequency phase-shifting algorithm, using two sets of lower-frequency fringes projected alternately with the high-frequency ones, and converted into 3D shape with the calibrated camera–projector model.
5. Slide the window. The window moves on by one frame, so each newly acquired image, together with the four before it, gives a new 3D shape of the object.

Results

The system uses a high-speed camera at 512 × 512 pixels and 30 fps and a DLP projector. To fine-tune COTR, the authors captured 45 dynamic scenes of sculptures, toys and industrial parts on a shallow box of sand (about 5 cm deep), with 60 motions each, giving 2,700 pairs of pre- and post-motion fringe images (2,400 for training, 300 for validation). Ground-truth matches came from an extra blank image projected at each pose, on which a robust local optical flow method computed dense displacements. Training converged after about 1,400 epochs and took about 42 hours.

The ceramic plate was moved by a translation stage at 40° to the camera’s optical axis in 1 mm steps, with a 12-step phase-shifting measurement as ground truth. Standard PSP shows clear ripple distortion. Guo’s method smooths part of it, Lu’s method compensates only motion in the X–Y plane, and Wang’s method simplifies the phase-shift error, which leads to even larger errors. With steps of ±1, ±2 and ±3 mm, the RMS error of the proposed method stays below 0.1 mm (Fig. 6(f)). On the statue, PSP shows stripe artifacts and discontinuities along the edges, Guo’s method still has large errors at the edges, and Lu’s method leaves periodic ripples because Z-direction motion is not compensated; the proposed method recovers the contour more accurately.

The method was also tested on a static statue next to a rotating one, on a rotating fan model with a smooth, weakly textured surface, on a statue on a continuously rotating platform, and on objects that translate and rotate at the same time. Under combined translation and rotation, the rotation adds phase deviations of up to about 1 rad in the outer regions, and the method still keeps most surface areas accurate. The authors note its limits: when the translation along Z is too large, phase-error estimation can still fail (marked regions in Fig. 12); the authors suggest increasing the projection rate or combining the framework with geometry-constraint-based or Gray-code-based unwrapping. Because the estimate relies on a five-frame window, the method works reliably for small inter-frame motion; standard frame rates suffice for slow to moderate motion, while fast objects need high-frame-rate cameras and projectors.