Automated reconstruction of multiple objects with individual movement based on PSP

The problem

Phase shifting profilometry (PSP) projects several phase-shifted sinusoidal fringe patterns, normally at least three, onto an object and recovers its shape from the phase seen by a camera. Because it needs several images, the object has to stay still while they are captured. If it moves, the captured patterns no longer line up with each other and the designed phase shift between them is violated, so errors appear in the result.

The authors’ earlier work handled movement by describing it with a rotation matrix and a translation vector and building this information into the reconstruction model. It could also reconstruct several isolated objects, but only when they all had the same movement. In practice it is more common for objects to move differently, and existing methods introduce errors when multiple objects with individual movements are reconstructed.

How it works

The method gives every object its own movement description. It first finds the objects, then follows each one through the fringe sequence, computes its rotation and translation separately, and finally retrieves the phase of each object with that object’s movement.

1. Separate fringes from the object image. The fringe patterns are projected in red and captured by a colour camera. The red component carries the fringes for phase retrieval; in the blue component the fringes are filtered out, leaving a pure object image for tracking and feature extraction.
2. Identify the objects. In the blue image, OTSU automatic thresholding separates the objects from the background. Thresholding can split an object into several parts, so a dilation step merges them again, and the contour of each object gives its bounding box.
3. Track each object with KCF. The object inside each bounding box is used as a template, and the Kernelized Correlation Filters (KCF) tracker finds its new bounding box in the next fringe image. The new box then serves as the template for the images that follow.
4. Estimate each object’s movement. SIFT finds feature points and their correspondences between the object before and after movement. From these points, singular value decomposition (SVD) gives a rotation matrix and translation vector for each object.
5. Reconstruct object by object. With its own movement information, the phase of each object is retrieved from the red component by a least-squares solution of the reconstruction model that includes the movement, and the object is reconstructed. This repeats until the last object is done.

Results

The experiment uses a colour camera (Allied Vision Manta 504C, 2452 × 2056) and a projector (Wintech DLP PRO 4500, 912 × 1140). Two car models are measured with 3-step PSP using red fringes while each one rotates and translates on the reference plane in its own way. The reference is a 6-step PSP reconstruction of the static objects. Traditional PSP shows clear movement-induced errors, which are most serious where the height changes sharply, such as at the edges of the objects; with the proposed method these errors are removed. The method can also reconstruct only selected objects and ignore the others, by choosing the objects of interest after identification.

The authors state the conditions of the method. It considers only rigid objects with two-dimensional movement. Object identification works best with a large enough intensity difference between objects and background, and because dilation enlarges the objects, they must keep some distance from each other. The fringe colour should differ strongly from the object colour, and when ambient light is weak, blue background light should be added to the projected pattern. There is no limit on movement speed, but the capture must be fast enough to give clear images.