Reconstruction of transparent objects using phase shifting profilometry based on diffusion models

The problem

Phase shifting profilometry (PSP) projects a set of sinusoidal fringe patterns onto an object, recovers the phase from the captured intensities and turns it into 3D shape using the system calibration. Because it relies on the correct intensity of the captured fringes, it works well for objects with diffusely reflecting surfaces. On a transparent object such as glass, the projected fringes are reflected by the front and the back surface at the same time, and some light bounces several times between the two surfaces before reaching the camera. The captured pattern is then an aliased mixture of several fringe patterns, and the phase computed from it does not describe the surface being measured.

Earlier work on transparent objects falls into reflection-based methods (for example heating the surface with a laser and imaging it with an infrared camera, which is time-consuming), refraction-based methods that trace light through the object, and deep learning methods. This paper keeps a standard PSP set-up and instead removes the unwanted light from the fringe images themselves.

How it works

The authors model each captured fringe image as a sum of the front-surface pattern, the back-surface pattern, the multiple-reflection pattern and background light. They then train a conditional diffusion model to generate only the front-surface pattern from the aliased one, and pass the generated patterns to ordinary PSP.

1. Three light paths. Fringes reach the camera after reflection from the front surface (F), a single reflection from the back surface (B), and multiple reflections between the two surfaces (M). F and B are normally the strongest components and M the weakest.
2. A sum to undo. Each captured fringe image is S = F + B + M + A, where A is the background light. Only F, together with the background light, can be used to reconstruct the front surface, so a network p is trained to sample an estimate F* ∼ p(S; θ).
3. Training. The front-surface pattern x₀ is captured after spraying a developer onto the object, which gives clean front-surface fringes, together with the aliased pattern y of the same scene. Gaussian noise e is added to x₀ in a forward diffusion process (T = 2600 steps, β rising linearly from 0.0001 to 0.002), and a U-Net with residual blocks, given the noisy pattern x_t, the step t and the condition y, learns to predict the noise.
4. Sampling. Starting from pure Gaussian noise, the trained network, conditioned on the captured aliased pattern y, estimates and removes noise one step at a time until t = 0, giving the front-surface pattern F*.
5. Conventional PSP. The estimated patterns are processed with the 4-step PSP algorithm to obtain the wrapped phase and the 3D shape of the front surface.

Results

The set-up uses a 2048 × 1536 camera, a 1280 × 1024 DLP projector and 4-step PSP. The authors tried simulation and pasting cardboard stickers onto the surface to obtain clean front-surface fringes, but simulated images differed too much from real ones and stickers are hard to apply to non-planar objects, so the dataset was built by spraying a developer. For each object, 15,000 image pairs were used for training and 1500 for testing, with the objects placed at random positions and poses. The test objects are an acrylic transparent cube (25 mm side), a glass transparent hemisphere (20 mm radius), an acrylic translucent object and a glass partially-transparent object.

For each object, the estimated front-surface pattern is compared with the pattern captured after spraying the developer, in intensity, in wrapped phase and in the reconstructed shape. The intensities are not strictly identical, but the estimated patterns have the same characteristics, and the phase values in the object regions closely agree. The translucent object is a case where refraction-based methods do not work, because light cannot pass through the object to be refracted. Partially-transparent objects are hard for most existing methods because the transparent and the solid parts need different reconstruction algorithms; here, light that does not come from the front surface is simply treated as noise and removed by the diffusion model.

The authors note one limitation: because the generated patterns start from Gaussian noise, some residual noise, invisible in the images, remains and can reduce the accuracy of the phase and of the 3D reconstruction. Using more diffusion steps reduces it, but they found that this increases the computational cost of both training and sampling.