Generative deep-learning-embedded asynchronous structured light for three-dimensional imaging

The problem

Structured light measures 3D shape by projecting a sequence of stripe images, called fringe patterns, onto an object and photographing them from a second viewpoint. The surface bends the stripes, and the stripe phase at each pixel encodes depth. In fringe projection profilometry (FPP), one of the most popular forms of the technique, the camera must capture each pattern while it alone is on screen. That requires synchronisation between projector and camera, usually a hardware trigger signal.

Synchronisation is a practical burden. Hardware trigger circuits are found in expensive industrial equipment, while affordable consumer devices often have no interface for them, even though the cameras in smartphones, tablets and laptops now offer high resolution and frame rates. Software synchronisation waits longer than the exposure time between patterns, which lowers the frame rate and does not always work stably. Signal wires also limit how far apart camera and projector can be and are susceptible to electromagnetic interference. If projector and camera simply run independently, though, the projector can switch patterns in the middle of an exposure, and the photo records a blend of two neighbouring patterns: an aliased fringe pattern (AFP). Computing phase directly from such images leads to reconstruction errors. Earlier work on unsynchronised structured light used simple binary (black-and-white) patterns or addressed other aspects of asynchrony; to the authors’ knowledge, asynchronous capture with the widely used sinusoidal fringes had not been studied.

How it works

The mixing itself has a simple form. Each pixel of an AFP adds up what the two patterns contributed during the exposure, and the split depends on the moment the projector switched (Eq. 5 of the paper). The mixing does not move pixels; it is the same at every location, a property that suits a convolutional network. The switching moment, however, is unknown and very difficult to measure, so recovering two images from one is an ill-posed, one-to-many problem. The paper treats it as image generation: a generative network learns to produce the two clean patterns, and standard phase-shifting reconstruction then applies unchanged.

1. Synchronised capture (the reference). Each exposure sees exactly one projected pattern, so each photo holds one clean pattern.
2. Asynchronous capture. Projector and camera each run at their own constant speed, with the camera faster so that every pattern is captured. A pattern switch inside an exposure produces an AFP that integrates the two neighbouring patterns. The paper addresses mixes of two patterns.
3. Separation with APSNet. APSNet (aliased fringe pattern separation network) is an encoder–decoder based on U-Net: four encoding blocks with max pooling, then a decoder that upsamples with skip connections back to full resolution. It maps one AFP to the two patterns it was mixed from. It is trained as the generator of a conditional generative adversarial network (GAN): a discriminator judges generated pattern pairs against real ones, and two further loss terms, mean-squared error and structural similarity, keep the generated patterns statistically close to the projected ones. Equal weights for the adversarial, mean-squared-error and structural-similarity terms gave the smoothest training (Fig. 8).
4. Phase. Each AFP mixes two neighbouring patterns, so three consecutive AFPs, which share a pattern with their neighbours, give four successive patterns, each shifted by a quarter period. The well-established four-step phase-shifting algorithm computes the wrapped phase, and adding the fringe order (multiples of 2π) unwraps it into a continuous phase map.
5. 3D shape. The continuous phase is converted into the object’s 3D shape with an established FPP reconstruction model.

The training data come from a real system: an industrial camera (2048 × 1536 pixels), a DLP projector (1280 × 1024) and a self-developed microcontroller that delays the camera by a set time Δt after the projector. With Δt = 0 the system captures ordinary, non-aliased fringes. The source patterns were five sets of 256 × 256 sinusoidal fringes with 50 to 90 periods, each in four phase shifts (0, 0.5π, π, 1.5π), captured with delays from 1 to 10 ms in 1 ms steps. This gave 1500 AFPs, each paired with its two source patterns: 1125 for training and 375 for testing.

Results

The test objects, including statues, a car model and a sphere, were not seen by the network during training or validation. Every comparison treats the reconstruction from synchronised patterns as the benchmark. Reconstructing directly from the AFPs loses much of the shape, while the shape from APSNet’s patterns is remarkably close to the benchmark (Fig. 12). In the paper’s summary, this improves reconstruction accuracy by three orders of magnitude over using the aliased fringes directly.

The network was trained only on sinusoidal patterns, yet it also separated aliased binary patterns projected onto a sphere, a car model and a statue, without further tuning (Fig. 14). It also handled a multi-object scene of a propeller prototype, a 3D-printed bolt and a statue, with different colours and reflective materials (Fig. 15). The method assumes that each photo mixes two patterns. If the projector runs much faster than the camera, more than two patterns fall within one exposure; the authors name this more complex aliasing as a limitation and a direction for further work.