30 Overview of 3D Imaging
Everything in the preceding twenty-nine chapters rests on a single premise: a two-dimensional image. A grayscale image can tell you “where it is dark and where it is bright”, and armed with the first seven parts of this book you can already squeeze that information to its limit — localizing to 0.02 pixel, measuring a width of 5 micrometers, catching a defect covering 0.1% of an area. But there is a class of problems that a 2D image is fundamentally powerless against: camera imaging is a single projection, and at the instant the three-dimensional world is flattened into a plane along the optical axis, the height information is lost forever. A cold-soldered solder ball and a full one may look like two nearly identical bright blobs from directly above; whether the seam between two pieces of metal on a phone’s middle frame is flush (the step height) is, in a top-down view, one and the same line.
Industry’s wish list for 3D is very concrete: the coplanarity of connector pins — whether the tips of dozens of pins lie in the same plane, where a 0.05 mm difference means they will not seat into the pads; the step height and gap of an assembly; the volume of dispensed glue and solder paste — how much glue there is directly determines bond strength, and volume is area times height, of which 2D can give only the first half; the depth of scratches and dents — the very same scratch, 2 μm deep, can ship, while at 20 μm it must be scrapped, yet in a grayscale image the two may look equally dark. What these judgments share is that the quantity being measured is height itself; any two-dimensional method can only guess at it indirectly, and the cost of guessing wrong is paid by the production line.
And so this book enters the third dimension. Part VIII spans seven chapters: this one is the map, and each of the following six covers a single 3D imaging technique — stereo vision, structured light, laser triangulation, photometric stereo, deflectometry, and focus methods. None of the six is inherently better or worse than the others; there is only the matter of fit: for one and the same height-measurement need, change the surface material and the optimal solution may be entirely different. The task of this chapter is to make “how to choose” clear: first to unify the language of 3D data, then to give each technique a physical intuition, and finally to land on a single selection table and a few real decision paths.
A note up front: this is a theory chapter (an overview of technique selection), with no standalone companion code project and no figures; the principle experiments, real data, and implementation code for each technique are given in its own dedicated chapter. The core deliverable of this chapter is the selection comparison table in Section 30.3.
30.1 The Forms of 3D Data
Before discussing techniques, let us settle what “3D data looks like”. Industrial 3D vision has three principal representations, and they carry different amounts of information and suit different algorithms.
A depth map, also called a range image, is the representation closest to 2D thinking: it is still an “image” on a regular grid, except that each pixel stores not a gray value but the height or distance at that position (typically a 32-bit float in mm). Nearly all the tools of the earlier chapters — filtering, thresholding, blobs, calipers — can be carried over almost untouched, except that “gray-value difference” becomes “height difference”. This representation is also called 2.5D: each \((x, y)\) position holds only one height value, expressing “the surface seen from a particular viewpoint” rather than a complete 3D solid — the inner wall of a cup, or the side and bottom faces of a part, simply do not exist in a single-view depth map.
A point cloud is an unordered set of three-dimensional coordinate points \(\{(x_i, y_i, z_i)\}\), optionally carrying attributes such as normals and color. It does not rely on a regular grid and can express a surface of arbitrary viewpoint and arbitrary topology; merging multi-view data can approximate a true 3D model. The price is the loss of the implicit structure “the neighbor lives in the adjacent pixel” — finding nearest neighbors now relies on a spatial index (a kd-tree), and algorithm costs rise overall. A mesh restores connectivity on top of the point cloud — describing surface topology with triangular facets, it is the common language of CAD comparison and visualization.
The three convert into one another: back-projecting a depth map pixel by pixel using the calibration parameters yields an ordered point cloud; surface reconstruction from a point cloud yields a mesh; resampling a mesh along some direction returns to a depth map. The mainstream path of industrial metrology is “the sensor outputs a depth map or an ordered point cloud → convert to an unordered point cloud when necessary for registration and comparison”. The outputs of the six technique chapters that follow are basically depth maps; the data structures and operations of point clouds are developed systematically starting from Chapter 37.
“2.5D or true 3D” is the first clarifying question when selecting a solution. Most inline inspection (coplanarity, step height, volume) is fine with single-view 2.5D; only when a complete contour is needed (such as reverse engineering or comparison of complex curved surfaces) is multi-view stitching required, and its cost and cycle time must be budgeted separately. Mistaking the need for true 3D can multiply the budget several times over.
30.2 The Physical Principles of the Six Techniques
The six techniques look wildly different, but the underlying logic is a single sentence: height itself cannot be imaged directly, but using a known geometric or optical constraint it can be transformed into some measurable image quantity — disparity, phase, displacement, shading, slope, or sharpness — and then inverted back. Let us go through, one by one, what that “image quantity” is.
Stereo vision (Chapter 31): the image quantity is disparity. Two cameras separated by a baseline shoot simultaneously, and for the same spatial point the difference in column coordinates between the left and right images is inversely proportional to depth — near points “jump” a lot, far points “jump” little, the same mechanism by which human eyes judge distance. It is a passive technique that projects no light and has the simplest structure; but “the same point corresponds across the two images” must be established by matching the image content itself, so it depends on surface texture — a smooth, uniform surface (a white wall, polished metal) yields no correspondences, and the depth map is left with stretches of holes.
Structured light (Chapter 32): the image quantity is phase. If passive matching fears the lack of texture, then actively “print” texture onto the surface — a projector casts sinusoidal fringes onto the object, and a camera observes from another angle: where the surface rises and falls, the fringes are distorted, and the amount of distortion is encoded in the phase of the fringes. Phase-shifting can solve for the phase independently at each pixel, giving high spatial resolution and single-shot area imaging; it is the workhorse for full-field measurement of diffuse surfaces.
Laser triangulation (Chapter 33): the image quantity is the pixel displacement of the laser line. A line of laser light strikes the surface at a known angle, and a camera views that bright line from another angle: the higher the surface, the larger the offset of the bright line in the image — two angles and one side of the triangle are known, so the height follows. A single exposure yields only one height profile, and the full surface is stitched line by line from the scanning motion of the workpiece or sensor; it is therefore naturally suited to conveyor and robot-arm scenarios, and is the most widely installed solution for industrial inline 3D inspection.
Photometric stereo (Chapter 34): the image quantity is shading. Camera and workpiece both stay still while light is cast in turn from several known directions for several exposures: faces oriented toward the light are bright, faces turned away are dark, and combining the several shadings solves for the surface normal at each pixel. Note that it measures the normal (slope) directly rather than height; absolute height must be recovered by integration, with errors that accumulate. But it is extremely sensitive to microscopic relief — scratches, dents, embossed characters, and the like, features with “tiny height change but drastic slope change”, show up in vivid detail in the normal map.
Phase measuring deflectometry (PMD, Chapter 35): the image quantity is the phase distortion of the reflected fringes, corresponding to surface slope. The previous techniques all assume a diffuse surface and fail en masse on specular surfaces (phone glass, polished metal, car paint) — the light cast onto them bounces off by the law of reflection, and the camera sees a mirror image of the environment rather than the surface itself. PMD turns this around and uses it: it lets the fringes displayed on a screen enter the camera after specular reflection off the surface, and the slightest change in surface slope distorts the fringes in the reflected image noticeably — the more “perfect” the mirror, the more sensitive the measurement. It is the dedicated solution for specular and near-specular surfaces.
Focus variation and confocal (Chapter 36): the image quantity is sharpness. A lens has finite depth of field, and the object surface is in focus only where it lands on the focal plane. Scanning layer by layer along the optical axis to acquire an image stack, for each pixel one finds the layer at which it appears “sharpest”, and the Z position of that layer is the height at that point. Confocal uses a pinhole to physically reject out-of-focus light, achieving sub-micron axial resolution. This family is slow (the whole height range must be scanned) and has a small field of view, but its accuracy is the best of the six, targeting roughness and microstructure measurement at the microscopic scale.
Classifying by “image quantity” also lets you anticipate each technique’s failure mode: those relying on disparity fear the lack of texture, those on phase fear glare and interreflection, those on displacement fear occlusion, those on shading fear non-Lambertian surfaces, those on slope recognize only mirrors, and those on sharpness fear transparency and slow speed. The first round of elimination in selection is often simply matching failure modes against the workpiece surface.
30.3 Selection Decisions
Selection has four core dimensions. Surface type: diffuse, specular, or transparent? This is the hardest constraint and eliminates half the candidates outright. Measurement range and resolution: is the field of view at the millimeter scale or the meter scale? Does the height resolution need to be micron or sub-micron? The two are roughly proportional — the larger the field of view, the coarser the single-point accuracy. Speed and motion pattern: single-frame imaging, workpiece scanning required, or multiple frames in place? This decides whether the system can keep up with the cycle time and whether it can measure a workpiece in motion. Ambient-light immunity: active projection guarantees a signal-to-noise ratio, while passive techniques must be especially careful amid stray light on the shop floor.
The six techniques compare as follows (the numbers are order-of-magnitude figures for typical industrial products; specific models vary):
| Dimension | Stereo vision | Structured light | Laser triangulation | Photometric stereo | PMD | Focus / confocal |
|---|---|---|---|---|---|---|
| Image quantity measured | Disparity | Phase | Line displacement | Shading → normal | Reflected phase → slope | Sharpness |
| Active / passive | Passive | Active | Active | Active | Active | Active |
| Diffuse surface | Needs texture | Excellent | Excellent | Excellent | Not possible | Excellent |
| Specular surface | Not possible | Poor | Poor | Poor | Specialty | OK |
| Transparent surface | Not possible | Poor | Poor | Not possible | OK (surface) | Excellent (confocal) |
| Typical field of view | 0.1–10 m | 0.05–1 m | 0.01–1 m (wide) × scan length | Same as 2D FOV | 0.05–0.5 m | 0.1–5 mm |
| Height resolution | mm scale | 5–50 μm | 1–20 μm | Normal sensitivity / height difference | Slope, μrad scale | 0.01–1 μm |
| Imaging pattern | Single frame | Multi-frame (a few fringe casts) | Line-by-line scan | Multi-frame (switch light) | Multi-frame (screen fringes) | Scan along Z |
| Can measure moving workpiece | Yes | Hard | Naturally suited | Hard | Hard | No |
| Ambient-light immunity | Weak | Medium | Strong (narrowband filter) | Medium | Medium | Strong |
| Typical applications | Logistics volume, robot guidance | Solder paste / glue, full-surface appearance | Step height, burrs, weld seams | Scratches, embossed characters | Glass / car-paint surface shape | Roughness, microstructure |
Let us walk a few typical scenarios through the table. Phone middle-frame step height (highly reflective metal, on the order of 0.01 mm, inline cycle time): the surface rules out stereo and structured light, leaving PMD and laser triangulation as candidates — the former measures slope and is keener on step-like jumps such as step height, while the latter, combined with shading and angle optimization, can often be made to work too, and the two frequently compete head-to-head in trials. PCB solder-paste inspection (diffuse paste, requiring area + height + volume, tight cycle time): structured light covers the whole board in a single area shot and is the mainstream choice for SPI equipment; scanning laser triangulation competes on equal footing in high-precision cases. Battery electrode-foil burrs (continuous web feed, burr height tens of microns): the workpiece itself is in motion, so line-by-line scanning laser triangulation is the natural answer. Polished-metal scratches (specular, micron-deep, judging depth rather than presence): PMD reads the slope jump and photometric stereo reads the normal jump, one of the two or a combination of them, while the other techniques have essentially no foothold.
As you can see, selection is not “pick whichever has the highest accuracy” but rather: surface type makes the first cut, accuracy and field of view make the second, speed and motion pattern make the third, and the remaining candidates go to trial.
30.4 Common Engineering Elements
The six techniques each have their own chapter, but a few matters are common to all 3D systems, and we set up the framework here first.
Calibration. The raw quantity each technique measures is an image quantity — disparity, phase, pixel displacement — and converting it into a physical height in millimeters relies on calibration in every case: stereo must calibrate the intrinsics and extrinsics of the two cameras, laser triangulation must calibrate the pose of the light plane relative to the camera, structured light must calibrate the projector, and PMD must calibrate the screen. The camera model and calibration methodology established in Chapter 5 are the foundation of all of Part VIII; one might say that the accuracy ceiling of a 3D system is written down already at calibration time.
Occlusion and missing data. Any technique whose projection direction differs from its observation direction (the entire triangulation family) has dead zones where “it can be projected onto but not seen, or seen but not projected onto”, manifesting as holes in the depth map; steep walls, deep holes, and reflective flyspecks also produce invalid data. Whether and how to fill holes (conservatively flag as invalid, or interpolate from the neighborhood) is an engineering decision that affects the credibility of later measurements, and the system’s handling is left to Chapter 38. The multi-view stitching needed when a single view is not enough — unifying point clouds from different viewpoints into one coordinate system — is borne by the registration algorithm of Chapter 39.
The language of accuracy. Chapter 20 distinguished repeatability from accuracy: the former is the spread of repeated measurements of the same workpiece, the latter the degree to which the measurement mean deviates from the true value. This methodology carries over to 3D unchanged — to evaluate a 3D sensor, repeatedly measure a standard gauge block or step gauge: repeatability corresponds to height noise (often expressed as a Z-direction σ), and accuracy is verified against a traceable standard artifact. A vendor’s nominal “resolution” is often the minimum resolvable quantity under ideal conditions, which is a different matter from the repeatability you can achieve with your workpiece at your site — and this is exactly the reason for the “trial verification” in the case below.
Industry Case: A Step-Height Measurement Selection Journey
A structural-parts plant required a 0.02 mm step-height judgment on the assembled metal middle frame, and the integrator’s selection went three rounds. The first round chose stereo vision on cost: it was the cheapest, but the middle-frame surface was uniform and textureless, stereo matching failed over large areas, and the depth map had holes exactly at the critical seam — eliminated outright. The second round switched to structured light: the diffuse regions gave beautiful data, but near the seam was precisely a highly reflective chamfer, the phase was stained into patches of noise at the glare, and the 0.02 mm judgment line drowned in 0.05 mm of noise. The third round tried laser triangulation: a short-wavelength narrowband laser paired with a filter was chosen, the incidence angle was tuned to avoid the specular reflection’s main lobe, and a shroud was added to isolate the shop’s overhead lights; a step gauge measured a Z-direction repeatability of 3 μm in practice, ample margin, and the solution went live. The project manager’s takeaway at the postmortem: selection is not scoring against a spec sheet but eliminating one by one along “surface → accuracy → cycle time”; and, trial verification always precedes purchase — the spec sheets of the first two rounds both “met the requirements”.
30.5 Summary
- A 2D image is a projection of the three-dimensional world, and height information is lost in principle; tasks where “the quantity being measured is height” — coplanarity, step height, volume, scratch depth — must use 3D imaging.
- 3D data has three forms — depth map (range image, 2.5D, one height per pixel), point cloud, and mesh — convertible into one another; most inline inspection is fine with single-view 2.5D, while true 3D requires multi-view stitching, costed separately.
- The six techniques share one logic: using a known geometric or optical constraint to turn height into a measurable image quantity — disparity (stereo), phase (structured light), line displacement (laser triangulation), shading → normal (photometric stereo), reflected phase → slope (PMD), sharpness (focus methods); the image quantity also determines each one’s failure mode.
- Selection eliminates in three cuts along “surface type → accuracy and field of view → cycle time and motion pattern” (see the selection table in this section): full-field diffuse goes to structured light, inline scanning to laser triangulation, specular to PMD, micro-relief to photometric stereo, microscopic accuracy to confocal, and large-scene low-cost to stereo; candidate solutions must be trial-verified before purchase.
- All 3D systems share three things in common: calibration sets the accuracy ceiling (Chapter 5), occlusion and missing data need an explicit handling strategy (Chapter 38, Chapter 39), and accuracy assessment follows the language of repeatability and accuracy (Chapter 20).
The next six chapters unfold one by one in the order of the table: Chapter 31 covers epipolar geometry and stereo matching, Chapter 32 covers phase shifting and unwrapping, Chapter 33 covers light-plane calibration and profile stitching, Chapter 34 covers normal solving and integration, Chapter 35 covers specular fringe reflection, and Chapter 36 covers sharpness evaluation and the focus stack. For a systematic treatment of 3D imaging, see further the chapters on 3D imaging and reconstruction in the book by Steger et al. (Steger, Ulrich, and Wiedemann 2018); Szeliski’s computer vision textbook (Szeliski 2022) offers a unified view of passive and active 3D methods from the perspective of multi-view geometry and depth estimation.