Deep learning-enabled 3D multimodal fusion of cone-beam CT and intraoral mesh scans for clinically applicable tooth-bone reconstruction

The problem

Orthodontic treatment and implant surgery are planned on 3D models of the teeth and the bone that holds them. Two kinds of scan feed these models, and each sees only part of the picture. Cone-beam CT (CBCT) is an X-ray scan that shows whole teeth, crowns and roots, inside the jaw bone. But CBCT measurements can be up to 6.9% smaller than the real objects, depending on resolution, and CBCT often fails to capture the biting surfaces accurately because of dense enamel, dental restorations, implants and orthodontic appliances. An intraoral scanner (IOS) projects light onto the teeth and records a digital impression as a surface mesh. Its measurements shrink by as little as 0.9%, making it the most accurate of the 3D digital modalities, and it captures the bite and tooth positions well, but it shows only the crowns and gums, nothing below the gum line.

Combining the two gives what clinicians need: precise crowns on the correct roots, inside the surrounding bone. Doing this by hand is slow. Both scans must first be segmented, which is hard because anatomy varies from patient to patient. The CBCT usually shows the upper and lower jaws together, often in a bite where teeth touch, while the IOS records each jaw separately. Experienced experts need at least five hours per patient, even with interactive software and semi-automatic tools, and a fully automatic, clinically applicable solution had yet to be developed.

How it works

Deep Dental Multimodal Fusion (DDMF) is an automatic pipeline with three modules: segment the CBCT, segment the intraoral scan, then fuse the two. The fused model takes the crowns from the IOS and the roots and bone from the CBCT.

1. Segment the CBCT. TSTNet, a network built on the Swin Transformer, labels every pixel of each 2D CBCT slice as tooth, alveolar bone or background, and 3D meshes of the teeth and bone are reconstructed from the result. Training needs labels on only about 20 of the 300–600 slices in a scan. An extra training head, tooth error calibration (TEC), corrects error-prone pixels at the tooth–bone boundary; it is removed at inference, so it adds no inference time.
2. Segment the intraoral scan. IOSNet labels each face of the IOS mesh as gum or as one of 32 teeth, numbered in the FDI system. Two new training losses help: a centroid loss keeps the predicted centre of each tooth in place, so that two teeth are not recognised as one (or one as two), and a boundary loss sharpens tooth–tooth and tooth–gum borders. The gum is then separated, leaving numbered crowns.
3. Split the jaws in the CBCT. A point-curvature feature, the average angle between a vertex’s surface normal and those of its neighbours, marks the boundaries where teeth meet. Removing the highest-curvature vertices separates the individual teeth; the removed vertices are merged back, and the teeth are grouped into upper and lower jaws.
4. Register. The IOS crowns of each jaw are aligned with the matching CBCT jaw in two stages: a global registration (RANSAC on FPFH features) gives a rough initial alignment, and point-to-plane ICP at three scales refines it.
5. Replace the crowns and fuse. CBCT crown points that lie close to the IOS crown are removed and replaced with the IOS points, and Poisson reconstruction turns the merged points into one mesh. The FDI tooth numbers come from the IOS, so the fused teeth are identified individually without instance-level annotation of the CBCT slices.

Results

The study used a large real-world dataset: 503 patients with both CBCT and intraoral scans, plus 28,559 further intraoral scans, collected from hospitals and clinics in 25 provinces of China from 2018 to 2021. The ground-truth fused models were made by a committee of human experts and had been used successfully in clinical practice. The errors were small and consistent across tooth types; the first and second molars showed slightly larger errors, mainly because they are larger. Since CBCT resolution ranged from 0.125 to 0.5 mm, an error of one to two pixels is acceptable for many clinical applications, as validated by human experts and a clinical applicability test. On 20 cases, DDMF also had lower errors than reconstruction from CBCT alone (Fig. 4C–E). The paper reports no other fusion baseline, as no systematic solution existed for this task.

Each segmentation module outperformed the compared networks. TSTNet reached a Dice coefficient of 93.99% in 5-fold cross-validation, with 50 test patients per fold, ahead of UNet, UNet++, the standard Swin Transformer and the other baselines (Table 1). IOSNet (trained on 28,000 scans) reached a mean intersection over union (mIoU) of 93.70% on lower-jaw scans and 95.70% on upper-jaw scans, 1.77 and 3.52 points above DCNet, the strongest baseline (Table 2).

The framework has been integrated into clinical software that assists orthodontic treatment planning. The authors also name its limits. The jaw splitting and registration steps may fail in 6–10% of cases, which then need manual correction, and segmentation may still be error-prone in extremely complicated cases, such as patients with metal artefacts or root canal therapy. The method has been evaluated mainly from an algorithmic perspective, and the authors call for rigorous multicentre clinical trials.