3D model hole filling in cultural relics' facial reconstruction fusion diffusion model

WANG Zhengrong; LIU Xinda; ZHOU Mingquan

doi:10.16152/j.cnki.xdxbzr.2025-01-018

您当前的位置：

首页 >

文章列表页 >

3D model hole filling in cultural relics' facial reconstruction fusion diffusion model

The digital protection of cultural heritage | 更新时间：2025-02-08

- 3D model hole filling in cultural relics' facial reconstruction fusion diffusion model
- Journal of Northwest University (Natural Science Edition) Vol. 55, Issue 1, Pages: 213-222(2025)
- 作者机构：
  
  1.西北大学　文化遗产数字化国家地方联合工程研究中心，陕西　西安　 710127
  2.西北大学　可视化技术研究所，陕西　西安　 710127
- 作者简介：
- 基金信息：
- DOI：10.16152/j.cnki.xdxbzr.2025-01-018
  CLC： TP391
- Published：25 February 2025，
  
  Received：24 September 2024，
- 稿件说明：
移动端阅览
WANG ZHENGRONG, LIU XINDA, ZHOU MINGQUAN. 3D model hole filling in cultural relics' facial reconstruction fusion diffusion model. [J]. Journal of northwest university (natural science edition), 2025, 55(1): 213-222.
DOI：

WANG ZHENGRONG, LIU XINDA, ZHOU MINGQUAN. 3D model hole filling in cultural relics' facial reconstruction fusion diffusion model. [J]. Journal of northwest university (natural science edition), 2025, 55(1): 213-222. DOI： 10.16152/j.cnki.xdxbzr.2025-01-018.

摘要

在三维几何建模领域，孔洞指三维模型表面的不完整性，通常产生在数据采集、处理或转换过程中，这些孔洞的存在可能会对模型的几何完整性和视觉质量造成显著影响。在文化遗产保护领域，传统的孔洞修复技术侧重于几何表面的修复，而忽略了纹理信息的修复，纹理信息对于恢复文物的真实感和材质特性至关重要。文中提出了一种结合几何和纹理修复的扩散模型方法，特别适用于文物面部模型的修复。由于缺乏文物面部模型的基准真值，使用相似的残缺黄种人面部模型进行评估，鉴于公开可用的黄种人面部数据集较为稀缺，构建了一个黄种人面部数据集，包含约20 000张黄种人面部图像、对应的三维模型及其纹理，以及渲染得到的高保真面部图像。首先，在构建的数据集上进行测试，以验证所提方法的有效性。结果显示，与现有基准方法相比，该方法在面部图像修复方面取得了显著改进，并且实现了对面部三维模型几何表面和纹理信息的同步修复。其次，将所提出的修复技术应用于文化遗产中的文物面部模型，通过实验验证该方法在文物面部模型的几何和纹理修复方面均展现出了优异的性能。研究结果为文物面部模型的数字化修复提供了一种有效的技术手段，有助于提高文物保护和数字化展示的质量。

Abstract

In the field of 3D geometric modeling

a hole refers to the incompleteness on the surface of a 3D model

often arising during data acquisition

processing

or transformation. The presence of such holes can significantly impact the geometric integrity and visual quality of the model. In the domain of cultural heritage preservation

traditional hole-filling techniques focus on the repair of geometric surfaces

neglecting the restoration of texture information

which is crucial for recovering the authenticity and material characteristics of artifacts. This paper proposes a diffusion model method that combines geometric and texture repair

particularly suitable for the restoration of facial models of cultural relics. Due to the absence of ground truth for facial models of cultural relics

this study uses similar incomplete Asian facial models for evaluation. Given the scarcity of publicly available Asian facial datasets

this paper constructs an Asian facial dataset comprising approximately 20 000 Asian facial images

corresponding 3D models and textures

as well as high-fidelity facial images rendered from the models. Experiments are first conducted on the constructed dataset to validate the effectiveness of the proposed method. The results show that compared to existing benchmark methods

our method achieves significant improvements in facial imageinpainting and simultaneously repairs the geometric surfaces and texture information of the facial 3D models. Subsequently

the proposed repair technique is applied to facial models of cultural relics

and experiments confirm its excellent performance in both geometric and texture repair of the relic facial models. These findings provide an effective technical means for the digital restoration of facial models of cultural relics

contributing to the enhancement of the quality of cultural heritage preservation and digital exhibition.

关键词

三维模型孔洞修补文物面部修复扩散模型图像修复

Keywords

3D model hole repairfacial models of cultural relicsdiffusion modelimage inpainting

references

LIEPA P. Filling holes in meshes[C]//In Proceedings of the 2003 Eurographics/ACM SIGGRAPH Symposium on Geometry Processing. Aachen: ACM, 2003: 200-205.

温佩芝, 雷永庆, 孙梦龙. 三维重建网格模型的缺陷孔洞识别与修复方法[J]. 计算机应用研究, 2020, 37(4): 1234-1238.

WEN P Z, LEI Y Q, SUN M L. Defect and hole identification and repair methods for 3D reconstructed mesh models[J]. Application Research of Computers, 2020, 37(4): 1234-1238.

谢倩茹, 耿国华. 三维模型孔洞修补方法的研究[J]. 计算机应用研究, 2013, 30(10): 3175-3177.

XIE Q R, GENG G H. Research on the method of repairing holes in 3D models[J]. Application Research of Computers, 2013, 30(10): 3175-3177.

DAVIS J, MARSCHNER S R, GARR M, et al. Filling holes in complex surfaces using volumetric diffusion[C]//In Proceedings of First International Symposium on 3D Data Processing Visualization and Transmission. Padua: IEEE, 2002: 428-441.

BISCHOFF S, KOBBELT L. Structure preserving CAD model repair[J]. Computer Graphics Forum, 2005, 24(3): 527-536.

WANG X, LIU X, LU L, et al. Automatic hole-filling of CAD models with feature-preserving[J]. Computers & Graphics, 2012, 36(2): 101-110.

刘震, 王艳宾, 白丽丽, 等. 曲面细节特征保持的三维模型孔洞修复方法[J]. 计算机辅助设计与图形学学报, 2016, 28(12): 2052-2059.

LIU Z, WANG Y B, BAI L L, et al. 3D model hole repair method for preserving surface detail features[J]. Journal of Computer Aided Design and Graphics, 2016, 28(12): 2052-2059.

王鹏飞, 徐敏峰, 辛士庆, 等. 鲁棒的水密流形网格修复[J]. 计算机辅助设计与图形学学报, 2024, 36(7): 1047-1056.

WANG P F, XU M F, XIN S Q, et al. Robust watertight manifold mesh repair[J]. Journal of Computer-Aided Design and Computer Graphics, 2024, 36(7): 1047-1056.

HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models[J]. Advances in Neural Information Processing Systems, 2020, 33: 6840-6851.

DHARIWAL P, NICHOL A. Diffusion models beat GANs on image synthesis[J]. Advances in Neural Information Processing Systems, 2021, 34: 8780-8794.

RAMESH A, DHARIWAL P, NICHOL A, et al. Hierarchical text-conditional image generation with CLIP latents[EB/OL]. (2022-04-13) [2024-07-02]. https://arxiv.org/abs/2204.06125v1https://arxiv.org/abs/2204.06125v1.

彭进业, 余喆, 屈书毅, 等. 基于深度学习的图像修复方法研究综述[J]. 西北大学学报(自然科学版), 2023, 53(6): 943-963.

PENG J Y, YU Z, QU S Y, et al. A survey of image inpainting methods based on deep learning[J]. Journal of Northwest University (Natural Science Edition), 2023, 53(6): 943-963.

LUGMAYR A, DANELLJAN M, ROMERO A, et al. RePaint: Inpainting using denoising diffusion probabilistic models[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans: IEEE, 2022: 11451-11461.

ROUT L, PARULEKAR A, CARAMANIS C, et al. A theoretical justification for image inpainting using denoising diffusion probabilistic models[EB/OL]. (2023-02-02) [2024-07-02]. https://arxiv.org/abs/2302.01217v1https://arxiv.org/abs/2302.01217v1.

ZHANG G H, JI J B, ZHANG Y, et al. Towards coherent image inpainting using denoising diffusion implicit models[C]//Proceedings of the 40th International Conference on Machine Learning. Honolulu: ACM, 2023: 41164-41193.

YUE Z S, WANG J Y, LOY C C. Efficient diffusion model for image restoration by residual shifting[EB/OL]. (2024-03-12) [2024-07-02]. https://arxiv.org/abs/2403.07319v3https://arxiv.org/abs/2403.07319v3.

CORNEANU C, GADDE R, MARTINEZ A M. LatentPaint: Image inpainting in latent space with diffusion models[C]//2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Waikoloa: IEEE, 2024: 4322-4331.

DURRER A, WOLLEB J, BIEDER F, et al. Denoising diffusion models for 3D healthy brain tissue inpainting[C]//MICCAI Workshop on Deep Generative Models. Cham: Springer Nature Switzerland, 2024: 87-97.

SONG J M, MENG C L, ERMON S. Denoising diffusion implicit models[EB/OL]. (2020-10-06) [2024-07-02]. https://arxiv.org/abs/2010.02502v4https://arxiv.org/abs/2010.02502v4.

DEJA K, KUZINA A, TRZCINSKI T, et al. On analyzing generative and denoising capabilities of diffusion-based deep generative models[EB/OL]. (2022-05-31) [2024-07-02]. https://arxiv.org/abs/2206.00070v1https://arxiv.org/abs/2206.00070v1.

ZHANG K, LI Y W, ZUO W M, et al. Plug-and-play image restoration with deep denoiser prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(10): 6360-6376.

ZHU Y Z, ZHANG K, LIANG J Y, et al. Denoising diffusion models for plug-and-play image restoration[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Vancouver: IEEE, 2023: 1219-1229.

ALEXANDER O, ROGERS M, LAMBETH W, et al. The digital emily project: Photoreal facial modeling and animation[C]//ACM SIGGRAPH 2009 Courses. New Orleans: ACM, 2009: 1-15.

CAO X, CHEN Z, CHEN A P, et al. Sparse photometric 3D face reconstruction guided by morphable models[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 4635-4644.

MA W C, HAWKINS T, PEERS P, et al. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination[C]//Proceedings of the 18th Eurographics Conference on Rendering Techniques. Grenoble: ACM, 2007: 183-194.

DIB A, THÉBAULT C, AHN J, et al. Towards high fidelity monocular face reconstruction with rich reflectance using self-supervised learning and ray tracing[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal: IEEE, 2021: 12799-12809.

GECER B, PLOUMPIS S, KOTSIA I, et al. GANFIT: Generative adversarial network fitting for high fidelity 3D face reconstruction[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 1155-1164.

GECER B, PLOUMPIS S, KOTSIA I, et al. Fast-GANFIT: Generative adversarial network for high fidelity 3D face reconstruction[EB/OL]. (2021-05-16) [2024-07-02]. https://arxiv.org/abs/2105.07474v1https://arxiv.org/abs/2105.07474v1.

LATTAS A, MOSCHOGLOU S, GECER B, et al. AvatarMe: Realistically renderable 3D facial reconstruction "In-the-wild"[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle: IEEE, 2020: 760-769.

LEI B W, REN J Q, FENG M Y, et al. A hierarchical representation network for accurate and detailed face reconstruction from in-the-wild images[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver: IEEE, 2023: 394-403.

KARRAS T, LAINE S, AILA T M. A style-based generator architecture for generative adversarial networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 4401-4410.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

A survey of image inpainting methods based on deep learning

Related Author

PENG Jinye

YU Zhe

QU Shuyi

HU Qiyao

WANG Jun

Related Institution

School of Information Science and Technology, Northwest University

Shaanxi Province Silk Road Digital Protection and Inheritance of Cultural Heritage Collaborative Innovation Center

⁰