SN-CLPGAN: A method of style transfer in Chinese traditional landscape painting based on spectral normalization

HU Qiyao; LIU Qianlong; PENG Xianlin; ZHANG Xiang; PENG Shenglin; FAN Jianping

doi:10.16152/j.cnki.xdxbzr.2025-01-005

您当前的位置：

首页 >

文章列表页 >

SN-CLPGAN: A method of style transfer in Chinese traditional landscape painting based on spectral normalization

The digital protection of cultural heritage | 更新时间：2025-02-08

- SN-CLPGAN: A method of style transfer in Chinese traditional landscape painting based on spectral normalization
- Journal of Northwest University (Natural Science Edition) Vol. 55, Issue 1, Pages: 63-74(2025)
- 作者机构：
  
  1.西北大学　信息科学与技术学院，陕西　西安　 710127
  2.生成式人工智能与混合现实陕西省高等学校重点实验室，陕西　西安　 710127
  3.西北大学　网络与数据中心，陕西　西安　 710127
  4.西北大学　艺术学院，陕西　西安　 710127
  5.西北大学　新型网络智能信息服务国家地方联合工程研究中心，陕西　西安　 710127
- 作者简介：
- 基金信息：
- DOI：10.16152/j.cnki.xdxbzr.2025-01-005
  CLC： TP391.41
- Published：25 February 2025，
  
  Received：20 May 2024，
- 稿件说明：
移动端阅览
HU QIYAO, LIU QIANLONG, PENG XIANLIN, et al. SN-CLPGAN: A method of style transfer in Chinese traditional landscape painting based on spectral normalization. [J]. Journal of northwest university (natural science edition), 2025, 55(1): 63-74.
DOI：

HU QIYAO, LIU QIANLONG, PENG XIANLIN, et al. SN-CLPGAN: A method of style transfer in Chinese traditional landscape painting based on spectral normalization. [J]. Journal of northwest university (natural science edition), 2025, 55(1): 63-74. DOI： 10.16152/j.cnki.xdxbzr.2025-01-005.

摘要

中国传统山水画的风格迁移为文化遗产数字化保护与传承提供了新的路径，近年来，深度学习技术已实现了不同图像间的风格迁移，并展现出栩栩如生的效果。中国传统山水画的风格迁移旨在继承中国古代画家独特的绘画技巧，但存在3个缺陷：①缺乏高质量的中国传统山水画图像数据集；②忽略了中国传统山水画特有的技法和笔墨细节；③风格迁移效果与真实山水画有所差距。为了弥补上述缺陷，首先，创建了一个基于风格迁移的中国传统山水画数据集STCLP，包含4 281幅高质量的中国山水画以及自然景观图像，并提出了一种基于谱归一化的中国山水画风格迁移方法SN-CLPGAN。其次，提出了在生成器中使用融合反射填充层的残差稠密块（residual-in-residual dense block，RRDB）学习中国山水画独特的笔触和技法。接着，引入了多尺度结构相似性指数测量（multi-scale structural similarity index measure，MS-SSIM）损失以减少2幅图像之间的像素差异，使生成图像更接近传统绘画的色彩和颜料。最后，采用了融合谱归一化（spectral normalization，SN）的U-Net判别器增强图像纹理细节，并确保了模型训练过程的稳定性。大量实验验证了提出的方法在中国传统山水画风格迁移任务中的有效性和先进性。

Abstract

The style tranfer of Chinese landscape paintings offering new avenues for the digital preservation and inheritance of cultural heritage. In recent years

deep learning technologies have enabled style transfer between different images

achieving lifelike effects. Style transfer in Chinese landscape paintings aims to preserve the unique paintings skills of ancient Chinese painters

but faces three main challenges: ① The lack of high-quality datasets of traditional Chinese landscape paintings. ② The oversight of the unique techniques and ink details specific to traditional Chinese landscape paintings. ③ The gap between the style transfer outcomes and real landscape paintings. To address these deficiencies

this paper first introduces a Chinese landscape paintings dataset for style transfer

STCLP

which contains 4281 high-quality images of Chinese landscape paintings and natural landscapes. A generative adversarial network of style transfer in Chinese landscape painting based on spectral normalization is proposed

termed SN-CLPGAN. Additionally

it introduces the use of residual-in-residual dense blocks (RRDB) with reflect padding layers in the generator to learn the distinctive brushstrokes and techniques of Chinese landscape paintings. Furthermore

it employs the multi-scale structural similarity index measure (MS-SSIM) loss to minimize pixel-level differences between images

thereby producing images closer to traditional paintings in terms of color and pigmentation. Finally

the U-Net discriminator fused with SN is utilized to enhance the textural details of images

ensuring the stability of the model training process. Extensive experiments validate the effectiveness and advancement of the proposed method in the task of style transfer for Chinese landscape paintings.

关键词

风格迁移人工智能艺术中国传统山水画生成对抗网络

Keywords

style transferartificial intelligence artChinese landscape paintingsgenerative adversarial network

references

彭进业, 余喆, 屈书毅, 等. 基于深度学习的图像修复方法研究综述[J]. 西北大学学报(自然科学版), 2023, 53(6): 943-963.

PENG J Y, YU Z, QU S Y, et al. A survey of image inpainting methods based on deep learning[J]. Journal of Northwest University (Natural Science Edition), 2023, 53(6): 943-963.

胡琦瑶, 杨皓文, 王佳欣, 等. 基于弱监督深度学习的图像检索技术研究[J]. 西北大学学报(自然科学版), 2020, 50(5): 793-801.

HU Q Y, YANG H W, WANG J X, et al. Research on image retrieval based on weakly-supervised deep learning[J]. Journal of Northwest University (Natural Science Edition), 2020, 50(5): 793-801.

GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[EB/OL]. (2014-06-10) [2024-04-10]. http://arxiv.org/abs/1406.2661http://arxiv.org/abs/1406.2661.

ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2242-2251.

ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5967-5976.

ZHENG Z T, LIU J Y, ZHENG N N. P2-GAN: Efficient stroke style transfer using single style image[J]. IEEE Transactions on Multimedia, 2023, 25: 6000-6012.

WANG Z Z, ZHANG Z J, ZHAO L, et al. AesUST: Towards aesthetic-enhanced universal style transfer[C]//Proceedings of the 30th ACM International Conference on Multimedia. Lisboa: ACM, 2022: 1095-1106.

WANG X T, YU K, WU S X, et al. ESRGAN: Enhanced super-resolution generative adversarial networks[C]//Computer Vision-ECCV 2018 Workshops. Cham: Springer International Publishing, 2019: 63-79.

WANG X T, XIE L B, DONG C, et al. Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data[C]//2021 IEEE/CVF International Conference on Computer Vision Workshops. Montreal: IEEE, 2021: 1905-1914.

RONNEBERGER O, FISCHER P, BROX T. U-Net: Convolutional networks for biomedical image segmentation[M]//Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015. Cham: Springer International Publishing, 2015: 234-241.

MIYATO T, KATAOKA T, KOYAMA M, et al. Spectral normalization for generative adversarial networks[EB/OL]. (2018-02-16) [2024-04-10]. https://arxiv.org/abs/1802.05957https://arxiv.org/abs/1802.05957.

ZHAO H, GALLO O, FROSIO I, et al. Loss functions for image restoration with neural networks[J]. IEEE Transactions on Computational Imaging, 2017, 3(1): 47-57.

孟宪佳, 傅利平, 刘栋, 等. 高性能计算发展现状及其在文化遗产保护中的应用展望[J]. 西北大学学报(自然科学版), 2021, 51(5): 807-815.

MENG X J, FU L P, LIU D, et al. Development status of high performance computing and its application prospect in cultural heritage protection[J]. Journal of Northwest University (Natural Science Edition), 2021, 51(5): 807-815.

GATYS L A, ECKER A S, BETHGE M, et al. Texture synthesis using convolutional neural networks[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. Montreal: ACM, 2015: 262-270.

CHEN R F, HUANG W B, HUANG B H, et al. Reusing discriminators for encoding: Towards unsupervised image-to-image translation[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 8165-8174.

YI Z L, ZHANG H, TAN P, et al. DualGAN: Unsupervised dual learning for image-to-image translation[C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2868-2876.

KIM J, KIM M, KANG H, et al. U-GAT-IT: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation[EB/OL]. (2020-04-08) [2024-04-10]. https://arxiv.org/abs/1907.10830v4https://arxiv.org/abs/1907.10830v4.

张美玉, 刘跃辉, 秦绪佳, 等. 基于拉普拉斯算子抑制伪影的神经风格迁移方法[J]. 计算机科学, 2020, 47(S2): 209-214.

ZHANG M Y, LIU Y H, QIN X J, et al. Neural style transfer method based on Laplacian operator to suppress artifacts[J]. Computer Science, 2020, 47(S2): 209-214.

茹超, 周延, 陈晓璇, 等. 一种面向文本图像的颜色迁移算法[J]. 西北大学学报(自然科学版), 2017, 47(6): 815-821.

RU C, ZHOU Y, CHEN X X, et al. A color transfer algorithm for text image[J]. Journal of Northwest University (Natural Science Edition), 2017, 47(6): 815-821.

PENG X L, PENG S L, HU Q Y, et al. Contour-enhanced CycleGAN framework for style transfer from scenery photos to Chinese landscape paintings[J]. Neural Computing and Applications, 2022, 34(20): 18075-18096.

ZHANG F Q, GAO H M, LAI Y P. Detail-preserving CycleGAN-AdaIN framework for image-to-ink painting translation[J]. IEEE Access, 2020, 8: 132002-132011.

IOFFE S, SZEGEDY C, PARANHOS L, et al. Batch normalization: Accelerating deep network training by reducing internal covariate shift[EB/OL]. (2015-03-02) [2024-04-10]. https://arxiv.org/abs/1502.03167v3https://arxiv.org/abs/1502.03167v3.

LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 1132-1140.

NAH S, KIM T H, LEE K M. Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 257-265.

WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: From error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.

KINGMA D P, BA J, HAMMAD M M. Adam: A method for stochastic optimization[EB/OL]. (2017-01-30) [2024-04-10]. https://arxiv.org/abs/1412.6980v9https://arxiv.org/abs/1412.6980v9.

CHONG M J, FORSYTH D. Effectively unbiased FID and inception score and where to find them[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 6069-6078.

TALEBI H, MILANFAR P. NIMA: Neural image assessment[J]. IEEE Transactions on Image Processing, 2018, 27(8): 3998-4011.

孙满利, 张景科. 文物保护学的理论探讨[J]. 西北大学学报(自然科学版), 2022, 52(2): 192-198.

SUN M L, ZHANG J K. Theoretical discussion on conservation of cultural heritages[J]. Journal of Northwest University (Natural Science Edition), 2022, 52(2): 192-198.

HO J, JAIN A, ABBEEL P, et al. Denoising diffusion probabilistic models[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver: ACM, 2020: 6840-6851.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Skull-to-skin generation method based on multi-level bottleneck attention module

Breast MRI image generation method based on generative adversarial network

MeGAN: Multi-task enhanced generative adversarial network for image synthesis

Related Author

GENG Guohua

LIU Xiaoning

LIANG Zenglei

JIANG Jiaqi

JIANG Wenkai

WANG Jie

FENG Jun

CHEN Baoying

Related Institution

College of Information Science and Technology, Northwest University

School of Computer Science and Technology, Xi'an University of Posts and Telecommunications

Shaanxi Key Laboratory of Network Data Analysis and Intelligent Processing, Xi'an University of Post & Telecommunications

Xi'an Key Laboratory of Big Data and Intelligent Computing

Imaging Diagnosis and Treatment Center, Xi'an International Medical Center Hospital

⁰