Bogu-Wenjin: A cultural heritage domain multimodal large model enhanced by knowledge graphs

ZHAO Wanqing; XU Chaoyang; XIE Zhiwei; ZHANG Shaobo; ZHANG Xiaodan; PENG Jinye

doi:10.16152/j.cnki.xdxbzr.2025-06-006

您当前的位置：

首页 >

文章列表页 >

Bogu-Wenjin: A cultural heritage domain multimodal large model enhanced by knowledge graphs

更新时间：2026-01-02

- Bogu-Wenjin: A cultural heritage domain multimodal large model enhanced by knowledge graphs
- Journal of Northwest University (Natural Science Edition) Vol. 55, Issue 6, Pages: 1267-1284(2025)
- 作者机构：
  
  1.西北大学　电子信息学院〔人工智能学院〕，陕西　西安　710127
  2.陕西省高等学校生成式人工智能与混合现实重点实验室，陕西　西安　710127
- 作者简介：
- 基金信息：
- DOI：10.16152/j.cnki.xdxbzr.2025-06-006
  CLC： TP391.4
- Received：07 July 2025，
  
  Published：25 December 2025
- 稿件说明：
移动端阅览
ZHAO Wanqing, XU Chaoyang, XIE Zhiwei, et al. Bogu-Wenjin: A cultural heritage domain multimodal large model enhanced by knowledge graphs[J]. Journal of Northwest University (Natural Science Edition), 2025, 55(6): 1267-1284.
DOI：

ZHAO Wanqing, XU Chaoyang, XIE Zhiwei, et al. Bogu-Wenjin: A cultural heritage domain multimodal large model enhanced by knowledge graphs[J]. Journal of Northwest University (Natural Science Edition), 2025, 55(6): 1267-1284. DOI： 10.16152/j.cnki.xdxbzr.2025-06-006.

摘要

近年来，大语言模型（LLMs）和多模态大模型（MLMs）在自然语言处理和多模态内容理解方面取得了显著成就，然而，这些通用模型在处理文化遗产相关任务时存在明显缺陷，如对领域专业术语的理解存在偏差、缺乏文化历史背景导致回答不够深入以及知识幻觉等问题，使得输出结果难以满足实际需求。针对这些挑战，首次提出了面向文化遗产领域的多模态大模型——“博古问津”。首先，设计半自动化策略构建大规模的多模态文化遗产数据集并形成多模态知识图谱；然后，利用构建的数据集对通用大模型进行图文对齐和指令微调两阶段训练，以适应文化遗产领域的特定需求；此外，还引入了知识图谱作为辅助知识库，通过图文检索和关系检索策略，有效提升了模型在文化遗产领域问答任务上的可信度和可解释性。实验结果表明，“博古问津”在文物图像描述、属性问题解答及关系问题理解等多个方面表现优异，相较于通用多模态大模型，对复杂文化内容的理解和回答能力提升效果显著，分别在文物图像描述、文物属性问题和文物关系问题3个不同任务的综合分值上高出次优模型21.4%、53%和20.6%。

Abstract

In recent years

large language models (LLMs) and multimodal large models (MLMs) have made significant achievements in natural language processing and multimodal content understanding. However

these general-purpose models have obvious shortcomings when dealing with tasks related to cultural heritage

such as biased understanding of domain-specific terminology

lack of cultural and historical background leading to superficial answers

and knowledge hallucination issues

making it difficult for the results to meet actual needs. In response to these challenges

this paper first proposes a multimodal large model oriented towards the field of cultural heritage: Bogu-Wenjin. This study first designs a semi-automated strategy to construct a large-scale multimodal cultural heritage dataset and forms a multimodal knowledge graph. Using the constructed dataset

the general large model is trained in two stages: image-text alignment and instruction fine-tuning

to adapt to the specific needs of the cultural heritage field. In addition

a knowledge graph is introduced as an auxiliary knowledge base

and the credibility and interpretability of the model in the field of cultural heritage Q&A tasks are effectively improved through graph-text retrieval and relationship retrieval strategies. Experimental results show that Bogu-Wenjin performs excellently in various aspects such as artifact image description

attribute question answering

and relationship question understanding. Compared with general multimodal large models

it significantly improves the ability to understand and answer complex cultural content

with a comprehensive score increase of 21.4%

53% and 20.6% in artifact image description

artifact attribute questions

and artifact relationship questions respectively over the second-best model.

关键词

Keywords

references

KASNECI E , SESSLER K , KÜCHEMANN S , et al . ChatGPT for good? On opportunities and challenges of large language models for education [J ] . Learning and Individual Differences , 2023 , 103 : 102274 .

NAVEED H , KHAN A U , QIU S , et al . A comprehensive overview of large languagemodels [EB/OL ] . ( 2023-11-27 ) [ 2024-12-20 ] . https://arxiv.org/abs/2307.06435 https://arxiv.org/abs/2307.06435 .

CHANG Y P , WANG X , WANG J D , et al . A survey on evaluation of large language models [J ] . ACM Transactions on Intelligent Systems and Technology , 2024 , 15 ( 3 ): 1 - 45 .

KOE HWEE NGA J , WAI MUN S . The influence of MLM companies and agent attributes on the willingness to undertake multilevel marketing as a career option among youth [J ] . Journal of Research in Interactive Marketing , 2011 , 5 ( 1 ): 50 - 70 .

YIN S K , FU C Y , ZHAO S R , et al . A survey on multimodal large language models [J ] . National Science Review , 2024 , 11 ( 12 ): nwae403 .

ZHANG D Z , YU Y H , DONG J H , et al . MM-LLMs: Recent advances in multimodal large language models [EB/OL ] . ( 2024-05-28 ) [ 2024-12-20 ] . https://arxiv.org/abs/2401.13601 https://arxiv.org/abs/2401.13601 .

CUI C , MA Y S , CAO X , et al . A survey on multimodal large language models for autonomous driving [C ] // 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) . Waikoloa, HI, USA : IEEE , 2024 : 958 - 979 .

LIPKOVA J , CHEN R J , CHEN B W , et al . Artificial intelligence for multimodal data integration in oncology [J ] . Cancer Cell , 2022 , 40 ( 10 ): 1095 - 1110 .

SHARMA K , GIANNAKOS M . Multimodal data capabilities for learning: What can multimodal data tell us about learning? [J ] . British Journal of Educational Technology , 2020 , 51 ( 5 ): 1450 - 1484 .

OPENAI , ACHIAM J , ADLER S , et al . GPT-4 technical report [EB/OL ] . ( 2024-03-04 ) [ 2024-12-20 ] . https://arxiv.org/abs/2303.08774 https://arxiv.org/abs/2303.08774 .

LIU H T , LI C Y , WU Q Y , et al . Visual instruction tuning [EB/OL ] . ( 2024-11-11 ) [ 2024-12-20 ] . https://arxiv.org/abs/2304.08485 https://arxiv.org/abs/2304.08485 .

ASKELL A , BAI Y T , CHEN A N , et al . A general language assistant as a laboratory for alignment [EB/OL ] . ( 2021-11-09 ) [ 2024-12-20 ] . https://arxiv.org/abs/2112.00861 https://arxiv.org/abs/2112.00861 .

GAN Z , LI L J , LI C Y , et al . Vision-language pre-training: Basics, recent advances, and future trends [J ] . Foundations and Trends ® in Computer Graphics and Vision , 2022 , 14 ( 3/4 ): 163 - 352 .

BHATIA G , NAGOUDI E M B , CAVUSOGLU H , et al . FinTral: A family of GPT-4 level multimodal financial large language models [EB/OL ] . ( 2024-06-14 ) [ 2024-12-20 ] . https://arxiv.org/abs/2402.10986 https://arxiv.org/abs/2402.10986 .

LI C Y , WONG C , ZHANG S , et al . LLaVA-med: Training a large language-and-vision assistant for biomedicine in one day [EB/OL ] . ( 2023-06-01 ) [ 2024-12-20 ] . https://arxiv.org/abs/2306.00890 https://arxiv.org/abs/2306.00890 .

程启玥 . 我国文化遗产领域数字人文研究热点与趋势探究 [J ] . 现代信息科技 , 2024 , 8 ( 14 ): 106 - 111 .

CHEN Q Y . Research hotspots and trends of digital humanities in the field of cultural heritage in China [J ] . Modern Information Technology , 2024 , 8 ( 14 ): 106 - 111 .

MITTAL R , GARG A . Text extraction using OCR: A systematic review [C ] // 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA) . Coimbatore, India : IEEE , 2020 : 357 - 362 .

徐增林 , 盛泳潘 , 贺丽荣 , 等 . 知识图谱技术综述 [J ] . 电子科技大学学报 , 2016 , 45 ( 4 ): 589 - 606 .

XU Z L , SHENG Y P , HE L R , et al . A survey on knowledge graph techniques [J ] . Journal of University of Electronic Science and Technology of China , 2016 , 45 ( 4 ): 589 - 606 .

杜鹏飞 , 李小勇 , 高雅丽 . 多模态视觉语言表征学习研究综述 [J ] . 软件学报 , 2021 , 32 ( 2 ): 327 - 348 .

DU P F , LI X Y , GAO Y L . Survey on multimodal visual-linguistic representation learning [J ] . Journal of Software , 2021 , 32 ( 2 ): 327 - 348 .

DEVLIN J , CHANG M W , LEE K , et al . BERT: Pre-training of deep bidirectional transformers for language understanding [EB/OL ] . ( 2019-05-24 ) [ 2024-12-20 ] . https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805 .

BROWN T B , MANN B , RYDER N , et al . Language models are few-shot learners [C ] // Proceedings of the 34th International Conference on Neural Information Processing Systems . Vancouver, BC, Canada : ACM , 2020 : 1877 - 1901 .

TOUVRON H , LAVRIL T , IZACARD G , et al . LLaMA: Open and efficient foundation language models [EB/OL ] . ( 2019-05-24 ) [ 2024-12-20 ] . https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971 .

BAI J Z , BAI S , CHU Y F , et al . Qwen technical report [EB/OL ] . ( 2019-05-24 ) [ 2024-12-20 ] . https://arxiv.org/abs/2309.16609 https://arxiv.org/abs/2309.16609 .

YANG Z , LI L , LIN K , et al . The dawn of LMMs: Preliminary explorations with GPT-4V (ision) [EB/OL ] . ( 2023-10-11 ) [ 2024-12-20 ] . https://arxiv.org/abs/2309.17421 https://arxiv.org/abs/2309.17421 .

RADFORD A , KIM J W , HALLACY C , et al . Learning transferable visual models from natural language supervision [EB/OL ] . ( 2021-02-26 ) [ 2024-12-20 ] . https://arxiv.org/abs/2103.00020 https://arxiv.org/abs/2103.00020 .

ZHU D Y , CHEN J , SHEN X Q , et al . Minigpt-4: Enhancing vision-language understanding with advanced large language models [EB/OL ] . ( 2023-04-20 ) [ 2024-12-20 ] . https://arxiv.org/abs/2304.10592 https://arxiv.org/abs/2304.10592 .

CHIANG W L , LI Z H , LIN Z , et al . Vicuna: An open-source chatbot impressing GPT-4 with 90% * ChatGPT quality [EB/OL ] . ( 2023-03-30 ) [ 2024-12-20 ] . https://vicuna.lmsys.org https://vicuna.lmsys.org .

BAI J Z , BAI S , YANG S S , et al . Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond [EB/OL ] . ( 2023-10-13 ) [ 2024-12-20 ] . https://arxiv.org/abs/2308.12966 https://arxiv.org/abs/2308.12966 .

ZENG A H , XU B , WANG B W , et al . ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools [EB/OL ] . ( 2024-06-18 ) [ 2024-12-20 ] . https://arxiv.org/abs/2406.12793 https://arxiv.org/abs/2406.12793 .

HARTSOCK I , RASOOL G . Vision-language models for medical report generation and visual question answering: A review [EB/OL ] . ( 2024-04-15 ) [ 2024-12-20 ] . https://arxiv.org/abs/2403.02469 https://arxiv.org/abs/2403.02469 .

HAN T Y , LIAN Q , PAN R , et al . The instinctive bias: Spurious images lead to illusion in MLLMs [C ] // Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing . Miami, FLorida, USA : ACL , 2024 : 16163 - 16177 .

HU E J , SHEN Y , WALLIS P , et al . LoRA: Low-rank adaptation of large language models [EB/OL ] . ( 2021-10-16 ) [ 2024-12-20 ] . https://arxiv.org/abs/2106.09685 https://arxiv.org/abs/2106.09685 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Analysis of the spatiotemporal distribution characteristics and natural influencing factors of cultural heritage in the Hexi Corridor

Review of research on digital protection and application of cultural heritage

Protection of cultural heritage in the new rural construction

Research on the tourism performance of heritage sites of spatial pattern in Guanzhong-tianshui Economic Zone

The experiential exploitation of heritage tourism destination: Taking the Yungang Grottoes as an example

Related Author

YU Yi

ZHANG Wenqian

WEN Chao

XU Weimin

SHEN Yan

GENG Guohua

ZHOU Mingquan

ZHANG Yuhe

Related Institution

Museum of Hebei Normal University

School of Economics and Management, Changsha University

Intelligent Interaction and Information Art Research Center, Northwest University

School of Art, Northwest University

School of Cultural Heritage, Northwest University

⁰