“博古问津”：知识图谱增强的文化遗产领域多模态大模型

赵万青; 徐朝阳; 谢智伟; 张少博; 张晓丹; 彭进业

doi:10.16152/j.cnki.xdxbzr.2025-06-006

您当前的位置：

首页 >

文章列表页 >

“博古问津”：知识图谱增强的文化遗产领域多模态大模型

青年编委专刊 | 更新时间：2026-01-02

- “博古问津”：知识图谱增强的文化遗产领域多模态大模型
- Bogu-Wenjin: A cultural heritage domain multimodal large model enhanced by knowledge graphs
- 西北大学学报（自然科学版） 2025年55卷第6期页码：1267-1284
- 作者机构：
  
  1.西北大学　电子信息学院〔人工智能学院〕，陕西　西安　710127
  2.陕西省高等学校生成式人工智能与混合现实重点实验室，陕西　西安　710127
- 作者简介：
  
  [ "赵万青，工学博士，西北大学电子信息学院（人工智能学院）教授，博士生导师。中国图象图形学学会数字文化遗产专委会委员、陕西计算机学会理事、陕西图象图形学学会理事、中国计算机学会会员。主要研究方向为智能媒体检索、机器学习和计算机视觉。已发表学术论文四十余篇，其中以第一作者或通信作者在人工智能国际顶级期刊会议（IEEE Transactions on Image Processing、CVPR、AAAI、IJCAI）上发表多篇学术论文并做大会报告，作为主持人或骨干成员参与多项国家自然科学基金和国家重点研发计划等项目，授权发明专利及软件著作权十余项，两次获陕西省高校科学技术奖一等奖，日本大阪大学访问学者，担任人工智能国际会议IJCAI2021程序委员会成员，国际会议CVPR、AAAI、IJCAI审稿人。" ]
- 基金信息：
  
  国家重点研发计划(2024YFF0907600);国家自然科学基金(62273275);陕西省自然科学基础研究计划青年项目(2025JC-YBQN-847)
- DOI：10.16152/j.cnki.xdxbzr.2025-06-006
  中图分类号： TP391.4
- 收稿：2025-07-07，
  
  纸质出版：2025-12-25
- 稿件说明：
移动端阅览
赵万青, 徐朝阳, 谢智伟, 等. “博古问津”：知识图谱增强的文化遗产领域多模态大模型[J]. 西北大学学报（自然科学版）, 2025,55(6):1267-1284.

ZHAO Wanqing, XU Chaoyang, XIE Zhiwei, et al. Bogu-Wenjin: A cultural heritage domain multimodal large model enhanced by knowledge graphs[J]. Journal of Northwest University (Natural Science Edition), 2025, 55(6): 1267-1284.
赵万青, 徐朝阳, 谢智伟, 等. “博古问津”：知识图谱增强的文化遗产领域多模态大模型[J]. 西北大学学报（自然科学版）, 2025,55(6):1267-1284. DOI： 10.16152/j.cnki.xdxbzr.2025-06-006.

ZHAO Wanqing, XU Chaoyang, XIE Zhiwei, et al. Bogu-Wenjin: A cultural heritage domain multimodal large model enhanced by knowledge graphs[J]. Journal of Northwest University (Natural Science Edition), 2025, 55(6): 1267-1284. DOI： 10.16152/j.cnki.xdxbzr.2025-06-006.

摘要

近年来，大语言模型（LLMs）和多模态大模型（MLMs）在自然语言处理和多模态内容理解方面取得了显著成就，然而，这些通用模型在处理文化遗产相关任务时存在明显缺陷，如对领域专业术语的理解存在偏差、缺乏文化历史背景导致回答不够深入以及知识幻觉等问题，使得输出结果难以满足实际需求。针对这些挑战，首次提出了面向文化遗产领域的多模态大模型——“博古问津”。首先，设计半自动化策略构建大规模的多模态文化遗产数据集并形成多模态知识图谱；然后，利用构建的数据集对通用大模型进行图文对齐和指令微调两阶段训练，以适应文化遗产领域的特定需求；此外，还引入了知识图谱作为辅助知识库，通过图文检索和关系检索策略，有效提升了模型在文化遗产领域问答任务上的可信度和可解释性。实验结果表明，“博古问津”在文物图像描述、属性问题解答及关系问题理解等多个方面表现优异，相较于通用多模态大模型，对复杂文化内容的理解和回答能力提升效果显著，分别在文物图像描述、文物属性问题和文物关系问题3个不同任务的综合分值上高出次优模型21.4%、53%和20.6%。

Abstract

In recent years

large language models (LLMs) and multimodal large models (MLMs) have made significant achievements in natural language processing and multimodal content understanding. However

these general-purpose models have obvious shortcomings when dealing with tasks related to cultural heritage

such as biased understanding of domain-specific terminology

lack of cultural and historical background leading to superficial answers

and knowledge hallucination issues

making it difficult for the results to meet actual needs. In response to these challenges

this paper first proposes a multimodal large model oriented towards the field of cultural heritage: Bogu-Wenjin. This study first designs a semi-automated strategy to construct a large-scale multimodal cultural heritage dataset and forms a multimodal knowledge graph. Using the constructed dataset

the general large model is trained in two stages: image-text alignment and instruction fine-tuning

to adapt to the specific needs of the cultural heritage field. In addition

a knowledge graph is introduced as an auxiliary knowledge base

and the credibility and interpretability of the model in the field of cultural heritage Q&A tasks are effectively improved through graph-text retrieval and relationship retrieval strategies. Experimental results show that Bogu-Wenjin performs excellently in various aspects such as artifact image description

attribute question answering

and relationship question understanding. Compared with general multimodal large models

it significantly improves the ability to understand and answer complex cultural content

with a comprehensive score increase of 21.4%

53% and 20.6% in artifact image description

artifact attribute questions

and artifact relationship questions respectively over the second-best model.

关键词

Keywords

references

KASNECI E , SESSLER K , KÜCHEMANN S , et al . ChatGPT for good? On opportunities and challenges of large language models for education [J ] . Learning and Individual Differences , 2023 , 103 : 102274 .

NAVEED H , KHAN A U , QIU S , et al . A comprehensive overview of large languagemodels [EB/OL ] . ( 2023-11-27 ) [ 2024-12-20 ] . https://arxiv.org/abs/2307.06435 https://arxiv.org/abs/2307.06435 .

CHANG Y P , WANG X , WANG J D , et al . A survey on evaluation of large language models [J ] . ACM Transactions on Intelligent Systems and Technology , 2024 , 15 ( 3 ): 1 - 45 .

KOE HWEE NGA J , WAI MUN S . The influence of MLM companies and agent attributes on the willingness to undertake multilevel marketing as a career option among youth [J ] . Journal of Research in Interactive Marketing , 2011 , 5 ( 1 ): 50 - 70 .

YIN S K , FU C Y , ZHAO S R , et al . A survey on multimodal large language models [J ] . National Science Review , 2024 , 11 ( 12 ): nwae403 .

ZHANG D Z , YU Y H , DONG J H , et al . MM-LLMs: Recent advances in multimodal large language models [EB/OL ] . ( 2024-05-28 ) [ 2024-12-20 ] . https://arxiv.org/abs/2401.13601 https://arxiv.org/abs/2401.13601 .

CUI C , MA Y S , CAO X , et al . A survey on multimodal large language models for autonomous driving [C ] // 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) . Waikoloa, HI, USA : IEEE , 2024 : 958 - 979 .

LIPKOVA J , CHEN R J , CHEN B W , et al . Artificial intelligence for multimodal data integration in oncology [J ] . Cancer Cell , 2022 , 40 ( 10 ): 1095 - 1110 .

SHARMA K , GIANNAKOS M . Multimodal data capabilities for learning: What can multimodal data tell us about learning? [J ] . British Journal of Educational Technology , 2020 , 51 ( 5 ): 1450 - 1484 .

OPENAI , ACHIAM J , ADLER S , et al . GPT-4 technical report [EB/OL ] . ( 2024-03-04 ) [ 2024-12-20 ] . https://arxiv.org/abs/2303.08774 https://arxiv.org/abs/2303.08774 .

LIU H T , LI C Y , WU Q Y , et al . Visual instruction tuning [EB/OL ] . ( 2024-11-11 ) [ 2024-12-20 ] . https://arxiv.org/abs/2304.08485 https://arxiv.org/abs/2304.08485 .

ASKELL A , BAI Y T , CHEN A N , et al . A general language assistant as a laboratory for alignment [EB/OL ] . ( 2021-11-09 ) [ 2024-12-20 ] . https://arxiv.org/abs/2112.00861 https://arxiv.org/abs/2112.00861 .

GAN Z , LI L J , LI C Y , et al . Vision-language pre-training: Basics, recent advances, and future trends [J ] . Foundations and Trends ® in Computer Graphics and Vision , 2022 , 14 ( 3/4 ): 163 - 352 .

BHATIA G , NAGOUDI E M B , CAVUSOGLU H , et al . FinTral: A family of GPT-4 level multimodal financial large language models [EB/OL ] . ( 2024-06-14 ) [ 2024-12-20 ] . https://arxiv.org/abs/2402.10986 https://arxiv.org/abs/2402.10986 .

LI C Y , WONG C , ZHANG S , et al . LLaVA-med: Training a large language-and-vision assistant for biomedicine in one day [EB/OL ] . ( 2023-06-01 ) [ 2024-12-20 ] . https://arxiv.org/abs/2306.00890 https://arxiv.org/abs/2306.00890 .

程启玥 . 我国文化遗产领域数字人文研究热点与趋势探究 [J ] . 现代信息科技 , 2024 , 8 ( 14 ): 106 - 111 .

CHEN Q Y . Research hotspots and trends of digital humanities in the field of cultural heritage in China [J ] . Modern Information Technology , 2024 , 8 ( 14 ): 106 - 111 .

MITTAL R , GARG A . Text extraction using OCR: A systematic review [C ] // 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA) . Coimbatore, India : IEEE , 2020 : 357 - 362 .

徐增林 , 盛泳潘 , 贺丽荣 , 等 . 知识图谱技术综述 [J ] . 电子科技大学学报 , 2016 , 45 ( 4 ): 589 - 606 .

XU Z L , SHENG Y P , HE L R , et al . A survey on knowledge graph techniques [J ] . Journal of University of Electronic Science and Technology of China , 2016 , 45 ( 4 ): 589 - 606 .

杜鹏飞 , 李小勇 , 高雅丽 . 多模态视觉语言表征学习研究综述 [J ] . 软件学报 , 2021 , 32 ( 2 ): 327 - 348 .

DU P F , LI X Y , GAO Y L . Survey on multimodal visual-linguistic representation learning [J ] . Journal of Software , 2021 , 32 ( 2 ): 327 - 348 .

DEVLIN J , CHANG M W , LEE K , et al . BERT: Pre-training of deep bidirectional transformers for language understanding [EB/OL ] . ( 2019-05-24 ) [ 2024-12-20 ] . https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805 .

BROWN T B , MANN B , RYDER N , et al . Language models are few-shot learners [C ] // Proceedings of the 34th International Conference on Neural Information Processing Systems . Vancouver, BC, Canada : ACM , 2020 : 1877 - 1901 .

TOUVRON H , LAVRIL T , IZACARD G , et al . LLaMA: Open and efficient foundation language models [EB/OL ] . ( 2019-05-24 ) [ 2024-12-20 ] . https://arxiv.org/abs/2302.13971 https://arxiv.org/abs/2302.13971 .

BAI J Z , BAI S , CHU Y F , et al . Qwen technical report [EB/OL ] . ( 2019-05-24 ) [ 2024-12-20 ] . https://arxiv.org/abs/2309.16609 https://arxiv.org/abs/2309.16609 .

YANG Z , LI L , LIN K , et al . The dawn of LMMs: Preliminary explorations with GPT-4V (ision) [EB/OL ] . ( 2023-10-11 ) [ 2024-12-20 ] . https://arxiv.org/abs/2309.17421 https://arxiv.org/abs/2309.17421 .

RADFORD A , KIM J W , HALLACY C , et al . Learning transferable visual models from natural language supervision [EB/OL ] . ( 2021-02-26 ) [ 2024-12-20 ] . https://arxiv.org/abs/2103.00020 https://arxiv.org/abs/2103.00020 .

ZHU D Y , CHEN J , SHEN X Q , et al . Minigpt-4: Enhancing vision-language understanding with advanced large language models [EB/OL ] . ( 2023-04-20 ) [ 2024-12-20 ] . https://arxiv.org/abs/2304.10592 https://arxiv.org/abs/2304.10592 .

CHIANG W L , LI Z H , LIN Z , et al . Vicuna: An open-source chatbot impressing GPT-4 with 90% * ChatGPT quality [EB/OL ] . ( 2023-03-30 ) [ 2024-12-20 ] . https://vicuna.lmsys.org https://vicuna.lmsys.org .

BAI J Z , BAI S , YANG S S , et al . Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond [EB/OL ] . ( 2023-10-13 ) [ 2024-12-20 ] . https://arxiv.org/abs/2308.12966 https://arxiv.org/abs/2308.12966 .

ZENG A H , XU B , WANG B W , et al . ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools [EB/OL ] . ( 2024-06-18 ) [ 2024-12-20 ] . https://arxiv.org/abs/2406.12793 https://arxiv.org/abs/2406.12793 .

HARTSOCK I , RASOOL G . Vision-language models for medical report generation and visual question answering: A review [EB/OL ] . ( 2024-04-15 ) [ 2024-12-20 ] . https://arxiv.org/abs/2403.02469 https://arxiv.org/abs/2403.02469 .

HAN T Y , LIAN Q , PAN R , et al . The instinctive bias: Spurious images lead to illusion in MLLMs [C ] // Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing . Miami, FLorida, USA : ACL , 2024 : 16163 - 16177 .

HU E J , SHEN Y , WALLIS P , et al . LoRA: Low-rank adaptation of large language models [EB/OL ] . ( 2021-10-16 ) [ 2024-12-20 ] . https://arxiv.org/abs/2106.09685 https://arxiv.org/abs/2106.09685 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

河西走廊文化遗产时空分布特征及自然影响因素分析

文化遗产数字化保护与应用研究综述

文化遗产保护视野下的新农村建设

关中天水经济区文化遗产旅游绩效空间分异研究

文化遗产旅游地的体验式开发探讨——以云冈石窟为例