中文    English

Journal of library and information science in agriculture

   

Automatic Figure Description Generation for Scholarly Articles via Multimodal Large Language Models

HUANG Yongwen1, SUN Tan2,3(), XIAN Guojian1,2, KONG Lingbo1, LUO Tingting1   

  1. 1. Agricultural Information Institute, Chinese Academy of Agricultural Sciences, Beijing 100081
    2. Key Laboratory of Agricultural Big Data, Ministry of Agriculture and Rural Affairs, Beijing 100081
    3. Chinese Academy of Agricultural Sciences, Beijing 100081
  • Received:2026-09-10 Online:2026-11-11
  • Contact: SUN Tan

Abstract:

[Purpose/Significance] Scholarly figures are important visual resources in academic papers that contain rich scientific information and have attracted increasing attention. Leveraging multimodal large language models (MLLMs) to automatically understand the content of figures in academic papers and generate context-enriched enhanced descriptions can improve the readability and retrievability of figures, while advancing the development of multimodal understanding and cross-modal generation technologies. [Method/Process] This study applies MLLMs to the task of automatic academic image captioning. We constructed an experimental framework combining prompt engineering and few-shot learning to demonstrate the effectiveness of image description generation across different scenarios. First, we constructed the experimental dataset. Full-text articles were retrieved and parsed using databases such as Elsevier and Web of Science. Graphical Abstract (GA) images, in-text figures and their captions, as well as contextual information surrounding the figures, were identified and subsequently reviewed manually, with a particular focus on verifying the contextual information of the figures, to ensure the accuracy of the dataset. Subsequently, prompt templates for automatic figure description generation and description similarity judgment were designed. By using precise prompts to clearly specify the task the model needs to perform, the contextual information provided, and the expected output format, the model can quickly adapt to description generation tasks. Finally, we conducted experiments on automatic description generation for graphical abstract (GA) images in the biomedical field and main-text images in the rice breeding domain. The Qwen3-VL MLLM was employed to conduct automatic description generation experiments on both graphical abstract (GA) figures and in-text figures, using various combinations of images, figure contexts, source paper titles, and abstracts as inputs. Furthermore, few-shot learning was employed to further enhance the model's performance on the image captioning task. [Results/Conclusions] Experimental results show that for Graphical Abstract (GA) images in the biomedical domain, the average similarity of image descriptions automatically generated by LLMs exceeds 82%. Incorporating the paper title and abstract information helps generate more accurate image descriptions. However, the overall improvement is marginal. For in-text figures in the rice breeding domain, the fine-tuned LLM, when combined with the images, image contextual information, and the paper title and abstract, achieves an average similarity of over 75% for automatically generated image descriptions. These results indicate that MLLMs can generate more detailed and contextually relevant descriptions by leveraging additional contextual information, thereby improving both the content quality and accuracy of image descriptions. The quality of figure descriptions is influenced by factors such as figure type,complexity, and the relevance of contextual information. We propose a dynamic generation strategy based on "discipline classification + image position + image type." This approach adopts differentiated processing methods for different image types, rather than a one-size-fits-all generation mode. Furthermore, we suggest leveraging established MLLMs to perform few-shot learning across various disciplines and image types, strengthening the model's recognition of domain-specific figures and improving its adaptability in specialized fields. Crucially, sample diversity should be prioritized over sheer data scale.

Key words: multimodal large language model, AI-generated images, academic figure, graphical abstract, prompt, figure description

CLC Number: 

  • G254

Fig.1

Research framework diagram"

Table 1

Prompt template content for automatic figure description generation"

Instruction(提示指令)

Given the following information:1) Figure, 2) Context.

Based on the above information, analyse and summarize the figure in detail, paying attention to the following requirements: 1) The summary should be centred on the information in the figure. "Context" can be used as background information, but the summary should focus on the information in the figure. 2) The generated summary should concisely and clearly summarize the information conveyed by the figure in no more than 200 words.

Please note the following five-dimensional evaluation criteria, which is the key point for evaluating the quality of the summary.

Faithfulness: The summary must strictly adhere to the information presented in the figures anddescriptions.

Completeness: The summary should encompass all key information and trends depicted in the figures.

Conciseness: The summary should avoid redundant information or non-critical details.

Logicality: The summary should be logically coherent and consistent with common sense and expertknowledge.

Accuracy: The key parameters and trend details in the summary—such as numerical values, units, comparative relationships, and trend directions—are consistent with those presented in the chart.

Table 2

Prompt template content for figure description similarity judgment"

Instruction(提示指令)

You are an expert scientific data annotator. Evaluate the semantic similarity between two scientific figure descriptions (Text A and Text B) based strictly on their scientific content.

Guiding Principles:

Scientific Substance First: Judge based on scientific facts, experimental logic, and conclusions.

Ignore Stylistic Differences: Disregard variations in writing style, sentence length, rhetoric, or phrasing that do not alter the scientific meaning.

Objectivity: Base your analysis solely on the provided text without subjective assumptions.

Evaluation Dimensions:

Systematically compare Text A and Text B across the following six dimensions:

Research Subjects: Do they involve the same biological systems, materials, disease models, or specific entities

Thematic Consistency: Do they address the same scientific question, hypothesis, or mechanism

Accuracy of Key Parameters: Whether key parameters or trend details—such as numerical values, units, comparative relationships, and trend directions—are consistent

Core Arguments/Claims: Are the main findings, conclusions, or functional explanations consistent

Key Terminology Overlap: Is there a match in professional nouns, molecules, pathways, and technical methods

Semantic Overlap: Is the information complementary, progressive, or contradictory

Scoring Rubric:

1.0 (Almost Equivalent): Describes the exact same experimental results and conclusions; differences are purely stylistic.

0.8~0.9 (High Similarity): Core scientific content is consistent; differences are minor or limited to background details.

0.5~0.7 (Moderate Similarity): Same research subject or theme, but distinct differences in focus, specific findings, or methods.

0.2~0.4 (Low Similarity): Few shared keywords (e.g., same broad field), but different scientific questions.

0.0~0.1 (Unrelated): No relation in research field, subjects, or conclusions.

Output Format:

After synthesizing the analysis, provide a similarity score ranging from 0.0 to 1.0.

Table 3

Settings of experimental parameters"

参数 设定值 参数说明
model qwen3-vl:8b 底座大模型
temperature 0.2 较低的温度值(0~1),使输出更具确定性、更客观
num_ctx 4 096 上下文窗口大小,控制模型能处理的文本长度

Table 4

Overall similarity overview"

指标 平均相似度 最小值 最大值
score1 0.828 4 0.15 0.98
score2 0.832 1 0.30 0.99

Table 5

Similarity comparison of three generation methods"

指标 平均相似度 最小值 最大值
score1 0.604 8 0 0.90
score2 0.716 7 0 0.85
score3 0.719 7 0 0.85

Table 6

Similarity comparison of three generation methods before and after few-shot learning"

指标 少样本学习前的平均分 少样本学习后的平均分 平均分变化
score1 0.638 5 0.657 8 0.019 3
score2 0.730 2 0.724 4 -0.005 8
score3 0.734 4 0.750 3 0.015 9

Table 7

Figure types and similarity analysis"

图片类型 图片特征 相似度表现 原因分析
技术示意图 如实验流程图、通路图等,结构清晰、元素标准,有固定描述范式 普遍高且稳定 视觉元素与专业术语对应关系明确,模型易于识别并组织语言
数据统计图 如柱状图、折线图、箱线图等,展示量化数据和统计比较 中等偏高,但波动大 模型能识别趋势,但易遗漏统计标注、误差线、精确数值等,导致细节描述不准
微观影像 如电镜图、显微照片、荧光定位图等,细节丰富,依赖专业背景解读 波动剧烈,普遍偏低 模型能描述看到的实体,但难以准确推断其生物学意义或结构名称
抽象图谱 如系统发育树、结构方程模型等,信息高度编码,解读严重依赖领域知识 风险极高,易出现低分或零分 纯视觉信息无法让模型理解颜色或形状背后的复杂含义。极易生成无关内容,需要依赖论文背景信息进行解读
实物照片 如植株表型、田间试验、病斑照片等,细节丰富 中等 模型能描述形态(如植株更高),但难以精确量化或关联具体处理条件

Table 8

Comparison of three automated generation methods"

生成模式 依赖信息 优点 缺点 适用场景建议
仅图片 视觉特征 处理标准示意图等结构化图片速度快,不受文本干扰 对复杂图表、无背景知识图片解读能力弱 初筛或处理大量已知类型的简单图表
图片+上下文 视觉+局部文本 可以补充图中未显示表达的细节 上下文质量不可控,表现不稳定 建议先过滤上下文中的噪声信息
图片+上下文+论文题名和摘要 视觉+局部文本+全局语境 相似度最高、最稳定,能理解图片的深层研究意图 需要获取并处理更多文本信息,流程更复杂 推荐作为高质量生成的默认模式,尤其对于研究论文

Fig.2

A multi-dimensional feature-based strategy for figure description generation"

[1]
国家数据局. 关于推进行业高质量数据集建设行动的实施方案(征求意见稿)[EB/OL]. (2026-04-15)[2026-09-30].
[2]
丁培. 学术图表知识发现技术框架及研究进展[J]. 图书情报工作, 2021, 65(23): 136-148.
Ding Pei. The technical framework and research progress of knowledge discovery in academic figures and tables[J]. Library and Information Service, 2021, 65(23): 136-148.
[3]
李英群, 李亚菲, 裴雷, 等. 基于YOLOv5-ECA-BiFPN的学术期刊文献图表识别与提取方法研究[J]. 数据分析与知识发现, 2023, 7(11): 158-171.
Li Yingqun, Li Yafei, Pei Lei, et al. Identifying and extracting figures and tables from academic literature based on YOLOv5-ECA-BiFPN[J]. Data Analysis and Knowledge Discovery, 2023, 7(11): 158-171.
[4]
Agarwal S, Yu Hong. FigSum: Automatically generating structured text summaries for figures in biomedical literature[C]//Proceedings of the American Medical Informatics Association Annual Symposium. San Francisco: PMC, 2009: 6-10.
[5]
Hossain M Z, Sohel F, Shiratuddin M F, et al. A comprehensive survey of deep learning for image captioning[J]. ACM Computing Surveys, 2019, 51(6): 1-36.
[6]
Sapkota R, Raza S, Shoman M, et al. Multimodal large language models for image, text, and speech data augmentation: A survey[PP/OL]. V2. arXiv (2025-03-21)[2026-08-22].
[7]
Tanasă A M, Oprea S V. Rethinking chart understanding using multimodal large language models[J]. Computers, Materials & Continua, 2025, 84(2): 2905-2933.
[8]
魏忠钰, 范智昊, 王瑞泽, 等. 从视觉到文本: 图像描述生成的研究进展综述[J]. 中文信息学报, 2020, 34(7): 19-29.
Wei Zhongyu, Fan Zhihao, Wang Ruize, et al. From vision to text: A brief survey for image captioning[J]. Journal of Chinese Information Processing, 2020, 34(7): 19-29.
[9]
李志欣, 魏海洋, 张灿龙, 等. 图像描述生成研究进展[J]. 计算机研究与发展, 2021, 58(9): 1951-1974.
Li Zhixin, Wei Haiyang, Zhang Canlong, et al. Research progress on image captioning[J]. Journal of Computer Research and Development, 2021, 58(9): 1951-1974.
[10]
查梦娟. 融合多模态特征的学术论文图像标注与关联发现研究[D]. 武汉: 武汉大学, 2022.
Zha Mengjuan. Research on image annotation and association discovery of academic papers based on multimodal features[D]. Wuhan: Wuhan University, 2022.
[11]
孙坦, 张智雄, 周力虹, 等. 人工智能驱动的第五科研范式(AI4S)变革与观察[J]. 农业图书情报学报, 2023, 35(10): 4-32.
Sun Tan, Zhang Zhixiong, Zhou Lihong, et al. The transformation and observations of AI for science(AI4S) driven by artificial intelligence[J]. Journal of Library and Information Science in Agriculture, 2023, 35(10): 4-32.
[12]
张逸勤, 邓三鸿, 巩洪村, 等. 面向AI4Science的科学论文图像语义描述框架体系构建研究[J]. 图书馆杂志, 2026, 45(5): 15-26.
Zhang Yiqin, Deng Sanhong, Gong Hongcun, et al. Research on the construction of a semantic description framework for scientific document images oriented toward AI4Science[J]. Library Journal, 2026, 45(5): 15-26.
[13]
Elsevier. Elsevier figure requirements across 2500+ journals[EB/OL]. [2026-08-10].
[14]
Nature Springer. Nature figure guidelines: Quick reference[EB/OL]. [2026-08-10].
[15]
Krukowski R A, Goldstein C M. The potential for graphical abstracts to enhance science communication[J]. Translational Behavioral Medicine, 2023, 13(12): 891-895.
[16]
Romans B. Are graphical abstracts a good idea [EB/OL]. [2026-09-01].
[17]
Lee J, Yoo J J. The current state of graphical abstracts and how to create good graphical abstracts[J]. Science Editing, 2023, 10(1): 19-26.
[18]
Yoon J, Chung E. An investigation on Graphical Abstracts use in scholarly articles[J]. International Journal of Information Management, 2017, 37(1): 1371-1379.
[19]
Huang Jiani, Chen Haihua, Yu Fengchang, et al. From detection to application: Recent advances in understanding scientific tables and figures[J]. ACM Computing Surveys, 2024, 56(10): 1-39.
[20]
丁培, 叶兰. 科技文献中学术图表标注研究进展[J]. 现代情报, 2021, 41(4): 165-177.
Ding Pei, Ye Lan. Research progress of academic figures and tables annotation in scientific literature[J]. Modern Information, 2021, 41(4): 165-177.
[21]
蔡祎然, 胡正银, 刘春江. 大语言模型赋能科技文献数据挖掘进展分析[J]. 农业图书情报学报, 2025, 37(2): 4-22.
Cai Yiran, Hu Zhengyin, Liu Chunjiang. Analysis of progress in data mining of scientific literature using large language models[J]. Journal of Library and Information Science in Agriculture, 2025, 37(2): 4-22.
[22]
陈宇峰, 黄琪, 江爱文, 等. 基于自提示多模态大语言模型和语义感知离散扩散模型的图像描述生成算法[J]. 中文信息学报, 2026, 40(6): 150-159.
Chen Yufeng, Huang Qi, Jiang Aiwen, et al. Exploring semantic-aware discrete diffusion model via self-prompting multimodal LLMs for image captioning[J]. Journal of Chinese Information Processing, 2026, 40(6): 150-159.
[23]
Zhang Jiahao, Zhang Dehai, He Xu, et al. An overview of image captioning generation based on large language models[C]//Trends in Haptics and Virtual Reality. Cham: Springer Nature Switzerland, 2025: 50-56.
[24]
Bhatia S, Mitra P. Summarizing figures, tables, and algorithms in scientific publications to augment search results[J]. ACM Transactions on Information Systems, 2012, 30(1): 1-24.
[25]
Chen Jingqiang, Zhuge Hai. Extractive summarization of documents with images based on multi-modal RNN[J]. Future Generation Computer Systems, 2019, 99: 186-196.
[26]
Saini N, Saha S, Potnuru V, et al. Figure summarization: A multiobjective optimization-based approach[J]. IEEE Intelligent Systems, 2019, 34(6): 43-52.
[27]
Saini N, Saha S, Bhattacharyya P, et al. Textual entailment—Based figure summarization for biomedical articles[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2020, 16(1s): 1-24.
[28]
Li Lei, Wang Yuqi, Xu Runxin, et al. Multimodal ArXiv: A dataset for improving scientific comprehension of large vision-language models[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Kerrville: Association for Computational Linguistics, 2024: 14369-14387.
[29]
Roberts J, Han Kai, Houlsby N, et al. SciFIBench: Benchmarking large multimodal models for scientific figure interpretation[PP/OL]. V2. arXiv (2024-12-05)[2026-10-20].
[30]
Li Sihang, Huang Jin, Zhuang Jiaxi, et al. SciLitLLM: How to adapt LLMs for scientific literature understanding[PP/OL]. V5. arXiv (2025-04-18)[2026-09-31].
[31]
Lu Jiasen, Xiong Caiming, Parikh D, et al. Knowing when to look: Adaptive attention via a visual sentinel for image captioning[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey: IEEE, 2017: 3242-3250.
[32]
Mao Junhua, Xu Wei, Yang Yi, et al. Deep captioning with multimodal recurrent neural networks (m-RNN)[PP/OL]. V5. arXiv (2015-06-11)[2026-09-30].
[33]
Chen Fuhai, Ji Rongrong, Sun Xiaoshuai, et al. GroupCap: Group-based image captioning with structured relevance and diversity constraints[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway, New Jersey: IEEE, 2018: 1345-1353.
[34]
Yang Linjie, Tang K, Yang Jianchao, et al. Dense captioning with joint inference and visual context[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway, New Jersey: IEEE, 2017: 1978-1987.
[35]
Newman B, Lee Y, Naik A, et al. ArxivDIGESTables: Synthesizing scientific literature into tables using language models[C]//Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. ACL, 2024: 9612-9631.
[36]
Guo Ziyu, Zhang R, Chen Hao, et al. SciVerse: Unveiling the knowledge comprehension and visual reasoning of LMMs on multi-modal scientific problems[PP/OL]. arXiv (2025-03-13)[2026-09-12].
[37]
Hsu T Y, Giles C L, Huang Tinghao. SciCap: Generating captions for scientific figures[C]//Findings of the Association for Computational Linguistics: EMNLP 2021. ACL, 2021: 3258-3264.
[38]
Yue Ling, Zhang Tingwen, Wang Jiaying, et al. DiagramBank: A quality-audited dataset of scientific schematic diagrams with multi-level document context[PP/OL]. V2. arXiv (2026-05-27)[2026-09-31].
[39]
Hu Linmei, Wang Duokang, Pan Yiming, et al. NovaChart: A large-scale dataset towards chart understanding and generation of multimodal large language models[C]//Proceedings of the 32nd ACM International Conference on Multimedia. New York: ACM, 2024: 3917-3925.
[40]
Liu Mengchen, Li Qixiu, Chen Dongdong, et al. SynChart: Synthesizing charts from language models[PP/OL]. arXiv (2024-09-25)[2026-03-31].
[41]
Masry A, Shahmohammadi M, Parvez M R, et al. ChartInstruct: Instruction tuning for chart comprehension and reasoning[C]//Findings of the Association for Computational Linguistics ACL 2024. ACL, 2024: 10387-10409.
[42]
Meng Fanqing, Shao Wenqi, Lu Quanfeng, et al. ChartAssistant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning[C]//Findings of the Association for Computational Linguistics: ACL 2024. Kerrville: Association for Computational Linguistics, 2024: 7775-7803.
[43]
Yue Tan, Shi Xuzhao, Mao Rui, et al. AnaFig: A human-aligned dataset for scientific figure analysis[C]//Proceedings of the 33rd ACM International Conference on Multimedia. New York: ACM, 2025: 12837-12843.
[44]
Huang Siyuan, Zhou Yifan, Gao Yutong, et al. SciFig: Towards automating editable figure generation for scientific papers[PP/OL]. V3. arXiv (2026-06-26)[2026-03-20].
[45]
Ghosh S, Anupam S. CapText: Large language model-based caption generation from image context and description[PP/OL]. V2. arXiv (2023-06-06)[2026-10-28].
[46]
Bai Shuai, Cai Yuxuan, Chen Ruizhe, et al. Qwen3-VL technical report[PP/OL]. V2. arXiv (2025-11-27)[2026-08-10].
[47]
黄永文, 孙坦, 鲜国建, 等. 学术期刊图文摘要研究及启示——以农业领域为例[J]. 中国科技期刊研究, 2024, 35(6): 772-780.
Huang Yongwen, Sun Tan, Xian Guojian, et al. Research and implications on graphical abstracts in academic journals: A case study in agricultural field[J]. Chinese Journal of Scientific and Technical Periodicals, 2024, 35(6): 772-780.
[48]
葛澜, 黄永文, 孔令博, 等. 面向AI4S的科技文献多模态知识抽取工具链研究[J]. 农业图书情报学报, 2026, 38(8): 32-48.
Ge Lan, Huang Yongwen, Kong Lingbo, et al. Multimodal knowledge extraction toolchain for scientific literature towards AI4S[J]. Journal of Library and Information Science in Agriculture, 2026, 38(8): 32-48.
[49]
Cao Tianshi, Law M, Fidler S. A theoretical analysis of the number of shots in few-shot learning[PP/OL]. V2. arXiv (2020-02-14)[2026-10-20].
[1] MO Jingshi. Role of Orientation and Practical Path of Libraries in Promoting Literacy Cultivation in the AIGC Era [J]. Journal of library and information science in agriculture, 2026, 38(7): 110-117.
[2] ZHANG Xintong, DENG Xiaozhao. Behavioral Pathways and Influencing Factors of UGC Community Users' Content Update Prompting Behavior [J]. Journal of library and information science in agriculture, 2026, 38(10): 61-76.
[3] SANG Yuanyuan. Multimodal Learning Technology Aimed at Exploring the Innovative Path of Library Intelligence Service [J]. Journal of library and information science in agriculture, 2025, 37(3): 42-52.
[4] SUN Lijuan. Exploring the Transition of Public Librarians to "Prompt Librarians" from the Perspective of New Quality Productive Forces [J]. Journal of library and information science in agriculture, 2025, 37(1): 17-32.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!