大型预训练语言模型在网络健康信息鉴别中的应用探讨

doi:10.13998/j.cnki.issn1002-1248.23-0288

摘要/Abstract

摘要： [目的/意义]探讨ChatGPT等大规模预训练语言模型在网络健康信息识别中的应用效果,为人工智能在健康信息领域的应用提供参考。[方法/过程]以国内某权威辟谣平台与健康相关的信息为研究对象,使用“ChatGPT”和“讯飞星火”对其真实性进行鉴定,对其性能进行评估,并将鉴定结果与医学专家或权威机构的鉴定结果进行比较。[结果/结论]ChatGPT和讯飞星火的鉴别准确率分别为93.9%和92.9%,F1值分别为0.951和0.946,应用效果良好。两者生成的解释文本内容比较详细,语言比较流畅,文本长度和语义相似度与专家文本高度接近,但对个别信息的解释仍存在科学依据不够详细、逻辑错误等问题。实验结果表明,大规模预训练语言模型在辅助网络健康信息识别任务方面具有一定的优势,但仍需要人工干预以保证结果的准确性和可靠性。

关键词: 人工智能, 健康信息, 鉴别, ChatGPT

Abstract: [Purpose/Significance] Taking the popular "chat robot" ChatGPT and the recently launched similar product "iFLYTEK Spark" as the research object, this paper explores their applications in the identification of online health information, and discusses their advantages and disadvantages, in order to provide reference for the large-scale pre-training language model in the field of health information identification. Based on the review of relevant literature on online health information authentication, deep learning models have been widely applied in the task of online health information authentication in recent years. With the rapid development of large pre-training language models such as ChatGPT, it is a novel idea to explore their discriminating ability in online health information. [Method/Process] Researchers selected health-related information from the most authoritative rumor-refuting websites in China, used "ChatGPT" and "iFLYTEK Spark" to verify the authenticity of the online health information, evaluated their performance, and compared their identification results with the expert identification results. The identification accuracy of ChatGPT and iFLYTEK Spark language model was 93.9% and 92.9%, respectively, and the F1 value was 0.951 and 0.946, respectively, which had a good application effect. The generated explanatory texts were more detailed and the language was relatively smooth. In terms of the length and dispersion of the explanatory text, ChatGPT is closer to that of medical experts, while iFLYTEK Spark's explanatory text is relatively long and less discrete. In terms of semantic similarity, ChatGPT and iFLYTEK Spark were almost equal in performance, and their understanding of health information was close to that of human experts to some extent. Through the analysis of typical samples, it can be seen that an AI large model cannot accurately identify news or emergency information for the time being, and the understanding of individual health propositions with complex semantics will occasionally be biased. [Results/Conclusions] The experimental results show that ChatGPT and iFLYTEK Spark have good discriminative effect in the field of online health information identification, but there are shortcomings, and manual intervention is needed to ensure the accuracy and reliability of the results. In the future, in the field of AI large model research, researchers are suggested to attach importance to the construction and application of high-quality corpora in vertical fields. In the field of online health information identification, practitioners can use models such as ChatGPT as tools to help identify and refine health information. There are also limitations in this article. For example, the amount of data involved in the test is not large enough, ChatGPT uses GPT3.5 model, and the online application time of iFLYTEK Spark language model is relatively short. In future studies, the amount of online health information can be further increased, and the updated version of an AI large model can be tested and evaluated.

Key words: artificial intelligence, health information, identification, ChatGPT

中图分类号:

G252

王超, 孔祥辉. 大型预训练语言模型在网络健康信息鉴别中的应用探讨[J]. 农业图书情报学报, 2023, 35(6): 51-59.

WANG Chao, KONG Xianghui. Application of Large-scale Pre-Training Language Model in Network Health Information Identification[J]. Journal of Library and Information Science in Agriculture, 2023, 35(6): 51-59.

参考文献

[1] 秦盼盼, 谢莉琴, 雷行云, 等. 基于联邦学习的区域健康医疗大数据中心建设探析[J]. 卫生软科学, 2023, 37(5): 73-78.
QIN P P, XIE L Q, LEI X Y, et al.Research on the construction of regional big data center in healthcare based on federated learning[J]. Soft science of health, 2023, 37(5): 73-78.
[2] 王超, 卢智增. 农村大学生健康信息替代搜寻意向影响因素探析[J]. 情报探索, 2023(4): 24-29.
WANG C, LU Z Z.Analysis of influencing factors of intention to surrogate health information seeking of rural college students[J]. Information research, 2023(4): 24-29.
[3] 曹雅宁, 柯青. 为什么人们对虚假健康信息的易感性不同? ——基于信息加工过程组态的分析[J]. 现代情报, 2023, 43(1): 40-54.
CAO Y N, KE Q.Why people show differences in their susceptibility to health misinformation? - Research based on information processing configurations[J]. Journal of modern information, 2023, 43(1): 40-54.
[4] 王超, 卢智增. 农村老年人健康信息替代搜寻者特征及其行为影响因素探析[J]. 图书情报导刊, 2022, 7(10): 37-44.
WANG C, LU Z Z.Analysis on the characteristics and behavioral influencing factors of rural elderly health information surrogate seeker[J]. Journal of library and information science, 2022, 7(10): 37-44.
[5] 张智雄, 曾建勋, 夏翠娟, 等. 回应AIGC的信息资源管理学人思考[J]. 农业图书情报学报, 2023, 35(1): 4-28.
ZHANG Z X, ZENG J X, XIA C J, et al.Information resource management researchers' thinking about the opportunities and challenges of AIGC[J]. Journal of library and information science in agriculture, 2023, 35(1): 4-28.
[6] CHOU W Y S, OH A, KLEIN W M P. Addressing health-related misinformation on social media[J]. JAMA, 2018, 320(23): 2417-2418.
[7] WASZAK P M, KASPRZYCKA-WASZAK W, KUBANEK A.The spread of medical fake news in social media - The pilot quantitative study[J]. Health policy and technology, 2018, 7(2): 115-118.
[8] PURNOMO M H, SUMPENO S, SETIAWAN E I, et al.Keynote speaker II: Biomedical engineering research in the social network analysis era: Stance classification for analysis of hoax medical news in social media[J]. Procedia computer science, 2017, 116: 3-9.
[9] KINSORA A, BARRON K, MEI Q Z, et al.Creating a labeled dataset for medical misinformation in health forums[C]// 2017 IEEE International Conference on Healthcare Informatics (ICHI). Piscataway, New Jersey: IEEE, 2017: 456-461.
[10] GHENAI A, MEJOVA Y. Catching zika fever: Application of crowdsourcing and machine learning for tracking health misinformation on twitter[EB/OL].2017: arXiv: 1707.03778. https://arxiv.org/abs/1707.03778
[11] MRIDHA M F, KEYA A J, HAMID M A, et al.A comprehensive review on fake news detection with deep learning[J]. IEEE access, 2021, 9: 156151-156170.
[12] WANG Y T, NAZIR S, SHAFIQ M.An overview on analyzing deep learning and transfer learning approaches for health monitoring[J]. Computational and mathematical methods in medicine, 2021, 2021: 1-10.
[13] ABDULAZIZ A A, AHMED H M H. Deep learning prediction model for heart disease for elderly patients[J]. Intelligent automation & soft computing, 2023, 35(2): 2527-2540.
[14] 徐璐璐, 洪闅, 叶鹰. ChatGPT及GPT类技术的医学信息学应用前景探讨[J]. 情报理论与实践, 2023, 46(6): 38-42.
XU L L, HONG B, YE Y.Exploring the application prospect of ChatGPT and GPT-type technology in medical informatics[J]. Information studies: Theory & application, 2023, 46(6): 38-42.
[15] 李雅静, 孙佳佳. ChatGPT在网络健康信息鉴别中的应用探讨[J]. 图书馆论坛, 2023, 43(6): 85-92.
LI Y J, SUN J J.Discussion on the application of ChatGPT in online health information identification[J]. Library tribune, 2023, 43(6): 85-92.