Abstract:

[Purpose/Significance] The use of large language models (LLMs) for patent text mining has become a major research topic in recent years. However, existing studies mainly focus on the application of LLMs to specific tasks, and there is a lack of systematic evaluation of the application effects of fine-tuned LLMs across multiple scenarios. To address this problem, this study takes ChatGLM, an open-source LLM that supports local fine-tuning, as an example. We conduct a comparative evaluation of three types of patent text mining tasks-technical term extraction, patent text generation, and automatic patent classification-under a unified experimental framework. The performance of fine-tuned models is compared from six aspects: different training data sizes, different numbers of training epochs, different prompts, different prefix lengths, different datasets, and single-task versus multi-task fine-tuning. [Method/Process] This study was based on an open-source LLM and carried out fine-tuning research for specific patent tasks in order to clarify the impact of different fine-tuning strategies on the performance of LLMs in patent tasks. Considering task adaptability, model size, inference efficiency, and resource consumption, ChatGLM-6B-int4 was selected as the base model, and P-Tuning V2 was adopted as the fine-tuning method. Three categories of patent tasks are included: extraction, generation, and classification. The extraction task is patent keyword extraction. The generation tasks include: 1) innovation point generation; 2) abstract generation based on a given title; 3) rewriting an existing title; 4) rewriting an existing abstract; 5) generating novelty points based on an existing abstract; 6) generating patent advantages based on an existing abstract; and 7) generating patent application scenarios based on an existing abstract. Six experimental comparison dimensions are designed: 1) different training data sizes; 2) different numbers of training epochs; 3) different datasets with the same data size; 4) different prompts under the same task and data; 5) different P-Tuning V2 prefix lengths with the same training data; and 6) single-task fine-tuning versus multi-task fine-tuning. Two type of evaluation metrics were used. For extraction and generation tasks, the BLEU metric based on n-gram string matching was adopted. For classification tasks, accuracy, recall, and F1 score were used. [Results/Conclusions] Based on the fine-tuning results, several conclusions were obtained. First, a larger training data size does not always lead to better performance. Second, the appropriate number of training epochs depends on the data size. Third, under the same data distribution, different data subsets have limited influence on performance. Fourth, under the same task and dataset, different prompts have little impact on model performance. Fifth, the optimal prefix length is closely related to the training data size. Sixth, for a specific task, single-task fine-tuning performs better than multi-task fine-tuning. These conclusions provide reference and guidance for fine-tuning LLMs in practical patent information work.

Key words: patent text mining, large language model (LLM), fine-tuning, patent data analysis, information analysis method