Abstract:

[Purpose/Significance] In recent years, the rapid rise of large language model technology has shown significant advantages in understanding semantic context and capturing multidimensional sentiment tendencies. This study explores an aspect-level sentiment analysis method for science and technology policy comments based on large language models, aiming to uncover latent knowledge within these texts and provide data support for evaluating the effectiveness and subsequent optimization of policies. [Method/Process] Taking the electric vehicle industry as an example, a burgeoning sector vital to achieving the "dual carbon" goals and promoting green low-carbon development, this study proposed a policy satisfaction evaluation model. The model uses large language models for fine-grained aspect-level sentiment analysis of policy comment texts. The process includes the following steps: 1) Data collection and preprocessing: Comments related to electric vehicle policies were collected from the "Interactive Topics" section of the "Autohome" website using Python. Deep learning techniques were applied to set rules for the comment texts and automatically add punctuation marks to Chinese texts for data pre-processing. 2) Aspect word extraction: The steps include text tokenization, determining a candidate aspect word set, expanding the aspect word set, and clustering aspect words. A total of 3 405 aspect words were extracted from 35 000 comments, forming six clusters: infrastructure construction, vehicle performance configuration, national policies, technological development, automotive safety, and automotive sales market. Aspect-level sentences were extracted using aspect words and punctuation information, with a subset of sentences manually labeled to build training and validation corpora, resulting in 14 911 aspect-level sentences. 3) Sentiment tendency recognition model training: A prompt template for aspect-level sentiment classification tasks was designed, and the LoRA method was used to fine-tune the large language model with the manually labeled training set. The model's performance was evaluated using a validation set, resulting in the classification of comments on electric vehicle policies into positive, neutral, and negative sentiments. 4) Comparative experiment: The fine-tuned large model was compared with the mainstream sentiment classification model, BERT, to assess the performance of different models in aspect-level sentiment classification tasks. [Results/Conclusions] The results show that compared to the BERT model, the proposed method outperformed other methods in multiple metrics, including accuracy, recall, and F1 score, with improvements of 11.49%, 12.43% and 11.43%, respectively. Overall, public attention is higher towards vehicle performance configuration and automotive sales market, while infrastructure construction receives the lowest attention. The overall public satisfaction with electric vehicles is relatively low, with negative comments outweighing positive comments across all aspects, consistent with the "negative bias" theory in social psychology. Satisfaction issues are particularly prominent in the areas of automotive safety and infrastructure construction. Finally, policy recommendations have been proposed to optimize electric vehicle subsidy policies, strengthen policy promotion, improve infrastructure construction, and enhance after-sales service support systems.

Key words: science and technology policy review, aspect-level sentiment analysis, policy attention, policy satisfaction, large language model