Abstract:

[Purpose/Significance] As generative artificial intelligence (AI) transforms library services, existing evaluation systems fail to capture dynamic characteristics of AI-driven resource discovery. This study develops a dynamic evaluation framework for public libraries' AI-enhanced services, addressing the gap between technological innovation and service assessment. [Method/Process] The research employed a mixed-methods approach to develop and verify a multi-dimensional evaluation framework based on Knowledge Organization Systems (KOS) theory. The framework comprises five primary dimensions: physical environment, technical architecture, content organization, user interaction, and innovation capability-operationalized through fifteen secondary indicators. Each indicator was carefully designed to capture AI-specific capabilities, including cognitive guidance efficiency, multimodal interaction precision, semantic network depth, and generation-enhanced utilization rate. A sophisticated hybrid weighting methodology was implemented, integrating subjective and objective approaches. For subjective weights, the Analytic Hierarchy Process was employed with 30 domain experts constructing pairwise comparison matrices using standardized scaling methods. Geometric mean aggregation was applied to synthesize individual judgments, with consistency ratios maintained below the threshold to ensure logical coherence. For objective weights, the entropy method analyzed actual evaluation data variance, with greater variance indicating higher discriminatory power. The final weights were derived through multiplicative synthesis combining both approaches. The empirical validation study involved collecting 492 valid questionnaires from 14 strategically selected public libraries representing different stages of AI implementation between September and November 2024: one municipal library with comprehensive AI deployment, 11 district libraries with partial implementation, and 2 county libraries in early adoption phases. The questionnaire utilized a five-point Likert scale to assess real-time service performance across multiple scenarios. Statistical analysis employed fuzzy comprehensive evaluation to handle uncertainty in subjective assessments, structural equation modeling to validate construct relationships, and latent class analysis to identify distinct user interaction patterns. The framework demonstrated high reliability with Cronbach's alpha reaching 0.845 and strong construct validity with KMO value of 0.873. [Results/Conclusions] Content organization emerged as the most critical dimension with a combined weight of 0.302 2, while semantic network depth, cognitive guidance efficiency, and cross-media consistency ranked as top secondary indicators with weights of 0.090 3, 0.086 1, and 0.084 7 respectively. Performance evaluation revealed content organization scoring 74.873 points versus user interaction at 68.040 points, highlighting the gap between technical capabilities and user experience. Significant differences existed across library levels, with municipal libraries outperforming county libraries by over one point in technical architecture and semantic network depth. Four distinct user patterns emerged: technology-oriented, content-immersive, efficiency-focused, and assistance-dependent. Each requires a tailored service approach. The study proposes the following optimization strategies: multimodal interaction frameworks, adaptive user profiling, hierarchical collaboration mechanisms, and knowledge graph-based content reorganization.

Key words: generative artificial intelligence, resource discovery services, dynamic evaluation model, smart library, large language models