Abstract:

[Purpose/Significance] University libraries require efficient, data-driven academic evaluation to support management decisions. Traditional manual methods are slow, subjective, and untimely. While large language models (LLMs) offer automation potential, existing applications in this domain are limited, often focusing on auxiliary tasks and raising data security concerns with cloud-based processing. This study addresses these gaps by proposing a localized, intelligent agent for secure and interactive analysis of academic output. [Method/Process] A four-layer theoretical framework based on the DIKW model was established to guide the agent's design from data integration to wisdom generation. Grounded on the practical experience of academic evaluation services in libraries, this study systematically identified data requirements from dimensions of academic evaluation objects (institution, school, discipline, and researcher) and metrics (output, collaboration, impact, and quality), and formulated a metadata scheme to integrate bibliographic data, indexing data and evaluation data into a single structured table for research papers. A localized agent was implemented using open-source tools: Chainlit for the conversational interface, LangChain with the Kimi-K2-0905-Preview LLM as the core, and the ReAct framework to enable an iterative "Thought-Action-Observation" loop for complex reasoning and self-correction. The agent employs Text-to-SQL technology to translate natural language queries into executable PostgreSQL statements. Comprehensive prompt engineering was conducted to guide the LLM in accurate SQL generation, handling challenges such as data deduplication, multi-value fields, and entity disambiguation. This enables dynamic intent interpretation, multi-step data retrieval and validation, and output generation combining visualizations and structured reports. [Results/Conclusions] The agent was evaluated using a test dataset of over 30 000 structured academic papers and a multi-dimensional set of 20 test queries covering various evaluation scenarios and complex composite questions. The agent achieved a 100% final accuracy rate. The initial query accuracy was 85%, with errors primarily related to recognizing informal entity names (e.g., abbreviations). All errors were autonomously corrected within one ReAct iteration, demonstrating effective self-repair. Comparative analysis against two general-purpose data analysis agents showed the proposed agent's superior accuracy and stability, particularly in handling entity disambiguation and complex multi-turn tasks. The study confirms that the locally-deployed intelligent agent provides an effective, secure, and interactive solution for academic output analysis, successfully bridging natural language queries with precise data retrieval. Limitations include the evaluation's primary focus on data retrieval accuracy rather than narrative quality, and a test scope limited to core academic evaluation queries. Future work will expand the agent's capabilities to support diverse research outputs (e.g., patents and monographs), enhance visualization integration, and enable customizable report template generation.

Key words: large language model, intelligent agent, academic output data, academic evaluation