Abstract:

[Purpose/Significance] Academic literature retrieval supports scientific research and subject services in university libraries. As the volume of publications increases, researchers encounter mounting challenges in developing search strategies and screening results, particularly when their research questions span several disciplines. This study examines how a large language model multi-agent system (LLM-MAS) can support the literature retrieval process and help university libraries develop a system architecture for their services. [Method/Process] The study combined system development with a comparative evaluation. Requirements were identified through semi-structured interviews with five subject librarians and an analysis of retrieval tasks from actual coursework. The resulting workflow comprised three stages: finding relevant literature, screening the results, and using the selected literature. A prototype was built with LangGraph, incorporating a Search Agent, a Filter Agent, an Analysis and Synthesis Agent, and a coordinating agent. Each agent was assigned a defined role and exchanged information through standardized JSON messages. A centralized message queue managed task scheduling, while preset thresholds triggered feedback and further retrieval or analysis as needed. Human review incorporated two points: confirmation of the search strategy and review of the analysis and synthesis output. The evaluation used CNKI and ScienceDirect as the main databases and compared manual retrieval (Group A), single-agent retrieval (Group B), and LLM-MAS retrieval (Group C). Six tasks covered simple, medium, and complex retrieval needs. Three subject librarians, each with more than five years of professional experience, completed the tasks under a Latin square arrangement. Each librarian performed two tasks in each mode, yielding six observations per mode. Performance was assessed using recall, precision@20, total retrieval time, and the number of distinct retrieval strategies. A reference set of relevant literature was established through independent annotation by two senior subject librarians, and the union of their annotations was taken as the reference standard. Inter-annotator agreement was assessed using the Kappa coefficient. [Results/Conclusions] In the observed tasks, LLM-MAS achieved higher average recall than manual and single-agent retrieval, with differences of 19.0 and 7.2 percentage points, respectively. Keyword expansion and the use of multiple search strategies may have contributed to this broader coverage. LLM-MAS generated an average of 5.7 distinct retrieval strategies, compared with 3.2 for manual retrieval. Average retrieval time was 21 min for LLM-MAS, 52 min for manual retrieval, and 19 min for single-agent retrieval. LLM-MAS therefore requires less time than manual retrieval but slightly more than the single-agent approach. The additional coordination and review steps may account for part of this difference. Precision@20 varied a little across the three modes. In the reported feedback example, the Filter Agent initiated supplementary retrieval, while feedback from the Analysis and Synthesis Agent was followed by an increase in the clustering silhouette coefficient from 0.26 to 0.38. These observations illustrate how the feedback process operated, but they do not establish its independent contribution to overall performance. The evaluation involved only three participants and six observations per mode. The modes also differed in their access to supplementary retrieval sources and the extent of human intervention. Results are therefore reported as descriptive statistics, using means and standard deviations, without formal significance testing. The observed differences reflect the combined effects of system coordination, supplementary sources, and human review, and cannot be attributed to multi-agent coordination alone. Larger studies with matched retrieval sources and review procedures are needed to assess whether these findings hold across task types. Analysis by task complexity, together with statistical inference and effect-size estimates, would provide a firmer basis for evaluating the system. The prototype provides a framework for implementing library retrieval services. There is exploratory evidence suggesting that it provides broader literature coverage and faster retrieval time than manual searching.

Key words: large language model, multi-agent collaboration, LLM-MAS, LangGraph, retrieval efficiency