中文    English

Journal of library and information science in agriculture ›› 2022, Vol. 34 ›› Issue (2): 88-101.doi: 10.13998/j.cnki.issn1002-1248.21-0528

Previous Articles     Next Articles

The Validity of Peer Review Results of DEA Based Super Efficiency Projects

WAN Hao1, ZHANG Fujun2, LV Qianqian3   

  1. 1. Library of Shandong University of Science and Technology, Qingdao 266590;
    2. Development Planning Office, Shandong University of Science and Technology, Qingdao 266590;
    3. Document Information Center, Chinese Academy of Sciences, Beijing 100190
  • Received:2021-07-08 Online:2022-02-05 Published:2022-03-24

PDF (PC)

486

Abstract: [Purpose/Significance] This article uses some projects'peer review results as an example to explore the effectiveness of peer review based on expert knowledge. [Method/Process] It uses data envelopment analysis to measure the R&D efficiency of the projects. The evaluation model is a CCR model modified by super-efficiency, and the efficiency index is used as a verification standard for the effectiveness of peer review results. The data source is 126 "948 projects" implemented by the State Forestry Administration, covering 13 input and output evaluation indicators. [Results/Conclusions] The experiment found that there was a moderate positive correlation between the project peer review results and the actual R&D efficiency (DEA super efficiency), and the Spearman correlation coefficient ρ=0.250 (Sig.<0.01). The percentages of type I errors and type II errors made by subjective review were 17.5% and 18.2%, respectively, and effective peer review decisions accounted for 64.3%. It is confirmed that there are certain systematic errors in peer review, but they are inevitable.

Key words: peer review, project evaluation, effectiveness, super efficiency DEA

CLC Number: 

  • G301
[1] PRICE D J D. Little science, big science-and beyond[M]. New York: Columbia university press, 1986.
[2] 万昊, 谭宗颖, 朱相丽. 同行评议与文献计量在科研评价中的作用分析比较[J]. 图书情报工作, 2017, 61(1): 134-152.
WAN H, TAN Z Y, ZHU X L.Comparison of the role of peer review and bibliometrics in research evaluation[J]. Library and informationservice, 2017, 61(1): 134-152.
[3] MERTON R K.Social theory and social structure[M]. New York: Free press, 1968.
[4] 姜春林, 张立伟, 刘学. 中外同行评议研究现状及问题探讨[J]. 科技管理研究, 2015(3): 163-166.
JIANG C L, ZHANG L W, LIU X.The status quo of peer review research and its problems[J]. Science and technology management research, 2015(3): 163-166.
[5] 盛怡瑾, 初景利. 同行评议质量控制方法研究进展[J]. 出版科学,2018, 26(5): 48-55.
SHENG Y J, CHU J L.Research progress of peer review quality control method[J]. Publishing journal, 2018, 26(5): 48-55.
[6] CARTER B.Peer review a good but flawed system[J]. Journal of child health care, 2017, 21(3): 233-235.
[7] WESSELY S, WOOD F.Peer review of grant applications: A systematic review[M]//, GODLEE F, JEFFERSON T. Peer review in health sciences, London: British medical journal books, 1999.
[8] MAVIS B, KATZ M.Evaluation of a program supporting scholarly productivity for new investigators[J]. Academic medicine, 2003, 78(7): 757-765.
[9] BORNMANN L, DANIEL H D.Convergent validation of peer review decisions using the h index. Extent of and reasons for type I and type II errors[J]. Journal of informetrics, 2007, 1(3): 204-213.
[10] BORNMANN L, WALLON G, LEDIN A.Does the committee peer review select the best applicants for funding? An investigation of the selection process for two european molecular biology organization programmes[J]. PLoS ONE, 2008, 3(10): E3480.
[11] NEUFELD J, VON INS M.Informed peer review and uninformed bibliometrics?[J]. Research evaluation, 2011, 20(1): 31-46.
[12] LI D, AGHA L.Big names or big ideas: Do peer review panels select the best science proposals?[J]. Science, 2015, 348(6233): 434-438.
[13] MELIN G, DANELL R.The top eight percent: Development of approved and rejected applicants for a prestigious grant in Sweden[J]. Science and public policy, 2006, 33(10): 702-712.
[14] HORNBOSTEL S, BOHMER S, KLINGSPORN B, et al.Funding of young scientist and scientific excellence[J]. Scientometrics, 2009,79(1): 171-190.
[15] VAN DEN BESSELAAR P, LEYDESDORFF L. Past performance, peer review, and project selection: A case study in the social and behavioral sciences[J]. Research evaluation, 2009, 18(4): 273-288.
[16] BORNMANN L, LEYDESDORFF L, VAN DEN BESSELAAR P. A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications[J]. Journal of informetrics, 2010, 4(3): 211-220.
[17] BUXTON M, HANNEY S, MORRIS S, et al.Medical research - What's it worth? Estimating the economic benefits from medical research in the UK[R]. Health economics research group, office of health economics & rand, Europe, London: UK evaluation forum, 2008.
[18] MACILWAIN C.Science economics: What science is really worth?[J]. Nature, 2010, 465(10): 682-684.
[19] 王海宁, 李姗姗, 栾贞增. 高校科研成果转化能力与效率关系评价——基于2016年教育部直属61所高校的实证研究[J]. 科技管理研究, 2018, 406, 38(12): 147-154.
WANG H N, LI S S, LUAN Z Z. Evaluation on the relationship between transformation ability and efficiency of scientific research achievements in colleges and universities: An empirical study of 61 universities directly under the chinese ministry of education in2016[J]. Science and technology management research, 2018, 406,38(12): 147-154.
[20] HAZELKORN E.European commission: Expert group on assessment of university-based research, assessing Europe's university based research[R]. Dublin institute of technology, centre for social and educational research, 2010.
[21] IVANOVA I A, LEYDESDORFF L.Knowledge-generating efficiency in innovation systems the acceleration of technological paradigm changes with increasing complexity[J]. Technological forecasting &social change, 2015, 96: 254-265.
[22] SALTER A J, MARTIN B R.The economic benefits of publicly funded basic research: a critical review[J]. Research policy, 2001,30(3): 509-532.
[1] ZHANG Zhixiong, WANG Yuju, ZHAO Yang. Development Trends of International Open Peer Review Platforms and Recommendations for China [J]. Journal of library and information science in agriculture, 2024, 36(5): 14-22.
[2] FAN Beibei, LI Jin, FENG Xian. Effectiveness Evaluation of Rural Public Cultural Services under the Guidance of "People-Oriented": Taking Ecological Conservation Area of Beijing as an Example [J]. Journal of library and information science in agriculture, 2023, 35(9): 80-90.
[3] WNAG Lingfeng, WANG Shenpeng. Key Parameter Optimization Design of Self-organizing Peer Review in National Preprint Publishing Platform Based on Response Surface Analysis [J]. Journal of library and information science in agriculture, 2023, 35(7): 75-84.
[4] ZHAO Youlin, PANG Hangyuan, LIN Yini, PAN Yigai. The Policy Direction, Practice Development and Research Focus of Digital Marketing Activities [J]. Journal of library and information science in agriculture, 2023, 35(2): 4-15.
[5] HUO Zhenxiang, QU Lichun, LI Xiaoping. Resisting Measures and Thinking to Academic Misconduct from the Perspective of Scientific Journal Editor [J]. Journal of library and information science in agriculture, 2018, 30(7): 137-140.
[6] WANG Binying. Research on the Evaluation and Promotion of University Library's Characteristic Resource Utility Efficiency [J]. Journal of library and information science in agriculture, 2018, 30(1): 59-63.
[7] ZENG Yuanyuan. Study on the New Reading Service Model with the Theme of Social Current Affairs Hotspot A Case of Reading Promotion Project of “Dunhuang in Library” in East China Normal University [J]. Journal of library and information science in agriculture, 2017, 29(8): 186-189.
[8] LI Xiangmin, XU Su, CHEN Xin, TANG Xiaoyan, TAO Wenqi, WANG Changqun. Manuscript Handling Based on the Peer Review Comments:Cases Analysis [J]. Journal of library and information science in agriculture, 2017, 29(12): 159-163.
[9] LI Xiangmin, XU Su, CHEN Xin, WANG Changqun, TANG Xiaoyan, TAO Wenqi. Academic Journals Editor Should Appropriately Handle Revised Manuscripts Which Have Been Peer-Reviewed [J]. Journal of library and information science in agriculture, 2017, 29(10): 142-145.
[10] FANG Rui, GAO Chong-sheng, LI Jing. Quality Control of Three-level Peer Review System for Academic Agricultural Sci-tech Periodicals [J]. Journal of library and information science in agriculture, 2016, 28(4): 160-162.
[11] JIA Rui. Problems and Countermeasures of Librarian Training Effectiveness [J]. Journal of library and information science in agriculture, 2015, 27(8): 122-124.
[12] CHANG Fei, FU Xiu-ying. Evaluation on the Effectiveness of Enterprise Patent Strategy Implementation [J]. Journal of library and information science in agriculture, 2015, 27(2): 97-100.
[13] LIU Lan. The Study on the knowledge communication effectiveness of Library in the New Media Era——Investigation among readers of the Library of Liaoning University [J]. Journal of library and information science in agriculture, 2014, 26(2): 89-92.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Editorial Board of Journal of library and information science in agriculture
Tel：(010)82109667　Fax：(010)82109667　E-mail：xuekan＠caas.cn
Powered by Beijing Magtech Co., Ltd.
知识共享许可协议