基于DEA超效率验证的项目同行评议结果有效性研究

doi:10.13998/j.cnki.issn1002-1248.21-0528

农业图书情报学报 ›› 2022, Vol. 34 ›› Issue (2): 88-101.doi: 10.13998/j.cnki.issn1002-1248.21-0528

基于DEA超效率验证的项目同行评议结果有效性研究

万昊¹, 张福俊², 吕千千³

1.山东科技大学图书馆,青岛 266590;
2.山东科技大学发展规划处,青岛 266590;
3.中国科学院文献情报中心,北京 100190

收稿日期:2021-07-08 出版日期:2022-02-05 发布日期:2022-03-24
作者简介:万昊,男,博士,馆员,研究方向为科技评价与创新性评估。张福俊,男,教授,硕士生导师,山东科技大学发展规划处,副处长,研究方向为数据科学与知识可视化技术。吕千千,女,博士研究生,研究方向为科技评估

The Validity of Peer Review Results of DEA Based Super Efficiency Projects

WAN Hao¹, ZHANG Fujun², LV Qianqian³

1. Library of Shandong University of Science and Technology, Qingdao 266590;
2. Development Planning Office, Shandong University of Science and Technology, Qingdao 266590;
3. Document Information Center, Chinese Academy of Sciences, Beijing 100190

Received:2021-07-08 Online:2022-02-05 Published:2022-03-24

摘要/Abstract

摘要： [目的/意义] 本文以项目评审为例探索专家知识为基础的同行评议判断的有效性。[方法/过程] 本文采用数据包络分析测度项目的研发效率,评价模型为经超效率修正的CCR模型,并将该效率指标作为同行评议有效性的验证标准。数据源为国家林业局实施完毕的126项“948”项目,涵盖13项投入与产出评价指标。[结果/结论] 实验发现：项目同行评议结果与实际研发效率（DEA超效率）之间呈现适度的正相关性,Spearman相关系数ρ=0.250（Sig.<0.01）。主观评审犯Ⅰ类错误与Ⅱ类错误的比重分别为17.5%与18.2%,有效同行评议决策比重占64.3%。证实同行评议的确存在一定的系统误差但不可避免。

关键词: 同行评议, 项目评价, 有效性, 超效率DEA

Abstract: [Purpose/Significance] This article uses some projects'peer review results as an example to explore the effectiveness of peer review based on expert knowledge. [Method/Process] It uses data envelopment analysis to measure the R&D efficiency of the projects. The evaluation model is a CCR model modified by super-efficiency, and the efficiency index is used as a verification standard for the effectiveness of peer review results. The data source is 126 "948 projects" implemented by the State Forestry Administration, covering 13 input and output evaluation indicators. [Results/Conclusions] The experiment found that there was a moderate positive correlation between the project peer review results and the actual R&D efficiency (DEA super efficiency), and the Spearman correlation coefficient ρ=0.250 (Sig.<0.01). The percentages of type I errors and type II errors made by subjective review were 17.5% and 18.2%, respectively, and effective peer review decisions accounted for 64.3%. It is confirmed that there are certain systematic errors in peer review, but they are inevitable.

Key words: peer review, project evaluation, effectiveness, super efficiency DEA

中图分类号:

G301

万昊, 张福俊, 吕千千. 基于DEA超效率验证的项目同行评议结果有效性研究[J]. 农业图书情报学报, 2022, 34(2): 88-101.

WAN Hao, ZHANG Fujun, LV Qianqian. The Validity of Peer Review Results of DEA Based Super Efficiency Projects[J]. Journal of Library and Information Science in Agriculture, 2022, 34(2): 88-101.

参考文献

[1] PRICE D J D. Little science, big science-and beyond[M]. New York: Columbia university press, 1986.
[2] 万昊, 谭宗颖, 朱相丽. 同行评议与文献计量在科研评价中的作用分析比较[J]. 图书情报工作, 2017, 61(1): 134-152.
WAN H, TAN Z Y, ZHU X L.Comparison of the role of peer review and bibliometrics in research evaluation[J]. Library and informationservice, 2017, 61(1): 134-152.
[3] MERTON R K.Social theory and social structure[M]. New York: Free press, 1968.
[4] 姜春林, 张立伟, 刘学. 中外同行评议研究现状及问题探讨[J]. 科技管理研究, 2015(3): 163-166.
JIANG C L, ZHANG L W, LIU X.The status quo of peer review research and its problems[J]. Science and technology management research, 2015(3): 163-166.
[5] 盛怡瑾, 初景利. 同行评议质量控制方法研究进展[J]. 出版科学,2018, 26(5): 48-55.
SHENG Y J, CHU J L.Research progress of peer review quality control method[J]. Publishing journal, 2018, 26(5): 48-55.
[6] CARTER B.Peer review a good but flawed system[J]. Journal of child health care, 2017, 21(3): 233-235.
[7] WESSELY S, WOOD F.Peer review of grant applications: A systematic review[M]//, GODLEE F, JEFFERSON T. Peer review in health sciences, London: British medical journal books, 1999.
[8] MAVIS B, KATZ M.Evaluation of a program supporting scholarly productivity for new investigators[J]. Academic medicine, 2003, 78(7): 757-765.
[9] BORNMANN L, DANIEL H D.Convergent validation of peer review decisions using the h index. Extent of and reasons for type I and type II errors[J]. Journal of informetrics, 2007, 1(3): 204-213.
[10] BORNMANN L, WALLON G, LEDIN A.Does the committee peer review select the best applicants for funding? An investigation of the selection process for two european molecular biology organization programmes[J]. PLoS ONE, 2008, 3(10): E3480.
[11] NEUFELD J, VON INS M.Informed peer review and uninformed bibliometrics?[J]. Research evaluation, 2011, 20(1): 31-46.
[12] LI D, AGHA L.Big names or big ideas: Do peer review panels select the best science proposals?[J]. Science, 2015, 348(6233): 434-438.
[13] MELIN G, DANELL R.The top eight percent: Development of approved and rejected applicants for a prestigious grant in Sweden[J]. Science and public policy, 2006, 33(10): 702-712.
[14] HORNBOSTEL S, BOHMER S, KLINGSPORN B, et al.Funding of young scientist and scientific excellence[J]. Scientometrics, 2009,79(1): 171-190.
[15] VAN DEN BESSELAAR P, LEYDESDORFF L. Past performance, peer review, and project selection: A case study in the social and behavioral sciences[J]. Research evaluation, 2009, 18(4): 273-288.
[16] BORNMANN L, LEYDESDORFF L, VAN DEN BESSELAAR P. A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications[J]. Journal of informetrics, 2010, 4(3): 211-220.
[17] BUXTON M, HANNEY S, MORRIS S, et al.Medical research - What's it worth? Estimating the economic benefits from medical research in the UK[R]. Health economics research group, office of health economics & rand, Europe, London: UK evaluation forum, 2008.
[18] MACILWAIN C.Science economics: What science is really worth?[J]. Nature, 2010, 465(10): 682-684.
[19] 王海宁, 李姗姗, 栾贞增. 高校科研成果转化能力与效率关系评价——基于2016年教育部直属61所高校的实证研究[J]. 科技管理研究, 2018, 406, 38(12): 147-154.
WANG H N, LI S S, LUAN Z Z. Evaluation on the relationship between transformation ability and efficiency of scientific research achievements in colleges and universities: An empirical study of 61 universities directly under the chinese ministry of education in2016[J]. Science and technology management research, 2018, 406,38(12): 147-154.
[20] HAZELKORN E.European commission: Expert group on assessment of university-based research, assessing Europe's university based research[R]. Dublin institute of technology, centre for social and educational research, 2010.
[21] IVANOVA I A, LEYDESDORFF L.Knowledge-generating efficiency in innovation systems the acceleration of technological paradigm changes with increasing complexity[J]. Technological forecasting &social change, 2015, 96: 254-265.
[22] SALTER A J, MARTIN B R.The economic benefits of publicly funded basic research: a critical review[J]. Research policy, 2001,30(3): 509-532.

基于DEA超效率验证的项目同行评议结果有效性研究

The Validity of Peer Review Results of DEA Based Super Efficiency Projects

PDF (PC)

赞

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 6

Metrics

本文评价

推荐阅读 0

[1]	霍振响, 屈李纯, 李小平. 基于科技期刊编辑视角探讨抵制学术不端行为[J]. 农业图书情报学刊, 2018, 30(7): 137-140.
[2]	卫芸. 提高微信公众平台消息推送有效性的策略研究[J]. 农业图书情报学刊, 2018, 30(1): 120-123.
[3]	李祥敏, 许素, 陈昕, 唐晓燕, 陶文绮, 王昌群. 基于专家审稿意见的稿件取舍问题实例分析[J]. 农业图书情报学刊, 2017, 29(12): 159-163.
[4]	骆天阳, 李志刚. 基于DEA的高校图书馆效率评价及研究[J]. 农业图书情报学刊, 2015, 27(5): 9-13.
[5]	施婧. 基于KPI的高校图书馆绩效考核指标研究[J]. 农业图书情报学刊, 2014, 26(4): 67-70.
[6]	张琼，王芳，程平，何婧，胡启平，马建荣，戴俊生，李朝晖. 树立品牌意识，突出地域特色——《新疆农业科学》办刊实践与成绩[J]. 农业图书情报学刊, 2014, 26(2): 145-150.