Criteria, Reliability, and Fairness in Peer Review: A Study of Taiwanese Social Science and Humanities Journals

Peer review is a self-regulation mechanism for scientific inquiry. Institutionalized and incorporated into the structure and operation of science, it has received considerable support in the academic setting. The legitimacy of peer review is based on trust and integrity. In various ways, it allocates scarce resources such as journal space, research funding, faculty recruitment, career advancement, and rewards for academic achievements. But there are growing indications of unresolved deficiencies in the operation of peer review, leading to negative assessments as to whether it is effective, efficient, or reliable. Many studies have found links between potential sources of bias and judgments in peer review, such as institutional prejudice, cronyism, ageism, and conservatism. This study aims to achieve three objectives. First, it explores the origins of peer review and traces the process by which it has become a subject of academic research. This examination shows that peer review has gradually become more open and transparent, has inspired ongoing international exchange, and has embraced diverse approaches for higher-quality evaluation. There is also increased anticipation for mechanisms to be established for continual supervision, scrutiny, and improvement, as well as for a competitive–cooperative relationship to develop between peer review and bibliometrics. Second, this study seeks deeper insights into contemporary research on different assessment criteria and on the reliability and fairness of peer review. While review criteria continue to be a major focus of research, there is also a need for greater investigation of the reasons for low inter-evaluator reliability, doubt has been cast on the methodology of studies on the fairness of peer review, and the effectiveness of peer review remains to be demonstrated. Third, this paper presents an empirical study of peer review reports provided by three social science and humanities journals published in Taiwan. For this purpose, the actual criteria employed are examined and the degree of reliability between evaluators is scrutinized; this is accompanied by a discussion on the issues of fairness and accountability in peer review. The findings include: the core criteria employed by evaluators, as revealed by content analysis of review reports; the main reasons for manuscript acceptance or rejection, as revealed by detailed interpretation of publishing recommendations; and the special attributes of peer review in non-quantitative and humanities research, as revealed by analysis of manuscripts in different research categories and specialized domains. These findings also highlight the need for caution in inferring cause and effect with regard to issues of fairness; for systematic debate with regard to consistency between evaluators’ review comments and their publishing recommendations; for research into the behavior of evaluators; and for consideration of the specific characteristics of manuscript preparation in humanities fields in Taiwan. Overall, academic disciplines are becoming increasingly specialized and complex, and the research population is growing ever larger, yet there has been no great increase in academic resources. In these circumstances, competition will grow increasingly intense, and thus it seems likely that research into peer review will attract ever greater attention. This growing competition is evidenced by the facts that acceptance rates for manuscripts submitted to many leading peer-reviewed journals are in single figures, and that rates of funding allocation by government funding organizations in various countries are also trending downward year by year, falling to a ratio of 1:200 at some renowned institutions. Budgetary reductions and declining numbers of students in higher education have also heightened competition among faculty. Therefore, peer review practice will inevitably face increasing questions and challenges from the outside world. Creating an environment for transparent peer review and establishing a mechanism for continual supervision and scrutiny should be a direction for concerted effort within academia.