Peer review is a topic of considerable concern to many researchers, and there is a correspondingly large body of research on the topic. This issue of Science and Engineering Ethics presents recent work on peer review that is both grounded in empirical science and is applicable to policy decisions. This research raises two basic questions; (a) how does current peer review operate, and (b) how can it be improved? Topics addressed include descriptions of how (...) class='Hi'>peer review is used in Federal agencies. whether peer review leads to better manuscripts, demographic characteristics of authors or reviewers (status or institutional affiliation), blinding of reviewers, authors, or results, reliability and consistency of reviews, accepting a paper before the study is done, simultaneous submission, and use of dispute resolution procedures such as scientific dialectical and pleading protocols. (shrink)

At present, peer review is the most common method used by funding agencies to make decisions about resource allocation. But how reliable, efficient, and fair is it in practice? The ex ante evaluation of scientific novelty is a fundamentally uncertain endeavor; bias and chance are embedded in the final outcome. In the current study, I will examine some of the most central problems of peer review and highlight the possible benefits of using a lottery as an alternative decision-making (...) mechanism. Lotteries are driven by chance, not reason. The argument made in the study is that the epistemic landscape could benefit in several respects by using a lottery, thus avoiding all types of bias, disagreement, and other limitations associated with the peer review process. Funding agencies could form a pool of funding applicants who have minimal qualification levels and then select randomly within that pool. The benefits of a lottery would not only be that it saves time and resources, but also that it contributes to a more dynamic selection process and increases the epistemic diversity, fairness, and impartiality within academia. (shrink)

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected (...) 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated. (shrink)

Supporting young researchers to peer-review and edit journal submissions will also accelerate their training, says Quan-Hoang Vuong. -/- *Citation: -/- Quan-Hoang Vuong. (2022, Sept. 11). Early career researchers can help fix peer review delays. Times Higher Education.

Peer review process helps in evaluating and validating of research that is published in the journals. U.S. Office of Research Integrity reported that data fraudulence was found to be involved in 94% cases of misconduct from 228 identified articles between 1994–2012. If fraud in published article are significantly as high as reported, the question arise in mind, were these articles peer reviewed? Another report said that the reviewers failed to detect 16 cases of fabricated article of (...) Jan Hendrick Schon. Superficial peer reviewing process does not reveals suspicion of misconduct. Lack of knowledge of systemic review process not only demolish the academic integrity in publication but also loss the trust of the people of the institution, the nation, and the world. The aim of this review article is to aware stakeholders specially novice reviewers about the peer review system. Beginners will understand how to review an article and they can justify better action choices in dealing with reviewing an article. (shrink)

Mission-oriented climate change research is often unverifiable. Therefore, many stakeholders look to peer-reviewed climate change research for trustworthy information about deeply uncertain and impactful phenomena. This is because peer-review signals that research has been vetted for scientific standards like reproducibility and replicability. Here we evaluate the transparency of research methodologies in mission-oriented computational climate research. We find that only five percent of our sample meets the minimal standard of fully open data and code (...) required for reproducibility and replicability. The widespread lack of minimal openness standards in peer-reviewed climate change research jeopardizes scientific credibility and risks poor societal outcomes as we manage growing climate risks. The scientific community can and must do better. We, the authors, outline necessary openness standards that we commit to as researchers, reviewers, and readers. We invite you to join us. (shrink)

Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and self-regulate their work. Despite vocal concerns, a closer look at the empirical and methodological limitations of research on bias raises questions about the existence and extent of many hypothesized forms of bias. In addition, the notion of bias is predicated on an implicit ideal that, once articulated, raises questions about (...) the normative implications of research on bias in peer review. This review provides a brief description of the function, history, and scope of peer review; articulates and critiques the conception of bias unifying research on bias in peer review; characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. We close by identifying ways to expand conceptions and studies of bias to countenance the complexity of social interactions among actors involved directly and indirectly in peer review. (shrink)

As universities in the United Kingdom gear themselves up for the next Research Excellence Framework, REF2021, with peer review at its core, we critically re-visit the idea of peer review as a gold standard proxy for research excellence. We question the premise that anonymous peer review is a necessary and enabling condition for impartial, expert judgement. We argue that the intentions and supposed benefits underlying peer review and its associated concepts have become congealed in (...) received discourse about research quality. Hence we explore the key conceptual issues raised by the nested assumptions and concepts that come into play in peer review as currently practised: primarily those of secrecy, anonymity, legitimacy, trust, impartiality and openness. After delineating the benefits attributed to peer review, we contrast its declared virtues with its problematic features. We locate peer review in an audit culture in which the reviewer is an academic labourer. Drawing on recent trends in moral and political philosophy, we question the usefulness of the ideal of impartiality when tied to secrecy. Then we raise more deliberative, intersubjective possibilities for a revised understanding of peer review in the context of an academic community. Finally, we suggest ways in which the academic community could pursue quality in research by recasting peer review to be less secret and more open. (shrink)

BackgroundThe Patient-Centered Outcomes Research Institute is obligated to peer review and to post publicly “Final Research Reports” of all funded projects. PCORI peer review emphasizes adherence to PCORI’s Methodology Standards and principles of ethical scientific communication. During the peer review process, reviewers and editors seek to ensure that results are presented objectively and interpreted appropriately, e.g., free of spin.MethodsTwo independent raters assessed PCORI peer review feedback sent to authors. We calculated the proportion of reports (...) in which spin was identified during peer review, and the types of spin identified. We included reports submitted by April 2018 with at least one associated journal article. The same raters then assessed whether authors addressed reviewers’ comments about spin. The raters also assessed whether spin identified during PCORI peer review was present in related journal articles.ResultsWe included 64 PCORI-funded projects. Peer reviewers or editors identified spin in 55/64 submitted research reports. Types of spin included reporting bias, inappropriate interpretation, inappropriate extrapolation of results, and inappropriate attribution of causality. Authors addressed comments about spin related to 47/55 of the reports.Of 110 associated journal articles, PCORI comments about spin were potentially applicable to 44/110 articles, of which 27/44 contained the same spin that was identified in the PCORI research report. The proportion of articles with spin was similar for articles accepted before and after PCORI peer review.DiscussionJust as spin is common in journal articles and press releases, we found that most reports submitted to PCORI included spin. While most spin was mitigated during the funder’s peer review process, we found no evidence that review of PCORI reports influenced spin in journal articles. Funders could explore interventions aimed at reducing spin in published articles of studies they support. (shrink)

Two important aspects of the relationship between peer review and innovation includes the acceptance of articles for publication in journals and the assessment of applications for grants for the funding of research work. While there are well-known examples of the rejection by journals of first choice of many papers that have radically changed the way we think about the world outside ourselves, such papers do get published eventually, however tortuous the process required. With grant applications the situation differs (...) in that the refusal of a grant necessarily curtails the possible research that may be attempted. Here there are many reasons for conservatism and reservation as to the ability of a grant allocation process based on peer review to deliver truly innovative investigations. Other methods are needed; although such methods need not be applied across the board, they should constitute the methods whereby some 10–20% of the grant monies are assigned. The nomination of prizes for specific accomplishments is one way of achieving innovation although this presumes that investigators or institution already have available the money necessary to effect the innovations; otherwise it is a question of the selection and funding of particular individuals or institutions and requiring them to solve particular problems that are set in the broadest of terms. (shrink)

Pre-publication certification through peer review stands in need of philosophical examination. In this paper, philosopher-psychologist Steven James Bartlett recalls the arguments marshalled four hundred years ago by English poet John Milton against restraint of publication by the "gatekeepers of publication," AKA today's peer reviewers.

IntroductionAllocation of research funds relies on peer review to support funding decisions, and these processes can be susceptible to biases and inefficiencies. The aim of this work was to determine which past interventions to peer review and decision-making have worked to improve research funding practices, how they worked, and for whom.MethodsRealist synthesis of peer-review publications and grey literature reporting interventions in peer review for research funding.ResultsWe analysed 96 publications and 36 website sources. Sixty (...) publications enabled us to extract stakeholder-specific context-mechanism-outcomes configurations for 50 interventions, which formed the basis of our synthesis. Shorter applications, reviewer and applicant training, virtual funding panels, enhanced decision models, institutional submission quotas, applicant training in peer review and grant-writing reduced interrater variability, increased relevance of funded research, reduced time taken to write and review applications, promoted increased investment into innovation, and lowered cost of panels.ConclusionsReports of 50 interventions in different areas of peer review provide useful guidance on ways of solving common issues with the peer review process. Evidence of the broader impact of these interventions on the research ecosystem is still needed, and future research should aim to identify processes that consistently work to improve peer review across funders and research contexts. (shrink)

Peer review is often taken to be the main form of quality control on academic research. Usually journals carry this out. However, parts of maths and physics appear to have a parallel, crowd-sourced model of peer review, where papers are posted on the arXiv to be publicly discussed. In this paper we argue that crowd-sourced peer review is likely to do better than journal-solicited peer review at sorting papers by quality. Our argument rests on two (...) key claims. First, crowd-sourced peer review will lead on average to more reviewers per paper than journal-solicited peer review. Second, due to the wisdom of the crowds, more reviewers will tend to make better judgments than fewer. We make the second claim precise by looking at the Condorcet Jury Theorem as well as two related jury theorems developed specifically to apply to peer review. (shrink)

There is a paucity of studies addressing the nature of the social relations of production prevailing in academia prior to the commodification of academic research. By filling that gap, this paper enables us to better understand the historical presuppositions from which the process of knowledge commodification in academia has evolved. Our theoretically informed analysis will focus on peer review, given that it is one of the few academic practices where traces of that historical past can still be found. (...) On the basis of Marx’s exposition of the main features of pre-capitalist social relations of production in the Grundrisse, it will be concluded that peer review reveals that social relations of production in academia were of a pre-capitalist, or natural-like, nature: peer review is labour of a direct nature; is the practice through which the academic community mediates its own reproduction, determining how – and, sometimes, if – its members enter into contact with the objective conditions of production; finally, in this latter capacity, peer review presupposes that the academic community relates to the conditions of knowledge production as though they were their collective property. These findings are employed to account for the underlying logics of the ongoing ‘functional transformation’ of peer review. (shrink)

Peer review has long been regarded as the gold standard of scientific publication, essential to the integrity of science itself. But, as any publishing scientist knows, peer review has its downside, including long delays and reviewer bias. Until the coming of the Internet, there appeared to be no alternative. Now, articles appear online as preprints almost immediately upon submission. But they lack peer review and thus their scientific standing can be questioned. Post-publication discussion platforms such as PubPeer (...) have proven useful, but are no substitute for pre-publication peer review. Nevertheless, some may be tempted to believe that peer review can now be done without. This article challenges that view by analyzing a recent, non-peer-reviewed article in Skeptical Enquirer, a magazine published by the Committee for Skeptical Enquiry (CSI). The article, “Sodom Meteor Strike Claims Should Be Taken With a Pillar of Salt,” casts doubt on one of the most widely read scientific articles of the last decade and provides a stern warning of the cost of abandoning peer review. (shrink)

In May 2016, we launched Research Integrity and Peer Review, an international, open access journal with fully open peer review (reviewers are identified on their reports and named reports are published alongside the article) to provide a home for research on research and publication ethics, research reporting, and research on peer review. As the journal enters its third year, we reflect on recent events and highlights for the journal and explore how the (...) journal is faring in terms of gender and diversity in peer review. We also share the particular interests of our Editors-in-Chief regarding models of peer review, reporting quality, common research integrity issues that arise during the publishing process, and how people interact with the published literature. We continue to encourage further research into peer review, research and publication ethics and research reporting, as we believe that all new initiatives should be evidence-based. We also remain open to constructive discussions of the developments in the field that offer new solutions. (shrink)

Psychometrically oriented researchers construe low inter-rater reliability measures for expert peer reviewers as damning for the practice of peer review. I argue that this perspective overlooks different forms of normatively appropriate disagreement among reviewers. Of special interest are Kuhnian questions about the extent to which variance in reviewer ratings can be accounted for by normatively appropriate disagreements about how to interpret and apply evaluative criteria within disciplines during times of normal science. Until these empirical-cum-philosophical analyses are done, it (...) will remain unclear the extent to which low inter-rater reliability measures represent reasonable disagreement rather than arbitrary differences between reviewers. (shrink)

Previous research has found that funding disparities are driven by applications’ final impact scores and that only a portion of the black/white funding gap can be explained by bibliometrics and topic choice. Using National Institutes of Health R01 applications for council years 2014–2016, we examine assigned reviewers’ preliminary overall impact and criterion scores to evaluate whether racial disparities in impact scores can be explained by application and applicant characteristics. We hypothesize that differences in commensuration—the process of combining criterion scores (...) into overall impact scores—disadvantage black applicants. Using multilevel models and matching on key variables including career stage, gender, and area of science, we find little evidence for racial disparities emerging in the process of combining preliminary criterion scores into preliminary overall impact scores. Instead, preliminary criterion scores fully account for racial disparities—yet do not explain all of the variability—in preliminary overall impact scores. (shrink)

Despite the presumed frequency of conflicts of interest in scientific peer review, there is a paucity of data in the literature reporting on the frequency and type of conflicts that occur, particularly with regard to the peer review of basic science applications. To address this gap, the American Institute of Biological Sciences conducted a retrospective analysis of conflict of interest data from the peer review of 282 biomedical research applications via several onsite review panels. The overall (...) conflicted-ness of these panels was significantly lower than that reported for regulatory review. In addition, the majority of identified conflicts were institutional or collaborative in nature. No direct financial conflicts were identified, although this is likely due to the relatively basic science nature of the research. It was also found that 65 % of identified conflicts were manually detected by AIBS staff searching reviewer CVs and application documents, with the remaining 35 % resulting from self-reporting. The lack of self-reporting may be in part attributed to a lack of perceived risk of the conflict. This result indicates that many potential conflicts go unreported in peer review, underscoring the importance of improving detection methods and standardizing the reporting of reviewer and applicant conflict of interest information. (shrink)

Blind review is ubiquitous in contemporary science, but there is no consensus among stakeholders and researchers about when or how much or why blind review should be done. In this essay, we explain why blinding enhances the impartiality and credibility of science while also defending a norm according to which blind review is a baseline presumption in scientific peer review.

Recent science funding policy scholars and practitioners have advocated for the use of lotteries, or elements of random chance, as supplementations of traditional peer review for evaluating grant applications. One of the primary motivations for lotteries is their purported openness to innovative research. The purpose of this paper is to argue that current proponents of funding science by lottery overestimate the viability of peer review and thus unduly restrict the scope of lotteries in science funding practice. I (...) further show how this analysis suggests a different way of introducing lotteries into science funding policy. (shrink)

Peer review is a practice used in the evaluation of scientific and scholarly papers in order to select papers for publication in scholarly journals. The practice has also been extended to other domains, such as the evaluation of grant proposals, medical practice, book publication, and even to such areas as teaching evaluation. The primary area that has been of interest to sociologists, however, has been publication in scientific journals. The practice is usually understood to have begun in the seventeenth (...) century in the Royal Society in London, but it has also been claimed that there have been precursors to this practice. In the popular mind, peer review in science is a means of “bullet‐proofing” research, as a Wall Street Journal article once put it, that is to say as a guarantee of quality. But within science it has a very different meaning. (shrink)

Punctuated by joy, disappointments, and conflicts, research evaluation constitutes an intense, emotional moment in scientific life. Yet reviewers and research institutions often expect evaluations to be conducted objectively and dispassionately. Inspired by the scholarship describing the role of emotions in scientific practices, we argue instead, that reviewers actively define, display and manage their emotions in response to the structural organization of research evaluation. Our article examines reviewing practices used in the European Research Council’s (ERC) Starting and (...) Consolidator grants and in the Marie Skłodowska-Curie Action’s (MSCA) Individual Fellowships. These two European funding mechanisms offer different perspectives on the organization of grant evaluation. We conducted interviews with review panel members and analyzed various institutional documents. By drawing on the sociological concepts of feeling rules and emotional work, we demonstrate that reviewers define rules concerning how emotions should be experienced and expressed to ensure the proper functioning of evaluation, and that reviewers experience the need to actively regulate their emotions to comply with these rules. We present four feeling rules concerning the experience and expression of: (1) excitement for novelty during individual evaluation; (2) respect for others’ opinions and the absence of anger in review panels; (3) attentiveness and interest, which are seen as missing in online evaluations. Reviewers also expect ERC candidates to (4) avoid pride and manifest modesty during interviews. These rules demonstrate that proposal peer review is governed by emotional norms, and show the influence of organizational settings and moral requirements on research evaluation. (shrink)

The demand for greater public accountability is changing the nature of ex ante peer review at public science agencies worldwide. Based on a four year research project, this essay examines these changes through an analysis of the process of grant proposal review at two US public science agencies, the National Institutes of Health (NIH) and the National Science Foundation (NSF). Weaving historical and conceptual narratives with analytical accounts, we describe the ways in which these two agencies struggle with (...) the question of incorporating considerations of societal impact into the process of peer review. We use this comparative analysis to draw two main conclusions. First, evaluation of broader societal impacts is not different in kind from evaluation of intellectual merit. Second, the scientific community may actually bolster its autonomy by taking a broader range of considerations into its peer review processes. (shrink)

The way research is, and should be, funded by the public sphere is the subject of renewed interest for sociology, economics, management sciences, and more recently, for the philosophy of science. In this contribution, I propose a qualitative, epistemological criticism of the funding by lottery model, which is advocated by a growing number of scholars as an alternative to peer-review. This lottery scheme draws on the lack of efficiency and of robustness of the peer-review based evaluation to (...) argue that the majority of public resources for basic science should be allocated randomly. I first differentiate between two distinct arguments used to defend this alternative funding scheme based on considerations about the logic of scientific research. To assess their epistemological limits, I then present and develop a conceptual frame, grounded on the notion of “system of practice”, which can be used to understand what precisely it means, for a research project, to be interesting or significant. I use this epistemological analysis to show that the lottery model is not theoretically optimal, since it underestimates the integration of all scientific projects in densely interconnected systems of conceptual, experimental, or technical practices which confer their proper interest to them. I also apply these arguments in order to criticize the classical peer-review process. I finally suggest, as a discussion, that some recently proposed models that bring to the fore a principle of decentralization of the evaluation and selection process may constitute a better alternative, if the practical conditions of their implementation are adequately settled. (shrink)

Irregularities in data/results of scientific research might be spotted pre-publication by co-workers and reviewers, or post-publication by readers typically with vested interest. The latter might consist of fellow researchers in the same subject area who would naturally pay closer attention to a published paper. However, it is increasingly apparent that there are readers who interrogate papers in detail with a primary intention to identify potential problems with the work. Here, we consider post-publication peer review (PPPR) by individuals, or (...) groups of individuals, who perform PPPRs with a perceptible intention to actively identify irregularities in published data/results and to expose potential research fraud or misconduct, or intentional misconduct exposing (IME)-PPPR. On one hand, such activities, when done anonymously or pseudonymously with no formal discourse, have been deemed as lacking in accountability, or perceived to incur some degree of maleficence, and have been labelled as vigilantism. On the other, these voluntary works have unravelled many instances of research misconduct and have helped to correct the literature. We explore the tangible benefits of IME-PPPR in detecting errors in published papers and from the perspectives of moral permissibility, research ethics, and the sociological perspective of science. We posit that the benefits of IME-PPPR activities that uncover clear evidence of misconduct, even when performed anonymously or pseudonymously, outweigh their perceived deficiencies. These activities contribute to a vigilant research culture that manifests the self-correcting nature of science, and are in line with the Mertonian norms of scientific ethos. (shrink)

Purpose This paper aims to appraise current challenges in adopting generative AI by reviewers to evaluate the readability and quality of submissions. The paper discusses how to make the AI-powered peer-review process immune to unethical practices, such as the proliferation of AI-generated poor-quality or fake reviews that could harm the value of peer review. Design/methodology/approach This paper examines the potential roles of AI in peer review, the challenges it raises and their mitigation. It critically appraises current opinions (...) and practices while acknowledging the lack of consensus about best practices in AI for peer review. Findings The adoption of generative AI by the peer review process seems inevitable, but this has to happen (1) gradually, (2) under human supervision, (3) by raising awareness among stakeholders about all its ramifications, (4) through improving transparency and accountability, (5) while ensuring confidentiality through the use of locally hosted AI systems, (6) by acknowledging its limitations such as its inherent bias and unawareness of up-to-date knowledge, (7) by putting in place robust safeguards to maximize its benefits and lessen its potential harm and (8) by implementing a robust quality assurance to assess its impact on overall research quality. Originality/value In the current race for more AI in scholarly communication, this paper advocates for a human-centered oversight of AI-powered peer review. Eroding the role of humans will create an undesirable situation where peer review will gradually metamorphose into an awkward conversation between an AI writing a paper and an AI evaluating that paper. (shrink)

The major challenge facing today’s biomedical researchers is the increasing competition for available funds. The competitive review process, through which the National Institutes of Health (NIH) awards grants, is built upon review by a committee of expert scientists. The NIH is firmly committed to ensuring that its peer review system is fair and objective.

There seems to be reluctance amongst scientists to invest some of their own time in the peer-review of manuscripts. As a result, journal editors often struggle to secure reviewers for a given manuscript in a timely manner. Here, two simple principles are proposed, which could fairly allocate the contribution of individual researchers to the peer-review process.

Peer reviews of 84 organic farming grant applications from Sweden were analyzed to determine whether the reviewers’ affiliation to one of two types of agriculture (i.e., organic and conventional) influenced their reviews. Fifteen reviewers were divided into three groups: (1) scientists with experience in organic farming research; (2) scientists with no experience in organic farming research; and (3) users of organic farming research. The two groups of scientists assessed the societal relevance and scientific quality of the (...) grant applications based on three criteria (i.e., presentation, methodology, qualifications), whereas the user group only assessed societal relevance. The analysis showed that the two groups of scientists provided very different reviews. Scientist reviewers with experience in organic farming research agreed more with the user group on research relevance than did scientist reviewers without such experience, and the assessment of relevance was closely correlated to the assessment of scientific quality within both scientific groups. As both scientific groups did not clearly distinguish between societal relevance and scientific quality, the idea of an objective science is challenged. The contextual values associated with the norms of good agriculture were not clearly distinguished from the constitutive values of science associated with the traditional norms of good science. This raises the question of whether organic and conventional grant applications should be mixed for review regardless of the reviewers. (shrink)

BackgroundThere is increasing need for peer reviewers as the scientific literature grows. Formal education in biostatistics and research methodology during residency training is lacking. In this pilot study, we addressed these issues by evaluating a novel method of teaching residents about biostatistics and research methodology using peer review of standardized manuscripts. We hypothesized that mentored peer review would improve resident knowledge and perception of these concepts more than non-mentored peer review, while improving review quality.MethodsA (...) partially blinded, randomized, controlled multi-center study was performed. Seventy-eight neurology residents from nine US neurology programs were randomized to receive mentoring from a local faculty member or not. Within a year, residents reviewed a baseline manuscript and four subsequent manuscripts, all with introduced errors designed to teach fundamental review concepts. In the mentored group, mentors discussed completed reviews with residents. Primary outcome measure was change in knowledge score between pre- and post-tests, measuring epidemiology and biostatistics knowledge. Secondary outcome measures included level of confidence in the use and interpretation of statistical concepts before and after intervention, and RQI score for baseline and final manuscripts.ResultsSixty-four residents (82%) completed initial review with gradual decline in completion on subsequent reviews. Change in primary outcome, the difference between pre- and post-test knowledge scores, did not differ between mentored (−8.5%) and non-mentored (−13.9%) residents (p = 0.48). Significant differences in secondary outcomes (using 5-point Likert scale, 5 = strongly agree) included mentored residents reporting enhanced understanding of research methodology (3.69 vs 2.61; p = 0.001), understanding of manuscripts (3.73 vs 2.87; p = 0.006), and application of study results to clinical practice (3.65 vs 2.78; p = 0.005) compared to non-mentored residents. There was no difference between groups in level of interest in peer review (3.00 vs 3.09; p = 0.72) or the quality of manuscript review assessed by the Review Quality Instrument (RQI) (3.25 vs 3.06; p = 0.50).ConclusionsWe used mentored peer review of standardized manuscripts to teach biostatistics and research methodology and introduce the peer review process to residents. Though knowledge level did not change, mentored residents had enhanced perception in their abilities to understand research methodology and manuscripts and apply study results to clinical practice. (shrink)

In some recent articles, Dr. Leigh Turner [Doffing the Mask: Why Manuscript Reviewers Ought to Be Identifiable,” Journal of Academic Ethics, 1 (2003), pp. 41–48; “Promoting F.A.I.T.H. in Peer Review: Five Core Attributes in Effective Peer Review,” Journal of Academic Ethics, 1 (2003), pp. 181–188.] makes some rather critical observations regarding the processes of peer-review in academic journals. I shall note them in turn, note wherein I concur and wherein I disagree, and discuss some of Turner's suggestions (...) to resolve such difficulties. It is hoped that my comments on Turner's much-appreciated points will engage readers of this august and well-edited journal to take more seriously Turner's arguments for the sake of the betterment of academic research. (shrink)

BackgroundThe aim of this survey was to determine the level of awareness and understanding of peer review and peer review models amongst junior hospital doctors and whether this influences clinical decision-making.MethodsA 30-question online anonymous survey was developed aimed at determining awareness of peer review models and the purpose of peer review, perceived trustworthiness of different peer review models and the role of peer review in clinical decision-making. It was sent to 800 trainee doctors in (...) medical specialties on the University College London Partners trainee database.ResultsThe response rate was (178/800) 22%. Most respondents were specialist registrars. Checking that research is conducted correctly (152/178, 85%) and the data interpreted correctly (148/178, 83%) were viewed as the most important purposes of peer review. Most respondents were aware of open (133/178, 75%), double-blind (125/178, 70%) and single-blind peer review (121/178, 68%). 101/178 (57%) had heard of collaborative, 87/178 (49%) of post publication and 29/178 (16%) of decoupled peer review. Of those who were aware of double-blind, single-blind open and collaborative peer review, 85 (68%), 82 (68%), 74 (56%) and 24 (24%), respectively, understood how they worked. The NEJM, Lancet and The BMJ were deemed to have most trustworthy peer review, 137/178 (77%), 129/178 (72%) and 115/178 (65%), respectively. That peer review had taken place was important for a journal content to be used for clinical decision-making 152/178 (85%), but the ability to see peer review reports was not as important 22/178 (12%). Most felt there was a need for peer review training and that this should be at the specialist registrar stage of training.ConclusionsJunior hospital doctors view peer review to be important as a means of quality control, but do not value the ability to scrutinize peer review themselves. The unquestioning acceptance of peer review as final validation in the field of medicine emphasises not only the responsibility held by medical journals to ensure peer review is done well but also the need to raise awareness amongst the medical community of the limitations of the current peer review process. (shrink)

The principles and practices of research peer review are described. While the principles are fundamentally generic and apply to peer review across the full spectrum of performing institutions as well as to manuscript/proposal/program peer review, the focus of this paper is peer review of proposed and ongoing programs in federal agencies. The paper describes desireable characteristics and important intangible factors in successful peer review. Also presented is a heuristic protocol for the conduct of successful (...) peer review research evaluations and impact assessments. Problems with peer review are then outlined, followed by examples of peer review of proposed and existing programs in selected federal agencies. Some peer review variants, such as the Science Court, are described, and then research requirements to improve peer review are discussed. (shrink)

Problems in peer review, the backbone of maintaining high standards in scientific publishing, have led to wide spread discontent within the scientific community. Training in the peer review process and a simpler format to assist in decision making are possible courses to improve and expedite the process of peer review and scientific publishing. The authors discuss problems in the peer review process focusing on challenges related to major revisions and reviewer's wish list of experiments; this leads (...) to the loss of time and money. Major revisions rarely lead to significant improvements, while simultaneously delay the dissemination of knowledge. The authors propose simple solutions including training peer reviewers to improve the peer review system. (shrink)

Peer review facilitates quality control and integrity of scientific research. Although publishing policies have adapted to include the use of Artificial Intelligence (AI) tools, such as Chat Generative Pre-trained Transformer (ChatGPT), in the preparation of manuscripts by authors, there is a lack of guidelines or policies on whether peer reviewers can use such tools. The present article highlights the lack of policies on the use of AI tools in the peer review process (PRP) and argues that (...) we need to go beyond policies by creating transparent procedures that will enable journals to investigate allegations of non-compliance and take decisions that will protect the integrity of the peer review system. Reviewers found to violate relevant policies must be excluded from the process to safeguard the integrity of the peer review system. (shrink)

Open peer review (OPR) is moving into the mainstream, but it is often poorly understood and surveys of researcher attitudes show important barriers to implementation. As more journals move to implement and experiment with the myriad of innovations covered by this term, there is a clear need for best practice guidelines to guide implementation. This brief article aims to address this knowledge gap, reporting work based on an interactive stakeholder workshop to create best-practice guidelines for editors and journals who (...) wish to transition to OPR. Although the advice is aimed mainly at editors and publishers of scientific journals, since this is the area in which OPR is at its most mature, many of the principles may also be applicable for the implementation of OPR in other areas (e.g., books, conference submissions). (shrink)

Peer review of manuscripts for biomedical journals has become a subject of intense ethical debate. One of the most contentious issues is whether or not peer review should be anonymous. This study aimed to generate a rich, empirically-grounded understanding of the values held by journal editors and peer reviewers with a view to informing journal policy. Qualitative methods were used to carry out an inductive analysis of biomedical reviewers’ and editors’ values. Data was derived from in-depth, open-ended (...) interviews with journal editors and peer reviewers. Data was “read for” themes relevant to reviewer anonymisation and interactions among editors, reviewers, and authors. Editors and peer reviewers provided three arguments that would support a more open and interactive peer-review process. First, a number of participants emphasised the importance of not only ensuring the scientific quality of published research but also nurturing their colleagues and supporting their communities. Second, many spoke about the ongoing moral responsibilities that reviewers and editors felt toward authors. Finally, participants spoke at length about their enjoyment of social interactions and of the value of collective, rather than isolated, reasoning processes. Whether or not journal editors decide to allow anonymous review , the values of editors and reviewers need to be seriously addressed in codes of publication ethics, in the management of biomedical journals, and in the establishment of journal policies. (shrink)

A growing body of literature has identified potential problems that can compromise the quality, fairness, and integrity of journal peer review, including inadequate review, inconsistent reviewer reports, reviewer biases, and ethical transgressions by reviewers. We examine the evidence concerning these problems and discuss proposed reforms, including double-blind and open review. Regardless of the outcome of additional research or attempts at reforming the system, it is clear that editors are the linchpin of peer review, since they make decisions (...) that have a significant impact on the process and its outcome. We consider some of the steps editors should take to promote quality, fairness and integrity in different stages of the peer review process and make some recommendations for editorial conduct and decision-making. (shrink)

Peer review of manuscripts has recently become a subject of academic research and ethical debate. Critics of the review process argue that it is a means by which powerful members of the scientific community maintain their power, and achieve their personal and communal aspirations, often at others' expense. This qualitative study aimed to generate a rich, empirically‐grounded understanding of the process of manuscript review, with a view to informing strategies to improve the review process. Open‐ended interviews were carried (...) out with 35 journal editors and peer reviewers in the United Kingdom, the USA and Australia. It is clear from this research that relations of power and epistemic authority in manuscript review are complex and dynamic, and may have positive and negative features, and that even where power is experienced as controlling, restrictive and illegitimate, it can also be resisted. In conclusion, the manuscript review process is best thought of not in terms of simple dominance of reviewers and editors over authors, but rather as a shifting “net” of power relations. These complex power relations need to be understood if reviewers are to be encouraged to participate in the process and to do so in the the most ethical and effective manner. (shrink)

This article examines the evolution of peer review and the modern editorial processes of scholarly journals by analyzing a novel data set derived from the Royal Society’s archives and covering 1865-1965, that is, the historical period in which refereeing became firmly established. Our analysis reveals how the Royal Society’s editorial processes coped with both an increasing reliance on refereeing and a growth in submissions, while maintaining collective responsibility and minimizing research waste. By engaging more of its fellows in (...) editorial activity, the society was able to establish an equilibrium of number of submissions per reviewer that was relatively stable over time. Nevertheless, our analysis shows that the distribution of editorial work was significantly uneven. Our findings reveal interesting parallels with current concerns about the scale and distribution of peer review work and suggest the strategic importance of the management of the editorial process to achieve a creative mix of community commitment and professional responsibility that is essential in contemporary journals. (shrink)

The main goal of publicly funded biomedical research is to generate social value through the creation and application of knowledge that can improve the well-being of current and future people. Prioritising research with the greatest potential social value is crucial for good stewardship of limited public resources and ensuring ethical involvement of research participants. At the National Institutes of Health (NIH), peer reviewers hold the expertise and responsibility for social value assessment and resulting prioritisation at the (...) project level. However, previous research has shown that peer reviewers place more emphasis on a study’s methods (‘Approach’) than on its potential social value (best approximated by the criterion of ‘Significance’). Lower weighting of Significance may be due to reviewers’ views on the relative importance of social value, their belief that social value is evaluated at other stages of the research priority-setting process or the lack of guidance on how to approach the challenging task of assessing expected social value. The NIH is currently revising its review criteria and how these criteria contribute to overall scores. To elevate the role of social value in priority setting, the agency should support empirical research on how peer reviewers approach the assessment of social value, provide more specific guidance for reviewing social value and experiment with alternative reviewer assignment strategies. These recommendations would help ensure that funding priorities align with the NIH’s mission and the obligation of taxpayer-funded research to contribute to the public good. (shrink)

Inappropriate behaviors in peer review may involve bullying and can have negative effects on authors and the dissemination of knowledge. In this paper, we have discussed the causes, forms, harms, and potential mitigation measures of bullying in peer review. We call for attention to these behaviors and advocate for future research within a reasonable framework.

Peer review is widely recognized as a mechanism for quality control of academic content. This research article aims at comparing the review reports and decisions of reviewers who are members of the editorial board of the European Scientific Journal with those reviewers suggested by the authors and who are not affiliated with the journal. 457 review reports on 378 papers submitted to the ESJ in the period of October–December 2017 were analysed. Statistical methods including OLS and Wilcoxon rank-sum (...) test were applied based on the score approach toward the reviewers’ assessments of the papers and their characteristics related to the country, gender, and time of revisions. Results show the difference between the decisions these two groups of reviewers made. Even though editor-suggested and author-suggested reviewers need equal time to review a paper, the former are less favourable towards the authors of the papers. It is also concluded that factors such as time and country of the reviewers influence their decisions. In this regard, the editors should avoid relying their decisions solely on review reports received from reviewers suggested by the authors. However, further research with larger sample sizes should be conducted. (shrink)