Despite the clear link between reciprocity and trust building, the majority of researchers invest little time in reciprocal relationships beyond their disciplinary networks. This remains one of the most powerful ways researchers can build trusting, impactful relationships beyond the academy.

Trust is necessary for research impact because it enables people to co-operate without the need for contracts, non-disclosure agreements and, other cumbersome arrangements, reducing complexity and facilitating efficient collaboration.

We need to shift our focus from amassing more and more knowledge about the problems the world is facing, to devising and testing solutions that might tackle the underlying drivers of the problems we have studied for so long.

There is also growing pressure on researchers to make “policy recommendations” from single studies, whether in response to journal editors and reviewers who want the research to be more widely read (and cited) or funders who want to see impacts from their investment. However, while there is growing recognition that such recommendations should only be made on the basis of evidence synthesis, there are limited incentives from funders or universities to prioritize synthesis work over conducting new original research.

To create a more inclusive impact culture, in which all researchers can feel valued and find deeper meaning in their work, it is important to create opportunities for researchers to engage with impact authentically, on their own terms, in ways that are consistent with their unique purpose, identity, and values, and hence build their intrinsic motivation, rather than building yet more extrinsic incentives to push colleagues toward impact.

negative attitudes held toward the Research Excellence Framework, which assesses the impact of UK research (Weinstein et al., 2021).

As such, the transformation of universities to become purpose-driven, rather than being driven by the impact agenda, is an opportunity for universities to enable researchers to find their own purpose as much as it is an opportunity to connect with the purpose of the university or the stakeholders it seeks to serve.

the fewest possible negative unintended consequences for stakeholders and publics, and for researchers and their institutions/funders

However, we argue that participatory change from the bottom up is more likely to achieve meaningful and lasting change in the practices and behaviors of researchers, and so deliver impacts that are consistent with their values, beliefs, and norms, maintaining the motivation of researchers as they address twenty first century challenges.

However, it has less to say about the drivers of impact culture (including the role of researchers, stakeholders, and institutional co-ordination in constructing impact culture) or the ways in which universities may need to transform their operating models in response to these drivers, to facilitate more healthy impact cultures that are more likely to tackle twenty first century challenges.

Similarly, Chubb and Reed (2018), based on interviews with UK and Australian researchers, heard stories of researchers who had stopped asking the questions they thought were most important, because they were not impactful enough to be funded

Indeed, 75% of those responding to the Wellcome Trust survey felt their creativity was being stifled by an “impact agenda” that was increasingly driving their research (Moran et al., 2020)

These assessments have led to a narrowing of the types of knowledge and impact that are valued and deemed legitimate (de Lange et al., 2010; Smith et al., 2011; Parker and Van Teijlingen, 2012)

A powerful driver of reliable knowledge is science’s incentivizing of self-correction.

Science’s culture of critique discourages groupthink, countermands the effects of human biases, and protects knowledge, not only by rewarding a dispassionate stance toward the subject and institutionalizing organized skepticism but also by fostering competition among scientists able to replicate and hence challenge each other’s work.

Science is trustworthy in part because it honors its norms. Adherence to these norms increases the reliability of the resulting knowledge and the likelihood that the public views science as reliable. A 2019 survey (Fig. 1) found that the public recognizes key signals of the trustworthiness of scientific findings. Specifically, in deciding whether to believe a scientific finding, 68% said it matters whether the scientists make their data and methods available and are completely transparent about their methods; 63% reported that it matters whether the scientists involved in the study disclose the individuals and organizations that funded their work; and 55% indicated that it matters whether the study has been published in a peer-reviewed science journal.

Applications of metrics in HSS contexts, especially against the backdrop of efforts like the Research Excellence Framework (REF) in the United Kingdom, have generated considerable skepticism.

First, the potential to replicate and synthesise studies to provide reliable evidence of what works at national or system levels to inform wider policy and practice is compromised by a lack of common standards for collecting and reporting data

anthropocentric focus on “economy, society and/or culture” to the apparent exclusion of environmental impacts, non-human beneficiaries and future generations

Randomised controlled trials sit at the top of this notional hierarchy, followed by quasi-experiments, mixed methods and qualitative methods. Implicit in this hierarchy is the idea that quantitative measures are superior to qualitative approaches. This hierarchy may be valid in the evaluation of some types of impact in certain contexts, for example where it is possible to isolate and evidence the sole cause (e.g. an intervention based on research) of any given effect (the impact).

we would advise institutions to emphasize the spirit, rather than the letter, of any guidance they publish.

Virtual peer review using videoconferencing or teleconferencing appears promising for reducing costs by avoiding the need for reviewers to travel, but again any consequences for quality have not been adequately assessed

These studies showed that simplifying the process by shortening proposal forms, using smaller reviewer panels, or expediting processes can speed up the review process and reduce costs, but this might come at the expense of peer review quality, a key aspect that has not been assessed.

Even in the fairly barren landscape of evidence we explored, it was startling that we could find no studies examining the social processes that occur during panel discussions – a central part of the peer review process. Such studies will clearly be challenging and require the cooperation of funders working in concert, but we feel are essential to understand how to optimise one of the fundamental processes of science.

Technology provides ways to reduce the time burden of the peer review process for panel members and funders - for example by eliminating travel - and does not appear to significantly affect the outcomes. However, face-to-face discussion of applications brings other side-benefits, including social interaction and network formation, other research suggests these side-benefits may be important to the progress of science and hence may need to be supported in other ways if peer review is done remotely.

Barnett et al. (2015) examined the effect of reducing the complexity of the application. Surprisingly, they found that reducing application complexity slightly increased preparation time. They suggest that this may be because researchers allocate a fixed fraction of their time to application preparation

Research Councils UK: Efficiency and Effectiveness of Peer Review. 2006. Reference Source

There is a striking disconnect between the institutional and community support for the peer review system and the empirical evidence of its effectiveness – unfortunately the scope of our review excluded the types of research that might explain this divergence.

This suggests that peer review processes may not work well for transdisciplinary teams integrating both academic and non-academic experts.

If peer review is reliable, the judgements of different peer reviewers on the same proposal should be highly correlated

If inconsistency stems from discrepancies in review quality (which is by no means clear), it might be feasible to evaluate the quality of reviews, although this approach has its own challenges – for example, what is a ‘good’ review? If a review is not consistent with other review does that intrinsically make it ‘bad’? It could be the outlier picking up on the true potential of an innovative application.

Two funders have experimented with, and evaluated, virtual peer review both by teleconference and through the use of Second Life, a virtual world. NIH estimated that using Second Life telepresence, peer review could cut panel costs by one third (Bohannon, 2011). Pier et al. (2015) compared videoconference and face-to-face panels. They set up one videoconference and three face-to-face panels modelled on NIH review procedures, concluding that scoring was similar between face-to-face and videoconference panels. Both the Bohannon and Pier studies of virtual panels noted that participants valued the social aspects of meeting in person and preferred the face-to-face arrangements.Gallo et al. (2013) examined four years of peer review discussions, two years face-to-face and two years teleconferencing. They found minimal differences in merit score distribution, inter-rater reliability or reviewer demographics. They also noted that panel discussion, of any type, affects the funding decision for around 10 per cent of applications relative to original scores.

Bibliometric analyses are by no means perfect measures of performance – only capturing a proxy of academic performance (Belter, 2015). Nonetheless, the findings suggest that peer review assessment is, at best, a crude predictor of performance.

Table 3. Summary of evidence from the literature regarding the effectiveness and burden of peer review.

(Ismail et al. (2009) estimated that >95% of UK medical research funding was allocated by peer review).

Good evidence shows that the burden of peer review is high and that around 75% of it falls on applicants. By contrast, many of the efforts to reduce burden are focused on funders and reviewers/panel members.

The authors declare they have no competing interests.

Previous research has reported the importance of score-calibration comments and even laughter in the effectiveness of panel discussion, although it is unclear if these are affected by chair facilitation

respondents who felt the outcome was affected by panel discussions viewed the discussion effectiveness and quality more favorably than those who felt the outcome was not affected by the discussions.

So, one grant with an excellent score by one reviewer and a mediocre score by the other ends up with a score outside the fundable range. And if the average of the scores places it out of the range to be discussed, the panel usually just lets it go so as to not increase the length of the meeting. So, a grant that one reviewer rated favorably can get torpedoed by a bad reviewer, even if that reviewer was totally off-base on their reasons for the bad score.

Biases may be exacerbated by short discussion times

However, some respondents felt the distribution of comments from panel members and their relative weight on panel opinion was not even across reviewers and sometimes dependent on whether the assessment was positive or negative:“My experience has been that folks do not use the discussion time most effectively. The primary reviewer is accepted at face value and when a dissenting opinion is voiced, the panel seems to be reluctant to discuss and rather defers to the primary reviewer. This is a flaw that makes the review only as good as the thoroughness of ONE reviewer. And in my limited experience I have seen some shoddy reviewing. This is unfair to the team that prepares the application.”“On the whole, I felt most panel members were too polite and unwilling to offer frank opinions of weak proposals.”

the magnitude of scoring shifts after discussion are typically relatively small and are even smaller (with shorter discussion times) in teleconference panels compared to face-to-face panels

to mitigate this subjectivity

their perceptions

This suggests that “excellence” resides between different communities and is ill-structured/defined in each context. Local groups and disciplines may have their own more specific (though sometimes conventional rather than explicit) measures of “excellence”: Biologists may treat some aspects of performance as “excellent” (for example, number of publications, author position, citations counts), while failing to recognize aspects considered equally or more “excellent” by English professors (large word counts, single authorship, publication or review in popular literary magazines and journals) (O’Donnell, 2015).

Policymakers, who wanted evidence of impact to demonstrate return on investment

This engagement is increasingly seen as important; for example, it represents 20% of the evaluation in the next version of the Research Excellence Framework (REF) in the UK.

Only metrics should be used in research evaluation when they reflect the quality of papers

We looked for factors that could explain this. In addition to international collaboration, scientific mobility was expected to contribute to impact4. So we also considered new researchers coming in, returnees and emigrating researchers, all of which are tracked by the Organisation for Economic Co-operation and Development (OECD). These variables, together with collaboration, proved to be highly correlated as measures of international engagement; so we used them to create an index of openness and were able to assign values to 33 of the countries that we looked at (data available at go.nature.com/2fzrnt3).

When aggregating article and citation counts, an integer counting method was employed where, for example, a paper with two authors from a UK address and one from a French address would be counted as one article for each country (i.e. 1 UK and 1 France)

Thus funding is apportioned by the U.K.’s Higher Education Funding Councils. In the academic year 2013–2014, the budget added up to ~£1.68 billion

The results assist HEIs investment and financial strategy by informing their potential income generation through performance-based research funding. For example, an HEI can recruit more academics in the Engineering department so as to increase their chances of acquiring funding. Hence, the investment decision can influence the future of science.

Interdisciplinary research sits in two different departments. For example, information sciences either come under UoA 11 or UoA 36. As UoA 11 offers higher income for good research outputs, HEIs which have submitted their information sciences research in UoA 36 may consider submitting in UoA 11 for the next exercise.

It informs an HEI the amount of funding an academic can bring into the department.

the perspectives on quality of research is either similar or superior to national research assessment exercises

The findings indicate that the outcome of judgements made on the quality of research either by peer-reviewed government ranking (REF results) and metrics-based ranking (JIF) largely remain the same in disciplines where journals are considered the main channels of research communication

All the outputs submitted by multiple UK HEIs in the chosen UoAs were filtered for journal article submissions and the impact rating of every article’s journal was mapped using Thomson Reuters’ Journal Citation Reports

A potential method to recognise quality of an output is to observe the quality or rank of its journal based on the journal impact factor.

this is the first time an investigation has estimated the monetary value of a good research outpu

monetary value

The financial awards for outputs in the remaining subject areas

HEFCE’s funding allocations

Subsequently, all the journal articles submitted by the HEIs, whose journals were in the Q1 category were considered, allowing the calculation of all the HEIs percentage of Q1 publications, which was compared against percentage of 4* publications.

However, since it was not possible to get the necessary data directly from the REF2014 results, i.e. it was not possible to find out which output got a 4* rating, we decided to use an alternative approach. By using the Thompson Reuter’s Journal Citation Report against the submitted journal papers for each HEI in the chosen UoA, we identified how many of the submitted articles were in top quartile journals, and accordingly we prepared a rank list of the HEIs in a given UoA based on the number of Q1 publications. This list was plotted against their percentage of 4* outputs to find any relationship.

as indicators of quality

1052

An HEI lost about 17.1% (£14.2 million) of its funding and in another exceptional case, an HEI lost 45% of its funding. The maximum gain by any HEI stood at 12.4% (£7.1 million) [13, 14]. The repercussions of such fluctuations are considerable to the future of research at UK HEIs

It may be argued that the amount of money available to distribute to the HEIs is very much dependent on the available budget for a particular government, and hence the monetary value of the research outputs will not provid

According to the Higher Education Statistics Agency (HESA), a typical UK HEI’s revenue break-down is as follows: 35% tuition fee, 30% funding council grants, 16% research grants and contracts, 1% from endowments/ investment income, and 18% from other sources i.e. alumni donations etc

Resetting the trajectory of the scientific enterprise requires multiple, difficult actions. Politicians must understand that job creation is not — and furthermore, should not be — a primary goal of the NIH or any other science-funding agency. Funds should be distributed on the basis of merit alone, not geopolitical considerations and interests. Institutions need to realign their mentality with their original academic mission, and reduce soft-money positions. Publishers should care less about publishing flashy stories and more about disseminating solid science. Individual scientists should emphasize excellence and rigour over stockpiling more and more papers and grants.

But more money is not enough to create better science. Indeed, the scientific enterprise has been betrayed by the mismanagement of its financial support.

there is some evidence to suggest that double blind review can increase the acceptance rate of women-authored articles in the published literature (Darling, 2015).

announcement of author’s career stage has potential consequences on their reputation

the making of the sausage, so to speak

This is likely dependent on whether research communities themselves choose to embrace any such crediting or accounting systems for peer review.

suggesting that the process could become commercialized as this domain rapidly evolves

Hence, this survey could represent a biased view of the actual situation

market-rate reimbursement

it is important to note that commercial publishers, including Wiley, have a vested interes

commercial

A vast majority of researchers see peer review as an integral and fundamental part of their work. They often even consider peer review to be part of an altruistic cultural duty or a quid pro quo service, closely associated with the identity of being part of their research community.

The inertia hindering widespread adoption of new models of peer review can be ascribed to what is often termed “cultural inertia” within scholarly research.

under an open license

Instead, reviewers are asked to focus on whether the research was conducted properly and that the conclusions are based on the presented results.

this uncertainty in standards and implementation can potentially lead to, or at least be viewed as the cause of, widespread failures in research quality and integrity

However, beyond editorials, there now exists a substantial corpus of studies that critically examines the technical aspects of peer review. Taken together, this should be extremely worrisome, especially given that traditional peer review is still viewed almost dogmatically as a gold standard for the publication of research results, and as the process which mediates knowledge dissemination to the p

This is especially the case as the (scientific) scholarly community appears overall to have a strong investment in a “creation myth” that links the beginning of scholarly publishing—the founding of The Philosophical Transactions of the Royal Society—to the invention of peer review. The two are often regarded to be coupled by necessity, largely ignoring the complex and interwoven history of peer review and publishing. This has consequences, as the individual identity as a scholar is strongly tied to specific forms of publication that are evaluated in particular ways (Moore et al., 2017). A scholar’s first research article, PhD thesis, or first book are significant life events. Membership of a community, therefore, is validated by the peers who review this newly contributed work. Community investment in the idea that these processes have “always been followed” appears very strong, but ultimately remains a fallacy.

Here, e-prints or pre-prints are not formally peer reviewed prior to publication, but still undergo a certain degree of moderation in order to filter out non-scientific content. This practice represents a significant shift, as public disse

Such a consequence is plausibly related to, or even a consequence of, broader shifts towards a more competitive neoliberal academia and society at large

soundness

for instance, space in peer reviewed print publication venues, research time at specialized research facilities, or competitive research funds.

expected impact

For the purposes of this article, we are exclusively addressing peer review in the context of manuscripts for research articles, unless specifically indicate

One problem is that the time it takes to produce a monograph may extend beyond the REF period, so in order to be REF-able, historians also need to be publishing articles that can be written relatively quickly allowing more to be produced in the same timeframe.

“now I target management journals, which is one way of hitting a four star”

For example, some key target journals in marketing moved down the ABS rankings in 2015, yet the pressure on academics to “hit a four-star” remained.

For example, during probationary periods, new lecturers were required to publish certain numbers of papers of a specified quality. For academics working in marketing, quality was determined using the Chartered Association of Business Schools’ (ABS) annually published Academic Journal Guide, which ranks business and management journals on a star-rating system similar to that employed by the REF.

Scientists whose output has been endorsedto a high degree will have more AEP to usefor endorsements, and thus have a largerin uence in the community

Scientic projects often take yearsfrom start to publication. Consequently,there is an inherent time lag in using cita-tions to appraise scientic work

Towards Open Science: The Case for a DecentralizedAutonomous Academic Endorsement System

Lots of criticism of publishers points out that “academics work for them for free”. Some do, if they do not have positions. Others, though, actually work for institutions (universities) and are paid by them. Their contracts often stipulate that they should participate in a research ecosystem that includes reviewing. So, actually, universities are often the ones donating time – in the form of their paid labour-force – to reviewing. There is a real flaw, then, in suggesting that reviewers should be paid because often they already are. If this really is to work, I’d suggest that: a. researchers without affiliation could be paid personally; but b. in the case of researchers with a permanent university position, the funds should go back to the university’s own open-access fund. The problem with point b is that it then doesn’t give academics the price sensitivity that is desired for their own labour.

it would not have given him an opportunity to consider the needs of other aspects of policy-making that are less about the future development of novel products and processes, but more focused on analysis of current knowledge to address political problems or on horizon scanning of future social, economic, environmental and technical developments, which might have political significance

I will contend that, in making this a commercial transaction, government scientists took on a lot of contract management work at the expense of analysis and advice, while the research community became more focused on contractual delivery than dialogue with users.

A dearth of clear, relevant and reliable research evidence, and difficulty finding and accessing it, were the main barriers to the use of research.

in contrast with the published evidence on research impact (which depicts much as occurring indirectly through non-linear mechanisms), this sub-panel seems to have captured mainly direct and relatively short-term impacts one step removed from patient outcomes.

Researchers and citizens prioritized high-quality scientific outcomes (papers, trainees) over other attributes. Both groups disvalued research targeted at economic priorities relative to health priorities. Researchers granted a premium to proposals targeting scientific priorities.