Despite the clear link between reciprocity and trust building, the majority of researchers invest little time in reciprocal relationships beyond their disciplinary networks. This remains one of the most powerful ways researchers can build trusting, impactful relationships beyond the academy.
Important insight. This is an argument for rewarding/incentivising the process of impact generation as well as its outcomes (See also normal vs extraordinary impact https://doi.org/10.1093/reseval/rvz032).
Trust is necessary for research impact because it enables people to co-operate without the need for contracts, non-disclosure agreements and, other cumbersome arrangements, reducing complexity and facilitating efficient collaboration.
This is true, but I suspect there are additional linkages between trust in research and impact, especially in the health sphere or where people are asked to make (impactful) changes to their behaviour on the basis of insights from research.
We need to shift our focus from amassing more and more knowledge about the problems the world is facing, to devising and testing solutions that might tackle the underlying drivers of the problems we have studied for so long.
While I strongly agree with this, there are those who would defend research as 'criticism' without necessarily the need to identify or test solutions. See https://doi.org/10.1111/area.12574
There is also growing pressure on researchers to make “policy recommendations” from single studies, whether in response to journal editors and reviewers who want the research to be more widely read (and cited) or funders who want to see impacts from their investment. However, while there is growing recognition that such recommendations should only be made on the basis of evidence synthesis, there are limited incentives from funders or universities to prioritize synthesis work over conducting new original research.
This is an important point. Is there evidence of the 'growing pressure' or the lack of incentives for synthesis work? Policy-makers are looking for nuanced reviews of the evidence and the extent to which a single study contributes to policy recommendations is very dependent on the study and the policy area.
To create a more inclusive impact culture, in which all researchers can feel valued and find deeper meaning in their work, it is important to create opportunities for researchers to engage with impact authentically, on their own terms, in ways that are consistent with their unique purpose, identity, and values, and hence build their intrinsic motivation, rather than building yet more extrinsic incentives to push colleagues toward impact.
The question of conflicting purposes needs to be addressed.
negative attitudes held toward the Research Excellence Framework, which assesses the impact of UK research (Weinstein et al., 2021).
Does Weinstein et al. allow the contribution of impact specifically to the reported attitudes? In addition, a key finding was the diversity of attitudes, not just negative.
As such, the transformation of universities to become purpose-driven, rather than being driven by the impact agenda, is an opportunity for universities to enable researchers to find their own purpose as much as it is an opportunity to connect with the purpose of the university or the stakeholders it seeks to serve.
How can the potential conflicts of purpose between groups be resolved?
the fewest possible negative unintended consequences for stakeholders and publics, and for researchers and their institutions/funders
Is there potential for conflicting objectives here? Limiting negative consequences for some stakeholders might introduce them for others.
However, we argue that participatory change from the bottom up is more likely to achieve meaningful and lasting change in the practices and behaviors of researchers, and so deliver impacts that are consistent with their values, beliefs, and norms, maintaining the motivation of researchers as they address twenty first century challenges.
I agree with the principle here, but there is logical leap at the end of the sentence - how confident can we be that researchers' values, beliefs, norms will always be in alignment with what is needed by society.
However, it has less to say about the drivers of impact culture (including the role of researchers, stakeholders, and institutional co-ordination in constructing impact culture) or the ways in which universities may need to transform their operating models in response to these drivers, to facilitate more healthy impact cultures that are more likely to tackle twenty first century challenges.
Also not clear whether the three stages represent more or less desirable cultures. It may be that different cultures are best suited to tackling particular challenges/issues, and so the ideal is a dynamic between the stages rather than valuing one over the others.
Similarly, Chubb and Reed (2018), based on interviews with UK and Australian researchers, heard stories of researchers who had stopped asking the questions they thought were most important, because they were not impactful enough to be funded
To an extent the interpretation depends on the framing of 'important' and 'impactful'. If 'important' means of purely academic influence, then the described behaviour is consistent with the policy intent to deliver more impact from research.
Indeed, 75% of those responding to the Wellcome Trust survey felt their creativity was being stifled by an “impact agenda” that was increasingly driving their research (Moran et al., 2020)
Important to note the limitations of the Moran et al. study; see referee report
These assessments have led to a narrowing of the types of knowledge and impact that are valued and deemed legitimate (de Lange et al., 2010; Smith et al., 2011; Parker and Van Teijlingen, 2012)
An interesting question is the extent to which there is evidence to back up this assertion. The cited references are more than a decade old and in a real sense predate the emphasis on impact. The first focuses on the Australian ERA, which didn't then have a focus on impact and the other two look at REF in advance of submission.
A powerful driver of reliable knowledge is science’s incentivizing of self-correction.
Also does not critically engage with this statement (e.g. evidence on propagation of negative/positive findings through citations).
Science’s culture of critique discourages groupthink, countermands the effects of human biases, and protects knowledge, not only by rewarding a dispassionate stance toward the subject and institutionalizing organized skepticism but also by fostering competition among scientists able to replicate and hence challenge each other’s work.
No critical engagement with the question of whether these aspirations actually play out in practice.
Science is trustworthy in part because it honors its norms. Adherence to these norms increases the reliability of the resulting knowledge and the likelihood that the public views science as reliable. A 2019 survey (Fig. 1) found that the public recognizes key signals of the trustworthiness of scientific findings. Specifically, in deciding whether to believe a scientific finding, 68% said it matters whether the scientists make their data and methods available and are completely transparent about their methods; 63% reported that it matters whether the scientists involved in the study disclose the individuals and organizations that funded their work; and 55% indicated that it matters whether the study has been published in a peer-reviewed science journal.
This paragraph and the associated figure are subject to a correction. See https://www.pnas.org/content/117/14/8212
Applications of metrics in HSS contexts, especially against the backdrop of efforts like the Research Excellence Framework (REF) in the United Kingdom, have generated considerable skepticism.
Implies the authors do not understand the UK REF which explicitly does not evaluate on the basis of metrics.
First, the potential to replicate and synthesise studies to provide reliable evidence of what works at national or system levels to inform wider policy and practice is compromised by a lack of common standards for collecting and reporting data
Related to questions of internal vs external validity. Even with standardisation then it may not be possible to translate from one context to another.
There is also the issue of the relationship between pilots and effects that might only be manifest at whole system level.
anthropocentric focus on “economy, society and/or culture” to the apparent exclusion of environmental impacts, non-human beneficiaries and future generations
Environmental impacts are explicitly included in the definition
Randomised controlled trials sit at the top of this notional hierarchy, followed by quasi-experiments, mixed methods and qualitative methods. Implicit in this hierarchy is the idea that quantitative measures are superior to qualitative approaches. This hierarchy may be valid in the evaluation of some types of impact in certain contexts, for example where it is possible to isolate and evidence the sole cause (e.g. an intervention based on research) of any given effect (the impact).
Link to debates about RCTs and hierarchies of evidence (external vs internal validity, risk of choosing measurable (but inappropriate) end points.
we would advise institutions to emphasize the spirit, rather than the letter, of any guidance they publish.
This is practically very difficult. In a large organisation it is pretty much impossible to achieve consistency and fairness by implementing the 'spirit'. There is a trade-off between flexibility and fairness.
Virtual peer review using videoconferencing or teleconferencing appears promising for reducing costs by avoiding the need for reviewers to travel, but again any consequences for quality have not been adequately assessed
Evidence on virtual peer review panels via video conferencing.
These studies showed that simplifying the process by shortening proposal forms, using smaller reviewer panels, or expediting processes can speed up the review process and reduce costs, but this might come at the expense of peer review quality, a key aspect that has not been assessed.
This review includes evidence on the trade-off between simplicity and quality in peer review.
Even in the fairly barren landscape of evidence we explored, it was startling that we could find no studies examining the social processes that occur during panel discussions – a central part of the peer review process. Such studies will clearly be challenging and require the cooperation of funders working in concert, but we feel are essential to understand how to optimise one of the fundamental processes of science.
This is a striking observation, but there are some studies that attempt to understand the inner workings of the process (e.g. Lamont 'How professors think' and the work of Gemma Derrick).
Technology provides ways to reduce the time burden of the peer review process for panel members and funders - for example by eliminating travel - and does not appear to significantly affect the outcomes. However, face-to-face discussion of applications brings other side-benefits, including social interaction and network formation, other research suggests these side-benefits may be important to the progress of science and hence may need to be supported in other ways if peer review is done remotely.
Important observation in the context of remote peer review, although important to separate the delivery of the process of peer review from the potential side benefits of interactions.
Barnett et al. (2015) examined the effect of reducing the complexity of the application. Surprisingly, they found that reducing application complexity slightly increased preparation time. They suggest that this may be because researchers allocate a fixed fraction of their time to application preparation
This is a very interesting finding from the perspective of the push to reduce 'bureaucracy' associated with research.
Research Councils UK: Efficiency and Effectiveness of Peer Review. 2006. Reference Source
This is some of the most thorough evidence on the costs of peer review for funding allocation, although it is now quite out of date.
There is a striking disconnect between the institutional and community support for the peer review system and the empirical evidence of its effectiveness – unfortunately the scope of our review excluded the types of research that might explain this divergence.
Is this an example of a general phenomenon - that the research community is not very sensitive to the conclusions of evidence about the research process? Although this question could be countered by the argument that the evidence on the problematic nature of peer review is not especially compelling.
This suggests that peer review processes may not work well for transdisciplinary teams integrating both academic and non-academic experts.
Although the context is different, this contrasts with evidence from REF panels where the combination of researcher and research user perspectives is perceived as being positive (evaluation reports and also possible mention in [[The Evaluator's Eye]] ^^CHECK^^).
If peer review is reliable, the judgements of different peer reviewers on the same proposal should be highly correlated
Not necessarily, as this assumes that all reviewers are using the same implicit as well as explicit criteria. It may actually be expected that reviewers' views will differ. See also: A Kuhnian Critique of Psychometric Research on Peer Review | Philosophy of Science: Vol 79, No 5
If inconsistency stems from discrepancies in review quality (which is by no means clear), it might be feasible to evaluate the quality of reviews, although this approach has its own challenges – for example, what is a ‘good’ review? If a review is not consistent with other review does that intrinsically make it ‘bad’? It could be the outlier picking up on the true potential of an innovative application.
Doesn't this argument undermine the previous suggestion that a lack of inter-review reliability is problematic?
Two funders have experimented with, and evaluated, virtual peer review both by teleconference and through the use of Second Life, a virtual world. NIH estimated that using Second Life telepresence, peer review could cut panel costs by one third (Bohannon, 2011). Pier et al. (2015) compared videoconference and face-to-face panels. They set up one videoconference and three face-to-face panels modelled on NIH review procedures, concluding that scoring was similar between face-to-face and videoconference panels. Both the Bohannon and Pier studies of virtual panels noted that participants valued the social aspects of meeting in person and preferred the face-to-face arrangements.Gallo et al. (2013) examined four years of peer review discussions, two years face-to-face and two years teleconferencing. They found minimal differences in merit score distribution, inter-rater reliability or reviewer demographics. They also noted that panel discussion, of any type, affects the funding decision for around 10 per cent of applications relative to original scores.
Summary of the evidence on remote/virtual peer review.
Bibliometric analyses are by no means perfect measures of performance – only capturing a proxy of academic performance (Belter, 2015). Nonetheless, the findings suggest that peer review assessment is, at best, a crude predictor of performance.
This conclusion bears further analysis. It really does depend on how imperfect a proxy currently used bibliometric approaches are. This is an example of where [[Research on Research]] is severely limited by the ability to robustly measure meaningful endpoints.
Table 3. Summary of evidence from the literature regarding the effectiveness and burden of peer review.
This table is a useful summary of the key issues and quality of the evidence.
(Ismail et al. (2009) estimated that >95% of UK medical research funding was allocated by peer review).
Check this source. The number feels very high even for medical research, but may depend on how block grant funding is being treated (or even if it is included).
Good evidence shows that the burden of peer review is high and that around 75% of it falls on applicants. By contrast, many of the efforts to reduce burden are focused on funders and reviewers/panel members.
Important observation, that relates to applying #[[Systems thinking]] to the research system. This is an example of where policy interventions potentially have limited net effect. #Bureaucracy
The authors declare they have no competing interests.
Surprising to not have a mention that three of the authors work for an organisation that provides peer review of grant applications as a service to funding organisations.
Previous research has reported the importance of score-calibration comments and even laughter in the effectiveness of panel discussion, although it is unclear if these are affected by chair facilitation
Potential evidence hinting at the importance of social dynamics on panel functioning, which has implications for virtual settings, if the panel has not had a face-to-face opportunity in the past.
respondents who felt the outcome was affected by panel discussions viewed the discussion effectiveness and quality more favorably than those who felt the outcome was not affected by the discussions.
A link between the perceived importance of discussion and its perceived quality. It isn't clear in which direction the causality might go, and there is no data in this paper to establish actual effects on outcome, beyond reviewer perceptions.
So, one grant with an excellent score by one reviewer and a mediocre score by the other ends up with a score outside the fundable range. And if the average of the scores places it out of the range to be discussed, the panel usually just lets it go so as to not increase the length of the meeting. So, a grant that one reviewer rated favorably can get torpedoed by a bad reviewer, even if that reviewer was totally off-base on their reasons for the bad score.
This phenomenon provides an underpinning rationale for the focal randomisation approach.
Biases may be exacerbated by short discussion times
This is important in the context of virtual meetings, which tend to have shorter discussion times. The implication is that less discussion is problematic, although elsewhere in the paper there is evidence that discussion is not as effective as it might be.
However, some respondents felt the distribution of comments from panel members and their relative weight on panel opinion was not even across reviewers and sometimes dependent on whether the assessment was positive or negative:“My experience has been that folks do not use the discussion time most effectively. The primary reviewer is accepted at face value and when a dissenting opinion is voiced, the panel seems to be reluctant to discuss and rather defers to the primary reviewer. This is a flaw that makes the review only as good as the thoroughness of ONE reviewer. And in my limited experience I have seen some shoddy reviewing. This is unfair to the team that prepares the application.”“On the whole, I felt most panel members were too polite and unwilling to offer frank opinions of weak proposals.”
This perception seems to undermine much of the perceived value of group discussion.
the magnitude of scoring shifts after discussion are typically relatively small and are even smaller (with shorter discussion times) in teleconference panels compared to face-to-face panels
These are the principal differences in panels not conducted face-to-face.
to mitigate this subjectivity
Interesting assertion, based on the assumption that a larger group is less subjective. Isn't there evidence elsewhere suggesting larger groups are more susceptible to bias (group-think). And if the assertion is correct then it support the notion that distributed peer funding mechanisms might be (more?) effective than conventional peer review (e.g. Bollen et al. 2014).
their perceptions
This study is focussed on perceptions, and so needs to be read alongside studies that investigate outcomes.
This suggests that “excellence” resides between different communities and is ill-structured/defined in each context. Local groups and disciplines may have their own more specific (though sometimes conventional rather than explicit) measures of “excellence”: Biologists may treat some aspects of performance as “excellent” (for example, number of publications, author position, citations counts), while failing to recognize aspects considered equally or more “excellent” by English professors (large word counts, single authorship, publication or review in popular literary magazines and journals) (O’Donnell, 2015).
Could portray disciplinary diversity in interpretation of notions of excellence as a strength, rather than a problem as here. And I am not sure that there is evidence that this is always ill-structured or defined. In many formal contexts excellence is operationalised through criteria that, while they might be themselves contested, do provide some structure and consistency within the specific context. The problem really comes when these criteria are implicit or unacknowledged.
Policymakers, who wanted evidence of impact to demonstrate return on investment
Not sure I agree with this characterisation of my own position. For the me the ‘impact agenda’ is very much about increasing the benefits from research as well as providing evidence of return on investment.
This engagement is increasingly seen as important; for example, it represents 20% of the evaluation in the next version of the Research Excellence Framework (REF) in the UK.
This statement is not accurate. Impact represented 20% of the evaluation in the last REF; in the next REF it will be 25% (see http://www.ref.ac.uk/media/ref,2021/downloads/REF2017_01.pdf)
Only metrics should be used in research evaluation when they reflect the quality of papers
But this depends on the defintition of 'quality'. The assumption in this statement is that there is only one dimension of quality (academic evaluation as per F1000), and so any metrics must align with that. But it is possible for metrics to relfect other aspects of 'quality' (for example, usefulness beyond academia).
We looked for factors that could explain this. In addition to international collaboration, scientific mobility was expected to contribute to impact4. So we also considered new researchers coming in, returnees and emigrating researchers, all of which are tracked by the Organisation for Economic Co-operation and Development (OECD). These variables, together with collaboration, proved to be highly correlated as measures of international engagement; so we used them to create an index of openness and were able to assign values to 33 of the countries that we looked at (data available at go.nature.com/2fzrnt3).
Composite measure if openness including both mobility and co-authorship. Data availble
When aggregating article and citation counts, an integer counting method was employed where, for example, a paper with two authors from a UK address and one from a French address would be counted as one article for each country (i.e. 1 UK and 1 France)
Key methodological point - so country comparisons effectively multiple-count articles. And other evidence suggests these collaborative articles will have the highest citation counts
Thus funding is apportioned by the U.K.’s Higher Education Funding Councils. In the academic year 2013–2014, the budget added up to ~£1.68 billion
This is an England-only figure and does not include the equivalent funding streams in Scotland, Wales and Northern Ireland
The results assist HEIs investment and financial strategy by informing their potential income generation through performance-based research funding. For example, an HEI can recruit more academics in the Engineering department so as to increase their chances of acquiring funding. Hence, the investment decision can influence the future of science.
This is a curious argument that implies acquisition of funding is an end in itself. Increasing the size of the engineering department at the expense of, say, the history department will bring more funds, but also bring more costs (engineering research is more costly than history, in general). So the question for a university should be how much engineering do we want?
Interdisciplinary research sits in two different departments. For example, information sciences either come under UoA 11 or UoA 36. As UoA 11 offers higher income for good research outputs, HEIs which have submitted their information sciences research in UoA 36 may consider submitting in UoA 11 for the next exercise.
This ignores the many factors that will influence UOA selection. Most importantly research should be submitted to the panel that is best equipped to assess it. Universities will also wish to consider how their departmental structure map to UOAs as well, as the score in the environment section of the assessment might be influenced by the extent to which the staff submitted form a coherent research grouping. It is also the case that HEIs are already aware of the relative cost weightings in the funding formula, and I am not sure this analysis give much information beyond that.
It informs an HEI the amount of funding an academic can bring into the department.
Might be misleading, as it assumes past performance is a predictor of the future, and the funding outcomes are always dependent on realtive performance of different HEIs and UOAs
the perspectives on quality of research is either similar or superior to national research assessment exercises
In my view it isn't possible to say that peer review or metrics based approaches are 'superior' to one another. They either give similar or different results, but there is no other (by implication, more reliable) measurement to compare then to, in order to assert superiority. .
The findings indicate that the outcome of judgements made on the quality of research either by peer-reviewed government ranking (REF results) and metrics-based ranking (JIF) largely remain the same in disciplines where journals are considered the main channels of research communication
In my view this conclusion is too strong for the analysis presented. The HEFCE analysis based on individual output scores shows that while there is a relationship between REF scores and journal-level citation metrics, there are also many outliers,and the predictive power of journal-level citation metrics is weak. This is to be expected given the wide distribution of article-level citation scores. The journal-level score is a poor predictor of the article-level score (see http://www.biorxiv.org/content/early/2016/07/05/062109).
Further it is important to relfect that this study only describes correlation and an equally plausible explanation of the data is that high quality research both scores well in REF assessement and is more likely to be submitted to journals with high citation rates.
All the outputs submitted by multiple UK HEIs in the chosen UoAs were filtered for journal article submissions and the impact rating of every article’s journal was mapped using Thomson Reuters’ Journal Citation Reports
This lists only include English HEIs but the text implies UK. It is not clear why HEIs from outside of England were not included.
A potential method to recognise quality of an output is to observe the quality or rank of its journal based on the journal impact factor.
But important to note that panel members were specifically told not to use venue of publication as a factor in assessment.
this is the first time an investigation has estimated the monetary value of a good research outpu
An analysis of the funding awarded relating to both outputs and impact case studies from REF2014 was published in a blog post on 1 Feb 2017 by Mark Reed and Simon Kerridge:
monetary value
The use of the word 'value' is curious. These figures don't represent the value of the outputs as such, but the funding that is earned as a result of them.
The financial awards for outputs in the remaining subject areas
Much of this variation is built into deliberate differences in the funding formula that recognise the different costs of research in different disciplines.
HEFCE’s funding allocations
Not clear exactly which data used
Subsequently, all the journal articles submitted by the HEIs, whose journals were in the Q1 category were considered, allowing the calculation of all the HEIs percentage of Q1 publications, which was compared against percentage of 4* publications.
The analysis does not compare like-with-like. The upper quartile figures refers only to journal articles whereas the overall REF score applies to all the outputs submitted for assessement.
However, since it was not possible to get the necessary data directly from the REF2014 results, i.e. it was not possible to find out which output got a 4* rating, we decided to use an alternative approach. By using the Thompson Reuter’s Journal Citation Report against the submitted journal papers for each HEI in the chosen UoA, we identified how many of the submitted articles were in top quartile journals, and accordingly we prepared a rank list of the HEIs in a given UoA based on the number of Q1 publications. This list was plotted against their percentage of 4* outputs to find any relationship.
The more detailed analysis based on individual output scores was carried out as part of the Metric Tide review. See this report (PDF file) for details and results:
as indicators of quality
This misrepresents the process; some panels used citation information to inform peer review judgements.
1052
Incorrect figure: it's actually 1124
An HEI lost about 17.1% (£14.2 million) of its funding and in another exceptional case, an HEI lost 45% of its funding. The maximum gain by any HEI stood at 12.4% (£7.1 million) [13, 14]. The repercussions of such fluctuations are considerable to the future of research at UK HEIs
Important to not that variations between exercises also depend on change in numbers of staff, numbers of PhD students, and research income from charity and business sources.
It may be argued that the amount of money available to distribute to the HEIs is very much dependent on the available budget for a particular government, and hence the monetary value of the research outputs will not provid
This is clearly the case, so the analysis here can only be viewed as exploring what a research output is 'worth' in the specific context of the U.K. Funding system
According to the Higher Education Statistics Agency (HESA), a typical UK HEI’s revenue break-down is as follows: 35% tuition fee, 30% funding council grants, 16% research grants and contracts, 1% from endowments/ investment income, and 18% from other sources i.e. alumni donations etc
Averages like this hide considerable variation across the UK HE sector, so aren't especially meaningful or useful.
Resetting the trajectory of the scientific enterprise requires multiple, difficult actions. Politicians must understand that job creation is not — and furthermore, should not be — a primary goal of the NIH or any other science-funding agency. Funds should be distributed on the basis of merit alone, not geopolitical considerations and interests. Institutions need to realign their mentality with their original academic mission, and reduce soft-money positions. Publishers should care less about publishing flashy stories and more about disseminating solid science. Individual scientists should emphasize excellence and rigour over stockpiling more and more papers and grants.
These aren't really actions, more just exhortations for everyone to behave better.
But more money is not enough to create better science. Indeed, the scientific enterprise has been betrayed by the mismanagement of its financial support.
Interesting analysis of perceived problems with biomedical funding, but limited consideration of solutions.
there is some evidence to suggest that double blind review can increase the acceptance rate of women-authored articles in the published literature (Darling, 2015).
There is more debate in the effect of double blind peer reviews on gender bias than is suggested here. See the discussion in Darling 2015 itself and also in the Lee et al. review article cited by this article
announcement of author’s career stage has potential consequences on their reputation
Didn't understand what was meant here.
the making of the sausage, so to speak
Not sure this metaphor will be globally understood, and may, in some cases, be culturally inappropriate.
This is likely dependent on whether research communities themselves choose to embrace any such crediting or accounting systems for peer review.
Important
suggesting that the process could become commercialized as this domain rapidly evolves
Having raised this, it would be helpful to consider the potential implications of the use of a commercial model. These could be either positive or negative.
Hence, this survey could represent a biased view of the actual situation
See comment above. This statement needs some justification beyond the author of the survey.
market-rate reimbursement
There is a non-sequitur in the argument here: there is a move from discussing very general 'recognition' to very specific ideas about 'market-rate reimbursement'. The switch isn't justified and seems a bit of a straw man, given the previous statement of recognition being about assessement etc.
it is important to note that commercial publishers, including Wiley, have a vested interes
Is the implication, therefore, that the Wiley survey and its analysis is somehow flawed? If not, then this should be removed. If it is then some justification of problems with the research itself (rather than just who conducted it) should be provided.
commercial
Not sure the word 'commercial' is needed or relevant to the discussion here.
A vast majority of researchers see peer review as an integral and fundamental part of their work. They often even consider peer review to be part of an altruistic cultural duty or a quid pro quo service, closely associated with the identity of being part of their research community.
Is there evidence for this assertion? There is also considerable difference between a 'good of the system' motivation and quid pro quo
The inertia hindering widespread adoption of new models of peer review can be ascribed to what is often termed “cultural inertia” within scholarly research.
Important point, relevant to broader issues than just journal peer review.
under an open license
Quite vague requirement. Would benefit from more precision (I.e what type of licence qualifies in the view of the authors) and a discussion of the rationale behind the requirement. For example, what are the benefits of permitting the reuse of peer reviews?
Instead, reviewers are asked to focus on whether the research was conducted properly and that the conclusions are based on the presented results.
This provides the definition of 'soundness' that was missing earlier. Some duplication of earlier point here.
this uncertainty in standards and implementation can potentially lead to, or at least be viewed as the cause of, widespread failures in research quality and integrity
I think there is a slightly circularity in the argument here. The previous paragraph implies, perhaps correctly, that traditional peer review of journal articles is not (has never been) able to detect fraud or misconduct. So if there is an increase in fraud/misconduct it is not necessarily related to constraints on the peer review system.
However, beyond editorials, there now exists a substantial corpus of studies that critically examines the technical aspects of peer review. Taken together, this should be extremely worrisome, especially given that traditional peer review is still viewed almost dogmatically as a gold standard for the publication of research results, and as the process which mediates knowledge dissemination to the p
Important to stress the limited scope of this statement which is limited to journal article review through traditional models
This is especially the case as the (scientific) scholarly community appears overall to have a strong investment in a “creation myth” that links the beginning of scholarly publishing—the founding of The Philosophical Transactions of the Royal Society—to the invention of peer review. The two are often regarded to be coupled by necessity, largely ignoring the complex and interwoven history of peer review and publishing. This has consequences, as the individual identity as a scholar is strongly tied to specific forms of publication that are evaluated in particular ways (Moore et al., 2017). A scholar’s first research article, PhD thesis, or first book are significant life events. Membership of a community, therefore, is validated by the peers who review this newly contributed work. Community investment in the idea that these processes have “always been followed” appears very strong, but ultimately remains a fallacy.
This is a very significant and important point
Here, e-prints or pre-prints are not formally peer reviewed prior to publication, but still undergo a certain degree of moderation in order to filter out non-scientific content. This practice represents a significant shift, as public disse
Indicates that some expert input is required even in this case.
Such a consequence is plausibly related to, or even a consequence of, broader shifts towards a more competitive neoliberal academia and society at large
This statement is not strongly evidenced
soundness
Defined as?
for instance, space in peer reviewed print publication venues, research time at specialized research facilities, or competitive research funds.
Slipping beyond the narrower scope defined earlier
expected impact
Presumably this refers to academic/scholarly impact rather than societal impact. Given the interest in the latter should perhaps be clarified.
For the purposes of this article, we are exclusively addressing peer review in the context of manuscripts for research articles, unless specifically indicate
In this case shouldn't this be reflected in the title?
One problem is that the time it takes to produce a monograph may extend beyond the REF period, so in order to be REF-able, historians also need to be publishing articles that can be written relatively quickly allowing more to be produced in the same timeframe.
Why? The REF rules for 2014 only required that outputs be produced within the assessment period, not that the work was completed within the period. There is a point about the amount of material required in disciplines where long form outputs are the norm, but this is in part addressed by double-weighting.
“now I target management journals, which is one way of hitting a four star”
This quote reflects a profound understanding of the REF assessment process, where "hitting a four star" is about the content not venue of publication.
For example, some key target journals in marketing moved down the ABS rankings in 2015, yet the pressure on academics to “hit a four-star” remained.
Again this seems to elide (incorrectly) publication in a specific journal with a particular REF score.
For example, during probationary periods, new lecturers were required to publish certain numbers of papers of a specified quality. For academics working in marketing, quality was determined using the Chartered Association of Business Schools’ (ABS) annually published Academic Journal Guide, which ranks business and management journals on a star-rating system similar to that employed by the REF.
Important to note that, while the rating system between the ABS list and the REF might appear superficially similar the processes behind then are completely different. The REF involves the expert review of each output, and is not based on the journal status.
Scientists whose output has been endorsedto a high degree will have more AEP to usefor endorsements, and thus have a largerin uence in the community
How would new researchers get points to allocate? Does this provide an unhelpful positive feedback loop. Could make a initial allocation to anyone and use rewards for others activities
Scienti c projects often take yearsfrom start to publication. Consequently,there is an inherent time lag in using cita-tions to appraise scienti c work
Not sure how the proposal solves this; still dependent on the rate at which researchers deploy their assessments. And they have to know of the existence of a research output which is a limit on the rate (and may vary between outputs)
Towards Open Science: The Case for a DecentralizedAutonomous Academic Endorsement System
Interesting proposal for deployment of blockchain tech to provide a decentralised research evaluation system
Lots of criticism of publishers points out that “academics work for them for free”. Some do, if they do not have positions. Others, though, actually work for institutions (universities) and are paid by them. Their contracts often stipulate that they should participate in a research ecosystem that includes reviewing. So, actually, universities are often the ones donating time – in the form of their paid labour-force – to reviewing. There is a real flaw, then, in suggesting that reviewers should be paid because often they already are. If this really is to work, I’d suggest that: a. researchers without affiliation could be paid personally; but b. in the case of researchers with a permanent university position, the funds should go back to the university’s own open-access fund. The problem with point b is that it then doesn’t give academics the price sensitivity that is desired for their own labour.
This is a really important point.
it would not have given him an opportunity to consider the needs of other aspects of policy-making that are less about the future development of novel products and processes, but more focused on analysis of current knowledge to address political problems or on horizon scanning of future social, economic, environmental and technical developments, which might have political significance
I will contend that, in making this a commercial transaction, government scientists took on a lot of contract management work at the expense of analysis and advice, while the research community became more focused on contractual delivery than dialogue with users.
Interesting perspective from someone at the heart of how the reforms were implemented.
A dearth of clear, relevant and reliable research evidence, and difficulty finding and accessing it, were the main barriers to the use of research.
Key point around access to research content
in contrast with the published evidence on research impact (which depicts much as occurring indirectly through non-linear mechanisms), this sub-panel seems to have captured mainly direct and relatively short-term impacts one step removed from patient outcomes.
Researchers and citizens prioritized high-quality scientific outcomes (papers, trainees) over other attributes. Both groups disvalued research targeted at economic priorities relative to health priorities. Researchers granted a premium to proposals targeting scientific priorities.
Summary of conclusion; study specific to biomedical research.