David Blumenthal and colleagues [17] found that university geneticists and otherlife scientists who perceive higher levels of competition in their fields are morelikely to withhold data or results. Such withholding took the form of omittinginformation from a manuscript or delaying publication to protect one’s scientificlead, maintaining trade secrets, or delaying publication to protect commercial valueor meet a sponsor’s requirements. John P. Walsh and Wei Hong [18] have reportedsimilar finding

ompetition in science has its bright side, which past analysts and commentatorstended to emphasize and current writers often affirm. It has been credited withensuring that ideas, work, proposals and qualifications of all interested parties areevaluated prior to the distribution of rewards, particularly funding and positions.From this perspective, competition promotes open examination and fair judgment.The norm of universalism [11] is supported when all qualified people have theopportunity to propose and defend their ideas and work in open competition [12].After all, absent competition, cronyism is likely to flouris

Increases in levels of competition in science are symptomatic of a moregeneral hypercompetitive shift in organizations

Thomas, L. G. III (1996). The two faces of competition: Dynamic resourcefulness and the hyper-competitive shift.Organization Science, 7, 221–242

Because science is a cumulative, interconnected, andcompetitive enterprise, with tensions among the various societies in which researchis conducted, now more than ever researchers must balance collaboration andcollegiality with competition and secrecy

The scientific enterprise is characterized by competition for priority, influence,prestige, faculty positions, funding, publications, and students

Bok, D. (2003).Universities in the marketplace: The commercialization of higher education.Princeton: Princeton University Press

Pfeffer, J. (1992).Managing with power: Politics and influence in organizations. Boston: HarvardBusiness School Press

Theirdiscussions suggest clearly that the downside of competition has been underesti-mated and that it may have more prominent effects now than in past years. Asreputation, respect and prestige are increasingly connected to resources and tosuccess in the competitions that distribute those resources, scientists find more oftheir work and careers caught up in competitive arenas. The six categories ofcompetition’s effects that emerged in our analyses suggest reason for concern aboutthe systemic incentives of the U.S. scientific enterprise and their implications forscientific integrity.

scienceWhen the actor Michael J. Fox was in the initial stages of creating his foundation forresearch on Parkinson’s Disease, he came to recognize the negative impact thatcompetition among scientific groups has on the overall progress of research on thedisease. The director of one group actually said to him, ‘‘Well, if you don’t help us,then, at least, don’t help them’’ [1, p. 236]. Such was his introduction to thecompetitive world of U.S. science.

The winner-take-all aspect of the priority rule has its drawbacks, however. It can encourage secrecy, sloppy practices, dishonesty and an excessive emphasis on surrogate measures of scientific quality, such as publication in high-impact journals. The editors of the journal Nature have recently exhorted scientists to take greater care in their work, citing poor reproducibility of published findings, errors in figures, improper controls, incomplete descriptions of methods and unsuitable statistical analyses as evidence of increasing sloppiness. (Scientific American is part of Nature Publishing Group.)As competition over reduced funding has increased markedly, these disadvantages of the priority rule may have begun to outweigh its benefits. Success rates for scientists applying for National Institutes of Health funding have recently reached an all-time low. As a result, we have seen a steep rise in unhealthy competition among scientists, accompanied by a dramatic proliferation in the number of scientific publications retracted because of fraud or error. Recent scandals in science are reminiscent of the doping problems in sports, in which disproportionately rich rewards going to winners has fostered cheating.

The role of external influences on the scientific enterprise must not be ignored. With funding success rates at historically low levels, scientists are under enormous pressure to produce high-impact publications and obtain research grants. The importance of these influences is reflected in the burgeoning literature on research misconduct, including surveys that suggest that approximately 2% of scientists admit to having fabricated, falsified, or inappropriately modified results at least once (24). A substantial proportion of instances of faculty misconduct involve misrepresentation of data in publications (61%) and grant applications (72%); only 3% of faculty misconduct involved neither publications nor grant applications.

The predominant economic system in science is “winner-take-all” (17, 18). Such a reward system has the benefit of promoting competition and the open communication of new discoveries but has many perverse effects on the scientific enterprise (19). The scientific misconduct among both male and female scientists observed in this study may well reflect a darker side of competition in science. That said, the preponderance of males committing research misconduct raises a number of interesting questions. The overrepresentation of males among scientists committing misconduct is evident, even against the backdrop of male overrepresentation among scientists, a disparity more pronounced at the highest academic ranks, a parallel with the so-called “leaky pipeline.” There are multiple factors contributing to the latter, and considerable attention has been paid to factors such as the unique challenges facing young female scientists balancing personal and career interests (20), as well as bias in hiring decisions by senior scientists, who are mostly male (21). It is quite possible that, in at least some cases, misconduct at high levels may contribute to attrition of woman from the senior ranks of academic researchers.

Editors, Publishers, Impact Factors, and Reprint Income

Of thee and of the white lylye flour

 O Lord, oure Lord, thy name how merveillous

It is important to have elite scientific research, but we should not pretend that the interests of scientists perfectly overlap with the public interest. By presuming that innovation is all about pace rather than direction, publicly-funded science risks following rather than counterbalancing private sector interests. The structure of what Michael Polanyi called the ‘republic of science’ makes it easy for scientists to offload responsibility. Polanyi’s science is self-organising and devoted to the pure pursuit of knowledge. As philosopher Heather Douglas describes it, the general responsibilities of scientists to society are trumped by their role responsibilities towards their disciplines. While Polanyi would not argue for irresponsibility, he would have been relaxed about scientists being divorced from social responsibilities. Polanyi had this response to those who would direct science: ‘You can kill or mutilate the advance of science, you cannot shape it’.

We need the ERC and other blue-skies funding as part of the innovation ecosystem. The danger is that, by identifying this as ‘excellence’, it damns everything else, including applied science, user-driven innovation, open science, meta-analysis, regulatory science, social innovation and engagement with policymakers and the public, to mediocrity. In the last thirty years, our sense of ‘excellence’ has narrowed, not broadened

journal rankings discourage interdisciplinarity by systematically evaluating disciplinary research more highly

‘Excellence’ is an old-fashioned word appealing to an old-fashioned ideal. ‘Excellence’ tells us nothing about how important the science is and everything about who decides. It is code for decision-making based on the autonomy of scientists. Excellence is judged by peers and backed up by numbers such as h-indexes and journal impact factors, all of which reinforces disciplinary boundaries and focuses scientists’ attention inwards rather than on the problems of the outside world.

Space missions are about technological problems with technological solutions. It is normally clear whether or not they have succeeded. There is far more disagreement about causes and cures for ‘wicked’ problems of poverty or climate change. Science alone cannot give us the answer.

the problems of space and the problems of poverty are qualitatively different, demanding very different approaches

Another important, related but distinct function is that general journals act as a filter: ideally, we publish the ‘best’ papers, reporting the stories most likely to be a wide interest, and those with the greatest claim to coverage in the general media (newspapers, television and so on). A hierarchy of journals, with the general ones at the ‘top’, helps journalists to find reports of the most significant developments – those that are of most interest, and also (importantly) ‘sound’, in the sense of having passed rigorous peer review. In this sense, general journals offer a link between the specialist scientific literature and the general media and public.

The selection of successful proposals is not the same as that for other HRC contracts. All proposals that meet the eligibility criteria will be assessed for compatibility with the scheme’s intent; proposals won’t be scored or ranked. All proposals that are considered eligible and compatible will be considered equally eligible to receive funding, and a random process will be used to select approximately three proposals to be offered funding. A full description of the assessment process to determine eligibility, compatibility and which applications will receive funding can be found in Appendix 1 of the Guidelines document.

These observations have important implications for the grant peer review system. If reviewers are unable to reliably predict which meritorious applications are most likely to be productive, then reviewers might save time and resources by simply identifying the top 20% and awarding funding within this group on a random basis or according to programmatic priorities. In this regard, we refer to our recent suggestion that the NIH consider a modified lottery system (Fang and Casadevall, 2014) and note that the New Zealand Health Research Council has already moved to a lottery system to select proposals for funding in its Explorer Grants program (Health Research Council of New Zealand, 2015).

17% (334 of 1987) of grants with a percentile score of zero failed to produce any citations.

In contrast, a recent analysis of over 130,000 grant applications funded by the NIH between 1980 and 2008 concluded that better percentile scores consistently correlate with greater productivity (Li and Agha, 2015). Although the limitations of using retrospective publication/citation productivity to validate peer review are acknowledged (Lindner et al., 2015; Lauer and Nakamura, 2015), this large study has been interpreted as vindicating grant peer review (Mervis, 2015; Williams, 2015). However, the relevance of those findings for the current situation is questionable since the analysis included many funded grants with poor percentile scores (>40th percentile) that would not be considered competitive today. Moreover, this study did not examine the important question of whether percentile scores can accurately stratify meritorious applications to identify those most likely to be productive.We therefore performed a re-analysis of the same dataset to specifically address this question. Our analysis focused on subset of grants in the earlier study (Li and Agha, 2015) that were awarded a percentile score of 20 or better: this subset contained 102,740 grants. This percentile range is most relevant because NIH paylines (that is, the lowest percentile score that is funded) seldom exceed the 20th percentile and have hovered around the 10th percentile for some institutes in recent years.

Most funding agencies employ panels in which experts review proposals and assign scores to them based on a number of factors (such as expected impact and scientific quality). However, several studies have suggested significant problems with the current system of grant peer review. One problem is that the number of reviewers is typically inadequate to provide statistical precision (Kaplan et al., 2008). Researchers have also found considerable variation among scores and disagreement regarding review criteria (Mayo et al., 2006; Graves et al., 2011; Abdoul et al., 2012), and a Bayesian hierarchical statistical model of 18,959 applications to the NIH found evidence of reviewer bias that influenced as much as a quarter of funding decisions (Johnson, 2008).Although there is general agreement that peer review can discriminate sound grant applications from those containing serious flaws, it is uncertain whether peer review can accurately predict those meritorious applications that are most likely to be productive. An analysis of over 400 competing renewal grant applications at one NIH institute (the National Institute of General Medical Sciences) found no correlation between percentile score and publication productivity of funded grants (Berg, 2013). A subsequent study of 1492 grants at another NIH institute (the National Heart, Lung and Blood Institute) similarly found no correlation between the percentile score and publication or citation productivity, even after correction for numerous variables (Danthi et al., 2014). These observations suggest that once grant applications have been determined to be meritorious, expert reviewers cannot accurately predict their productivity.

The mechanism used by the National Institutes of Health (NIH) to allocate government research funds to scientists whose grants receive its top scores works essentially no better than distributing those dollars at random, new research suggests

In the late 1980s, the United Kingdom became the first country to systematically evaluate the quality of its university research. The REF is the latest incarnation of these check-ups. Previously known as the Research Assessment Exercise (RAE), the evaluations are widely credited with helping to improve the country's research system. Between 2006 and 2010, citations of UK articles grew by 7.2%, faster than the world average of 6.3%; and the country's share of citations grew by 0.9% per year, according to a 2011 analysis conducted by publishing company Elsevier for the government.

At Cardiff University, around ten academics were pressured to switch to teaching-focused contracts after they scored poorly on a practice exercise, so as not to drag down their department, says Peter Guest, an archaeologist at Cardiff and the university's UCU liaison on the REF. This form of game-playing is discouraged, but not expressly forbidden, by the REF — however, making career decisions solely on the basis of the evaluation is against the university's own policies, as well as those of many other institutions, says Guest.

Whereas the literature tends to equate originality with substantiveinnovation and to consider the personal attributes of the researcher as irrelevant to theevaluation process, we show that panelists often view the originality of a proposal as anindication of the researcher’s moral character, especially of his/her authenticity andintegrity

And as Jonathan Culler pointed out in Framing the Sign: Criticism and Its Institutions, the external review process enabled the rise of many forms of innovative and even controversial work in the humanities

As Christopher Jencks and David Reis-­man argued in The Academic Revolution, the development of external peer review freed individual scholars from the vertical—and sometimes pa-­rochial and territorial—evaluation of their work by local college deans andupper-­level administrators

The document is the fundamental unit of analysis in most large-scale quantitative studies of scientific behavior

The evidence and theory outlined here suggest that most published findings in well-developed fields should be expected and unsurprising.11 Such findings fit with tradition: scientists with the appropriate habitus are disposed to generate and acknowledge them as valid science. By contrast, unexpected findings should rarely reach publication. Tradition will be reliably but modestly rewarded with citations, whereas subversive innovation, if published, should display a higher average and variance in acclaim.12 Finally, unexpected findings should be over-represented in the work of high-achieving scientists. Scientific capital is disproportionately awarded for work that alters the scope of accepted knowledge.

Kuhn ([1959] 1977) introduced the notion of the essential tension at a conference motivated by Cold War concerns about declining originality, innovation, and scientific competitiveness in the United States (a persistent concern; see Cowen 2011). The conveners, all psychologists, had framed the conference around a dichotomy between convergent and divergent styles of thinking. Convergent thinking was conservative, oriented toward consensus and shared patterns of thought (Kuhn [1959] 1977). Divergent thinking, by contrast, was radical, characterized by “flexibility and open-mindedness” (Kuhn [1959] 1977:226). According to most of the conference speakers, divergent thought was essential for scientific progress, yet it was being stifled by the U.S. educational system.

When following a conservative strategy and adhering to a research tradition in their domain, scientists achieve publication with high probability: they remain visibly productive, but forgo opportunities for originality. When following a risk-taking strategy, scientists fail more frequently: they may appear unproductive for long periods, like the seven years Andrew Wiles spent proving Fermat’s Last Theorem or the decade Frederick Sanger invested in developing the “Sanger method” of DNA sequencing.4 If a risky project succeeds, however, it may have a profound impact, generating substantial new knowledge and winning broad acclaim (Kuhn 1962). This strategic tension is repeatedly articulated as a dichotomy: in the sociology of science, as reliable “succession” versus risky “subversion” (Bourdieu 1975) or “relevance” versus “originality” (Whitley 2000); in the philosophy of science, as “conformity” versus “dissent” or “discipline” versus “rebellion” (Polanyi 1969); and in the study of innovation, as “exploitation” versus “exploration” (March 1991).5 Recent theoretical work supports this broad picture by highlighting the distinctive contributions (Weisberg and Muldoon 2009) and rewards (Kleinberg and Oren 2011) associated with traditional versus innovative strategies.

To remain in the research game requires productivity. Scientists typically achieve this by incremental contributions to established research directions. This may yield enough recognition to maintain a (relatively low) position. Achieving high status, by contrast, requires original and transformative contributions, often obtained by pursuing risky new directions (Merton 1957).

In this article, we examine scientific choice quantitatively and at scale, using published claims in contemporary biomedicine to make inferences about underlying choices and dispositions

In an expanding universe of possible research questions, a topic that attracts intense investigation is separated from neglected or abandoned topics by more than just the contours of nature. Scientists’ choices matter: in aggregate, patterned choices give scientific knowledge its shape and guide its future evolution

By studying prizewinners in biomedicine and chemistry, we show that occasional gambles for extraordinary impact are a compelling explanation for observed levels of risky innovation. Our analysis of the essential tension identifies institutional forces that sustain tradition and suggests policy interventions to foster innovation.

An innovative publication is more likely to achieve high impact than a conservative one,

Moreover, their research is cited by a more diverse set of journals, both relative to controls and to the pre-appointment period.

These keywords are also more likely to change after their HHMI appointment.

We show that the work of HHMI investigators is characterized by more novel keywords than controls.

ur results provide support for the hypothesis that appropriately designed incentives stimulate exploration. In particular, we find that the effect of selection into the HHMI program increases as we examine higher quantiles of the distribution of citations. Relative to early career prize winners (ECPWs), our preferred econometric estimates imply that the program increases overall publication output by 39%; the magnitude jumps to 96% when focusing on the number of publications in the top percentile of the citation distribution. Success is also more frequent among HHMI investigators when assessed with respect to scientists' own citation impact prior to appointment, rather than relative to a universal citation benchmark. Symmetrically, we also uncover robust evidence that HHMI-supported scientists "flop" more often than ECPWs: they publish 35% more articles that fail to clear the (vintage-adjusted) citation bar of their least well cited pre-appointment work. This provides suggestive evidence that HHMI investigators are not simply rising stars annointed by the program. Rather, they appear to place more risky scientific bets after their appointment, as theory would suggest.

In 1980, a scientist from the University of Utah, Mario Capecchi, applied for a grant at the National Institutes of Health (NIH). The application contained three projects. The NIH peer reviewers liked the first two projects, which were building on Capecchi's past research efforts, but they were unanimously negative in their appraisal of the third project, in which he proposed to develop gene targeting in mammalian cells. They deemed the probability that the newly introduced DNA would ever find its matching sequence within the host genome vanishingly small and the experiments not worthy of pursuit. The NIH funded the grant despite this misgiving, but strongly

An innovative publication is more likely to achieve high impact than a conservative one

We attempt to replicate 67 papers published in 13 well-regarded economics journalsusing author-provided replication files that include both data and code. Some journalsin our sample require data and code replication files, and other journals do not requiresuch files. Aside from 6 papers that use confidential data, we obtain data and codereplication files for 29 of 35 papers (83%) that are required to provide such files as acondition of publication, compared to 11 of 26 papers (42%) that are not required toprovide data and code replication files. We successfully replicate the key qualitativeresult of 22 of 67 papers (33%) without contacting the authors. Excluding the 6 papersthat use confidential data and the 2 papers that use software we do not possess, wereplicate 29 of 59 papers (49%) with assistance from the authors. Because we areable to replicate less than half of the papers in our sample even with help from theauthors, we assert that economics research is usually not replicable. We conclude withrecommendations on improving replication of economics research

The "file drawer problem" (a term coined in 1979 by Robert Rosenthal, a member of our Advisory Board) refers to the bias introduced into the scientific literature by selective publication--chiefly by a tendency to publish positive results but not to publish negative or nonconfirmatory results

My feeling is that the whole system is out of date and comes from a time when journal space was limited

Replication is key in science. Findings become robust and reliable only once they have survived various attempts to break them. Bem's second favour is to expose the well known secret that major journals simply won't publish replications. This is a real problem: in this age of Research Excellence Frameworks and other assessments, the pressure is on people to publish in high impact journals. Careful replication of controversial results is therefore good science but bad research strategy under these pressures, so these replications are unlikely to ever get run. Even when they do get run, they don't get published, further reducing the incentive to run these studies next time. The field is left with a series of "exciting" results dangling in mid-air, connected only to other studies run in the same lab.

It's back on my radar because several psychologists, including Richard Wiseman, recently submitted a failure to replicate the studies to the Journal of Personality & Social Psychology (JPSP), which is where Bem published his work. As reported here, Eliot Smith, the editor, refused to even send this (and another, successful replication as well) out for review. The reason Smith gives is that JPSP is not in the business of publishing mere replications - it prioritises novel results, and he suggests the authors take their work to other (presumably lesser) journals. This is nothing new - flagship journals like JPSP all have policies in place like this. But it's not a good look, and it got me thinking.

Scientists get criticised for not carrying out enough replications – there is little glory, after all, in merely duplicating old ground rather than forging new ones. Science journals get criticised for not publishing these attempts.  Science journalists get criticised for not covering them. This is partly why I covered Doyen’s study in the first place. In light of this “file drawer problem”, you might have thought that replication attempts would be welcome

These problems occur throughout the sciences, but psychology has a number of deeply entrenched cultural norms that exacerbate them. It has become common practice, for example, to tweak experimental designs in ways that practically guarantee positive results. And once positive results are published, few researchers replicate the experiment exactly, instead carrying out 'conceptual replications' that test similar hypotheses using different methods. This practice, say critics, builds a house of cards on potentially shaky foundations.

Three research teams independently tried to replicate the effect Bem had reported and, when they could not, they faced serious obstacles to publishing their results

Interesting papers are often highly cited by the scientific community regardless of journal.

To calculate the opportunity cost in citations as a result of delayed publication, I analyzed my first-authored papers that were first submitted to a high-impact journal and rejected. For each paper, there is a linear or quadratic relationship (r2 > 0.97) between years since publication and its number of citations in Google Scholar. If I had submitted each paper only to the journal it was published in, this would have resulted in an earlier publication date and more time to accrue citations. I calculated this opportunity cost. For example, the equation Citations = 2.3 ∗ YEARS2 + 12.7 ∗ YEARS – 3.9 (r2 = 0.996) describes the relationship between years since publication and number of citations of my most cited paper [7]. If I had first submitted this paper to the journal where it was eventually published in and had not lost time due to rejections and resubmissions, it would have been published 1.27 years earlier and would have accumulated approximately 70 more citations. On average, each resubmitted paper accumulated 47.4 fewer citations by being published later, with an overall opportunity cost of 190 lost citations.

Furthermore, the strict page limits of high-impact journals can force authors to omit valuable information [1], sometimes reducing the reach and citation rate of a paper. A rejection also forces a scientist to spend time revising and resubmitting, instead of working on new papers. The rejection–resubmission cycle sometimes demoralizes a scientist sufficiently that the paper is never published. Many of these papers have valuable ecology, natural history and life history data, often from the developing world, that are essential for improving the biodiversity databases and the global analyses based on them [7].

even when published in a lower-impact journal, good papers are usually recognized quickly by the scientific community, often achieving citation rates higher than those of high-impact journals

Çağan’s analysis shows quite unequivocally: the citation costs clearly outweigh the potential benefits of the rejection-resubmission cycle.

Morris, Sally. 2005. The True Costs of Scholarly Journal Publishing. Learned Publishing 18, no. 2: 115-126.

Rowland, Fytton. 2002. The Peer-Review Process. Learned Publishing 15, no. 4: 247-258.

Mark Ware Consulting Ltd. 2008. Peer Review in Scholarly Journals: Perspective of the Scholarly Community— an International Study. Bristol, UK: Publishing Research Consortium. http://www.publishingresearch.net/documents/PeerReviewFullPRCReport-final.pdf.

Research Information Network. 2008. Activities, Costs and Funding Flows in the Scholarly Communications System in the UK. London, UK: RIN. http://www.rin.ac.uk/our-work/communicating-and-disseminating-research/activities-costs-and-funding-flows-scholarly-commu.

Red flags in review Signs that an author might be trying to game the system A handful of researchers have exploited loopholes in peer-review systems to ensure that they review their own papers. Here are a few signs that should raise suspicions. The author asks to exclude some reviewers, then provides a list of almost every scientist in the field. The author recommends reviewers who are strangely difficult to find online. The author provides Gmail, Yahoo or other free e-mail addresses to contact suggested reviewers, rather than e-mail addresses from an academic institution. Within hours of being requested, the reviews come back. They are glowing. Even reviewer number three likes the paper.

Adopt preferred publication of negative over positive results; require very demanding reproducibility criteria before publishing positive results

Accept the current system as having evolved to be the optimal solution to complex and competing problems.Promote rapid, digital publication of all articles that contain no flaws, irrespective of perceived “importance”.Adopt preferred publication of negative over positive results; require very demanding reproducibility criteria before publishing positive results.Select articles for publication in highly visible venues based on the quality of study methods, their rigorous implementation, and astute interpretation, irrespective of results.Adopt formal post-publication downward adjustment of claims of papers published in prestigious journals.Modify current practice to elevate and incorporate more expansive data to accompany print articles or to be accessible in attractive formats associated with high-quality journals: combine the “magazine” and “archive” roles of journals.Promote critical reviews, digests, and summaries of the large amounts of biomedical data now generated.Offer disincentives to herding and incentives for truly independent, novel, or heuristic scientific work.Recognise explicitly and respond to the branding role of journal publication in career development and funding decisions.Modulate publication practices based on empirical research, which might address correlates of long-term successful outcomes (such as reproducibility, applicability, opening new avenues) of published papers.

scientists “selling” manuscripts.

The authority of journals increasingly derives from their selectivity. The venue of publication provides a valuable status signal. A common excuse for rejection is selectivity based on a limitation ironically irrelevant in the modern age—printed page space. This is essentially an example of artificial scarcity. Artificial scarcity refers to any situation where, even though a commodity exists in abundance, restrictions of access, distribution, or availability make it seem rare, and thus overpriced. Low acceptance rates create an illusion of exclusivity based on merit and more frenzied competition among scientists “selling” manuscripts.

Some unfavourable consequences may be predicted and some are visible. Resource allocation has long been recognised by economists as problematic in science, especially in basic research where the risks are the greatest. Rival teams undertake unduly dubious and overly similar projects; and too many are attracted to one particular contest to the neglect of other areas, reducing the diversity of areas under exploration [39]

Impact factors are widely adopted as criteria for success, despite whatever qualms have been expressed [27–32]. They powerfully discriminate against submission to most journals, restricting acceptable outlets for publication. “Gaming” of impact factors is explicit. Editors make estimates of likely citations for submitted articles to gauge their interest in publication. The citation game [33,34] has created distinct hierarchical relationships among journals in different fields. In scientific fields with many citations, very few leading journals concentrate the top-cited work [35]: in each of the seven large fields to which the life sciences are divided by ISI Essential Indicators (each including several hundreds of journals), six journals account for 68%–94% of the 100 most-cited articles in the last decade (Clinical Medicine 83%, Immunology 94%, Biochemistry and Biology 68%, Molecular Biology and Genetics 85%, Neurosciences 72%, Microbiology 76%, Pharmacology/Toxicology 72%).

The scientific publishing industry is used for career advancement [36]: publication in specific journals provides scientists with a status signal. As with other luxury items intentionally kept in short supply, there is a motivation to restrict access [37,38].

The acceptance rate decreases by 5.3% with doubling of circulation, and circulation rates differ by over 1,000-fold among 114 journals publishing clinical research [25]

Constriction on the demand side is further exaggerated by the disproportionate prominence of a very few journals. Moreover, these journals strive to attract specific papers, such as influential trials that generate publicity and profitable reprint sales. This “winner-take-all” reward structure [24] leaves very little space for “successful publication” for the vast majority of scientific work and further exaggerates the winner's curse.

A signalling benefit from the market—good scientists being identified by their positive results—may be more powerful in the basic biological sciences than in clinical research, where the consequences of incorrect assessment of positive results are more dire.

In the basic biological sciences, statistical considerations are secondary or nonexistent, results entirely unpredicted by hypotheses are celebrated, and there are few formal rules for reproducibility

Negative or contradictory data may be discussed at conferences or among colleagues, but surface more publicly only when dominant paradigms are replaced.

More alarming is the general paucity in the literature of negative data. In some fields, almost all published studies show formally significant results so that statistical significance no longer appears discriminating [15,16].

An empirical evaluation of the 49 most-cited papers on the effectiveness of medical interventions, published in highly visible journals in 1990–2004, showed that a quarter of the randomised trials and five of six non-randomised studies had already been contradicted or found to have been exaggerated by 2005 [9]

The scarcity of available outlets is artificial, based on the costs of printing in an electronic age and a belief that selectivity is equivalent to quality

The self-correcting mechanism in science is retarded by the extreme imbalance between the abundance of supply (the output of basic science laboratories and clinical investigations) and the increasingly limited venues for publication (journals with sufficiently high impact).

Scandals permeate social and economic life, but their consequences have received scant attention in the economics literature. To shed empirical light on this phenomenon, we investigate how the scientific community's perception of a scientist's prior work changes when one of his articles is retracted. Relative to non-retracted control authors, faculty members who experience a retraction see the citation rate to their earlier, non-retracted articles drop by 10% on average, consistent with the Bayesian intuition that the market inferred their work was mediocre all along. We then investigate whether the eminence of the retracted author and the publicity surrounding the retraction shape the magnitude of the penalty. We find that eminent scientists are more harshly penalized than their less distinguished peers in the wake of a retraction, but only in cases involving fraud or misconduct. When the retraction event had its source in “honest mistakes,” we find no evidence of differential stigma between high- and low-status faculty members.

Predictably, the strongest correlations were between Disruptive Innovativeness and Surprise, and between Surprise and Publication Difficulty

Overall, 123 scientists responded, scoring 1,214 papers between them. On average, investigators tended to give their blockbuster papers high scores for dimensions that reflect evolution:

Twenty scientists (16%) felt that their most important paper published in 2005–08 was not among their top ten most cited. However, most of these 20 papers were still heavily cited (on average in the top 3% published in the same year in terms of citations; seven were in the top 15 papers that the author published in 2005–08). Authors scored these papers higher for Disruptive Innovativeness (in nine cases) and Surprise (in five cases) than their ten most-cited papers.

They also indicated that blockbuster papers were easy to publish, with some exceptions.

We got some intriguing feedback. The vast majority of this elite group felt that their most important paper was indeed one of their most-cited ones. Yet they described most of their chart-topping work as evolutionary, not revolutionary.

We feel that by allowing grant holders to serve as grant reviewers, a conflict of interest becomes inescapable. Exceptional creative ideas may have difficulty surviving in such a networked system. Scientists who think creatively may be discouraged by the funding process and outcomes, or might not have time to contribute as reviewers to a process that is arduous and not perfectly meritocratic.

If NIH study-section members are well-funded but not substantially cited, this could suggest a double problem: not only do the most highly cited authors not get funded, but worse, those who influence the funding process are not among those who drive the scientific literature. We thus examined a random sample of 100 NIH study-section members. Not surprisingly, 83% were currently funded by the NIH. The citation impact of the 100 NIH study-section members was usually good or very good, but not exceptional: the most highly cited paper they had ever published as single, first or last author had received a median of 136 (90–229) citations and most were already mid- or late-career researchers (80% were associate or full professors). Only 1 of the 100 had ever published a paper with 1,000 or more citations as single, first or last author (see Appendix 1 of Supplementary Information for additional citation metrics).This overall picture (see 'Is funding tied to impact?') might, in part, be explained by the NIH policy to try to recruit reviewers who are successful in securing grants (see go.nature.com/kgtlrm). Even so, it is worrying that the majority of highly cited investigators do not have current NIH funding as principal investigators.

We found that the grants of study-section members were more similar to other currently funded NIH grants than were non-members' grants (median score 421.9 versus 387.6, p = 0.039). This could suggest that study-section members fund work that is more similar to their own, or that they are chosen to serve as study-section members because of similarities between their own and funded grants.

There are probably many reasons why highly cited scientists do not have current funding. They might have changed careers or moved to industry, for instance. Perhaps they are receiving some funding as co-investigators, or are still young and have just started their own lab. But the NIH's mandate is to fund “the best science, by the best scientists” — regardless of age or employment sector. We think our findings suggest that this aim is not being met.

However, concern is growing in the scientific community that funding systems based on peer review, such as those currently used by the NIH, encourage conformity if not mediocrity, and that such systems may ignore truly innovative thinkers2, 3, 4. One tantalizing question is whether biomedical researchers who do the most influential scientific work get funded by the NIH.The influence of scientific work is difficult to measure, and one might have to wait a long time to understand it5. One proxy measurement is the number of citations that scientific publications receive6. Using citation metrics to appraise scientists and their work has many pitfalls7, and ranking people on the basis of modest differences in metrics is precarious. However, one uncontestable fact is that highly cited papers (and thus their authors) have had a major influence, for whatever reason, on the evolution of scientific debate and on the practice of science.To explore the link between highly cited research and NIH funding, we evaluated scientists who have published papers since 2001 — as first, last or single authors — that have so far received 1,000 citations or more. We found that three out of five authors of these influential papers do not currently have NIH funding as principal investigators. Conversely, we found that a large majority of the current members of NIH study sections — the people who recommend which grants to fund — do have NIH funding for their work irrespective of their citation impact, which is typically modest.

Too many US authors of the most innovative and influential papers in the life sciences do not receive NIH funding, contend Joshua M. Nicholson and John P. A. Ioannidis.

While Nature felt they had already written enough about how the high-ranking journals publish unreliable research, Science had the impression the topic of journal rank and how it threatens the entire scientific enterprise was not general enough for their readership. Since there are not that many general science journals with sections fitting a review like ours, we next went to PLoS Biology. There, at least, the responsible editor, Catriona MacCallum (whom I respect very much and who is exceedingly likable) sent our manuscript out for review. To our surprise, the reviewers essentially agreed with Nature, that there wasn’t anything new in our conclusions: everybody already knows that high-ranking journals publish unreliable science, e.g.:

The common pattern seen where the decline effect has been documented is one of an initial publication in a high-ranking journal, followed by attempts at replication in lower-ranked journals which either failed to replicate the original findings, or suggested a much weaker effect (Lehrer, 2010).

Publication bias is also exacerbated by a tendency for journals to be less likely to publish replication studies (or, worse still, failures to replicate)

Some journals are devoted to publishing null results, or have sections devoted to these, but coverage is uneven across disciplines and often these are not particularly high-ranking or well-read (Schooler, 2011; Nosek et al., 2012)

However, a less readily quantified but more frequent phenomenon (compared to rare retractions) has recently garnered attention, which calls into question the effectiveness of this training. The “decline-effect,” which is now well-described, relates to the observation that the strength of evidence for a particular finding often declines over time (Simmons et al., 1999; Palmer, 2000; Møller and Jennions, 2001; Ioannidis, 2005b; Møller et al., 2005; Fanelli, 2010; Lehrer, 2010; Schooler, 2011; Simmons et al., 2011; Van Dongen, 2011; Bertamini and Munafo, 2012; Gonon et al., 2012). This effect provides wider scope for assessing the unreliability of scientific research than retractions alone, and allows for more general conclusions to be drawn.

At the same time, the current publication system is being used to structure the careers of the members of the scientific community by evaluating their success in obtaining publications in high-ranking journals. The hierarchical publication system (“journal rank”) used to communicate scientific results is thus central, not only to the composition of the scientific community at large (by selecting its members), but also to science's position in society.

Science is not a system of certain, or established, statements

we suggest that abandoning journals altogether, in favor of a library-based scholarly communication system, will ultimately be necessary

Now the study has been replicated. Three academics – Stuart Richie, Chris French, and Richard Wiseman – have re-run three of these backwards experiments, just as Bem ran them, and found no evidence of precognition. They submitted their negative results to the Journal of Personality and Social Psychology, which published Bem’s paper last year, and the journal rejected their paper out of hand. We never, they explained, publish studies that replicate other work.

In 2011, two American microbiologists looked at the rate of retractions among journals and found a strong correlation between the journal impact factor and the rate of retractions. You'll notice that some of the highest-impact journals (the New England Journal of Medicine, Science, and Cell) are furthest along in the "retraction index," a measure of the rate of retractions:

a) Non-retracted experiments reported in high-ranking journals are no more methodologically sound than those published in other journals. b) Non-retracted experiments reported in high-ranking journals are less methodologically sound than those published in other journals

Data from thousands of non-retracted articles indicate that experiments published in higher-ranking journals are less reliable than those reported in ‘lesser’ journals.

A single paper published in Nature, Cell, Science or other elite journals can set a scientist’s entire career on secure high ground. And a researcher with a grand string of such publication pearls, as well as prestigious grants, ascends to the scientific equivalent of a rock star. This leads to extreme competition for the precious few slots, and harms collaborative science

The right thing to do as a replicator of someone else's findings is to consult the original authors thoughtfully. If e-mails and phone calls don't solve the problems in replication, ask either to go to the original lab to reproduce the data together, or invite someone from their lab to come to yours. Of course replicators must pay for all this, but it is a small price in relation to the time one will save, or the suffering one might otherwise cause by declaring a finding irreproducible. When researchers at Amgen, a pharmaceutical company in Thousand Oaks, California, failed to replicate many important studies in preclinical cancer research, they tried to contact the authors and exchange materials. They could confirm only 11% of the papers3. I think that if more biotech companies had the patience to send someone to the original labs, perhaps the percentage of reproducibility would be much higher.

People trying to repeat others' research often do not have the time, funding or resources to gain the same expertise with the experimental protocol as the original authors, who were perhaps operating under a multi-year federal grant and aiming for a high-profile publication. If a researcher spends six months, say, trying to replicate such work and reports that it is irreproducible, that can deter other scientists from pursuing a promising line of research, jeopardize the original scientists' chances of obtaining funding to continue it themselves, and potentially damage their reputations.

So why am I concerned? Isn't reproducibility the bedrock of the scientific process? Yes, up to a point. But it is sometimes much easier not to replicate than to replicate studies, because the techniques and reagents are sophisticated, time-consuming and difficult to master. In the past ten years, every paper published on which I have been senior author has taken between four and six years to complete, and at times much longer. People in my lab often need months — if not a year — to replicate some of the experiments we have done on the roles of the microenvironment and extracellular matrix in cancer, and that includes consulting with other lab members, as well as the original authors.

research is becoming less pioneering and/or that the objectivity with which results are produced and published is decreasing.

Concerns that the growing competition for funding and citations might distort science are frequently discussed, but have not been verified directly. Of the hypothesized problems, perhaps the most worrying is a worsening of positive-outcome bias. A system that disfavours negative results not only distorts the scientific literature directly, but might also discourage high-risk projects and pressure scientists to fabricate and falsify their data.

Error was more common than fraud (73.5% of papers were retracted for error (or an undisclosed reason) vs 26.6% retracted for fraud). Eight reasons for retraction were identified; the most common reason was scientific mistake in 234 papers (31.5%), but 134 papers (18.1%) were retracted for ambiguous reasons. Fabrication (including data plagiarism) was more common than text plagiarism. Total papers retracted per year have increased sharply over the decade (r=0.96; p<0.001), as have retractions specifically for fraud (r=0.89; p<0.001). Journals now reach farther back in time to retract, both for fraud (r=0.87; p<0.001) and for scientific mistakes (r=0.95; p<0.001). Journals often fail to alert the naïve reader; 31.8% of retracted papers were not noted as retracted in any way.

Of these 119 statements, only 41.2% mentioned ethics at all (and only 32.8% named a specific ethical problem such as fabrication, falsification or plagiarism), whereas the other 58.8% described the reason for retraction or correction as error, loss of data or replication failure when misconduct was actually at issue.

What made the result “weak,” in Margot’s opinion, was that the authors presented only three times when the globules were present within one or two days of a full moon. With so few data points, it’s entirely possible the results were due to chance, he said: When I read their paper, I was very surprised to find that they only had three data points and they were claiming this association with the full moon on the basis of [these] three data points. You would expect to find multiple studies in the literature with that type of coincidence with the full moon. Margot appealed the journal’s decision to reject his letter, but was turned down again. Margot’s rebuttal was later published last October in the Journal of Biological Rhythms (JBR) after being rejected by two other biology journals (in addition to Biology Letters). (One month later, he published a corrigendum that presented a more accurate way of calculating the time to a full moon, but did not alter the argument or conclusions, he said.) Catarina Rydin, a botanist from Stockholm University in Sweden and co-author of the original study, defended her paper: The reasons we found it important to report our finding was not the strong statistical power of the results, but a number of small but relevant biological observations that all point in the same direction. And, more importantly, we wanted to report this so that also other scientists can contribute new information on the topic, in Ephedra and other plants. Commenting on Margot’s criticism, Rydin added: The paper by Margot is however not very interesting in our opinion, as it does not contribute any news to science; no new data, and no errors in our calculations. That our hypothesis is based on very few data points is clear from our paper, and the risk of Type I errors is known to every scientist. Sometimes, it is important to have the courage to present also quite bold hypotheses in order for science to progress.

Journals are inherently disinterested in negative findings

Journals are inherently disinterested in negative findings

“Science is great, but it’s low-yield,” Fang told me. “Most experiments fail. That doesn’t mean the challenge isn’t worth it, but we can’t expect every dollar to turn a positive result. Most of the things you try don’t work out — that’s just the nature of the process.” Rather than merely avoiding failure, we need to court truth.

“By default, we’re biased to try and find extreme results,” Ioannidis, the Stanford meta-science researcher, told me. People want to prove something, and a negative result doesn’t satisfy that craving.

Predatory journals flourish, in part, because of the sway that publication records have when it comes to landing jobs and grants, creating incentives for researchers to pad their CVs with extra papers.

From 2001 to 2009, the number of retractions issued in the scientific literature rose tenfold.

The setup was simple. Participants were all given the same data set and prompt: Do soccer referees give more red cards to dark-skinned players than light-skinned ones? They were then asked to submit their analytical approach for feedback from other teams before diving into the analysis. Twenty-nine teams with a total of 61 analysts took part. The researchers used a wide variety of methods, ranging — for those of you interested in the methodological gore — from simple linear regression techniques to complex multilevel regressions and Bayesian approaches. They also made different decisions about which secondary variables to use in their analyses. Despite analyzing the same data, the researchers got a variety of results. Twenty teams concluded that soccer referees gave more red cards to dark-skinned players, and nine teams found no significant relationship between skin color and red cards.

P-hacking is generally thought of as cheating, but what if we made it compulsory instead? If the purpose of studies is to push the frontiers of knowledge, then perhaps playing around with different methods shouldn’t be thought of as a dirty trick, but encouraged as a way of exploring boundaries. A recent project spearheaded by Brian Nosek, a founder of the nonprofit Center for Open Science, offered a clever way to do this.

Since publishing novel results can garner a scientist rewards such as tenure and jobs, there’s ample incentive to p-hack. Indeed, when Simonsohn analyzed the distribution of p-values in published psychology papers, he found that they were suspiciously concentrated around 0.05. “Everybody has p-hacked at least a little bit,” Simonsohn told me.

As you manipulated all those variables in the p-hacking exercise above, you shaped your result by exploiting what psychologists Uri Simonsohn, Joseph Simmons and Leif Nelson call “researcher degrees of freedom,” the decisions scientists make as they conduct a study. These choices include things like which observations to record, which ones to compare, which factors to control for, or, in your case, whether to measure the economy using employment or inflation numbers (or both). Researchers often make these calls as they go, and often there’s no obviously correct way to proceed, which makes it tempting to try different things until you get the result you’re looking for.

The p-value reveals almost nothing about the strength of the evidence, yet a p-value of 0.05 has become the ticket to get into many journals. “The dominant method used [to evaluate evidence] is the p-value,” said Michael Evans, a statistician at the University of Toronto, “and the p-value is well known not to work very well.” Scientists’ overreliance on p-values has led at least one journal to decide it has had enough of them. In February, Basic and Applied Social Psychology announced that it will no longer publish p-values. “We believe that the p < .05 bar is too easy to pass and sometimesserves as an excuse for lower quality research,”the editors wrote in their announcement. Instead of p-values, the journal will require “strong descriptive statistics, including effect sizes.”

Welcome to the wild world of p-hacking

how primitive the framework is that we use to evaluate policies and assess strength in science and technology

For decades, the DOD's legacy of innovation and economic growth concealed weaknesses in the civilian agencies, which is why so many people still believe that putting more money into civilian research and development is the panacea for what ails US innovation. Former presidential science adviser John Marburger publicly blew the whistle on this simple-minded notion in 2005, when he noted "how primitive the framework is that we use to evaluate policies and assess strength in science and technology". Partly in response to Marburger's provocation, the National Science Foundation initiated its Science of Science and Innovation Policy programme, with the explicit aim of guiding effective science policy-making by creating a foundation of data, theory, methods and models. This worthy goal carries an uncomfortable implication: that the nation's civilian research and development enterprise had been built on a foundation of hidden assumptions and unsubstantiated claims.

Meanwhile, the main civilian science agencies — the National Institutes of Health (NIH), the National Science Foundation, NASA and the Department of Energy (DOE) — developed roles in training scientists and creating knowledge that accelerated innovation. But such agencies were mere booster rockets for the DOD's main engine of innovation. They lacked, and continue to lack, the attributes that accounted for the military's successes — in particular, its focused mission, enduring ties to the private sector and role as an early customer for advanced technologies.

And yet excellence, measured by comparison with the work of others, is still used as the pre-eminent criterion to assess researchers and grant proposals. It seems that its main purpose is to act as a vague umbrella term, under which the genuine criteria can be concealed. As such, the term ‘excellence’ provides perfect cover and fulfils its role with distinction. Which is why it is in such ascendence.

We have become complicit in a system that systematically limits our funds to a fraction of what is needed to support us.

Scientists aspire to be—and drive each other to be—excellent. But excellence is a Trojan Horse that academics have accepted and taken into their midst. This is because the definition of excellence is unclear. For most people, it is one of those “I know it when I see it” things. But in policy terms this is too vague—in which case, it is clearly the top x per cent of proposals: 2, 3 or perhaps 5 per cent. This makes excellence very convenient. If an organisation commits to funding ‘excellence in science’, it has automatically committed to funding only x per cent of all the grants submitted. This saves the organisation a huge amount of money.

If most scientists are risk-averse, there is little chance that transformative research will occur, leading to significant returns from investments in research and development. Funding bodies sometimes give money specifically for field-changing research, but not nearly enough — Pioneer grants from the NIH fund fewer than 1% of applicants

Other economic incentives indirectly render the scientific process less efficient — such as the tendency of scientists to avoid risk by submitting to funding organizations only those proposals that they consider 'sure bets'

cash incentives adopted by countries such as China, South Korea and Turkey encourage local scientists to submit papers to high-end journals despite the low probability of success. These payments have achieved little more than overloading reviewers, taking them away from their work, and have increased submissions by the three countries to the journal Science by 46% in recent years, with no corresponding increase in the number of publications

Incentives that encourage people to make one decision instead of another for monetary reasons play an important part in science. This is good news if the incentives are right. But if they are not, they can cause considerable damage to the scientific enterprise.

FORCE11 Manifesto

better systems to permit collaborative work by geographically distributed colleagues; better systems to permit collaborative writing, with fail-safe versioning; better tools for richer interactive data and metadata visualization, enabling dynamic exploration; and easier data publication mechanisms, including better integration with data acquisition instrumentation, so that the process becomes automated.

Maximizing informal contacts through conferences, workshops, meetings, calls, webcasts

emergence of Force11 ̇OA provides

Not only are the products of research activity still firmly rooted in the past, so too are our means ofassessing the impact of those products and of the scholars who produce them. For five decades, theimpact of a scholarly work—an entity that is already narrowly defined, in the sciences as a journalarticle, and in the humanities as a monograph—has been judged by counting the number of citations itreceives from other scholarly works, or, worse, by attributing worth to an individual’s work based solelyon the overall impact factor of the journal in which it happens to be published. We now live in an agein which other methods of evaluation, including article-level usage metrics, blog comments, discussionon mail lists, press quotes, and other forms of media, are becoming increasingly important reflections ofscholarly and public impact. Failure to take these aspects into account means not only that the impactand/or quality of a publication is not adequately measured, but also that the current incentivization andevaluation system for scholars does not relate well to the actual impact of their activities

Third, research software developers typicallywork in a competitive environment, either academic or commercial, where innovation is rewarded muchmore highly than evolutionary and collaborative software reuse. This is especially true in a fundingenvironment driven by the need for intensive innovation, where reusing other peoples’ code is a likelysource of criticism

Theadvent of the internet has greatly reduced the monetary value that can be extracted from paper-basedacademic content

and so are un-likely to be a major source of continued revenue

Academic publishers have been slower to encounter, but are not immune from, the disruption that theinternet has wrought on other content industries

neitherhas an attractive business model