- Nov 2016
-
scienceintheclassroom.org scienceintheclassroom.org
-
P. Azoulay, J. Graff-Zivin, D. Li, B. Sampat, Public R&D investments and private sector patenting: Evidence from NIH funding rules, NBER working paper 20889
This paper shows a link between grants and private-sector innovations and created a model to quantify the variation in funding for different fields.
Their results show an increase in private-sector patents by NIH.
-
R. K. Merton, Science 159, 56–63 (1968).
In this article, the sociological expression "the rich get richer and the poor get poorer", also called Matthew effect, is presented in the context of scientific publication.
Scientists who have received grants in the past are more likely to get more grants and produce results.
-
B. A. Jacob, L. Lefgren, J. Public Econ. 95, 1168–1177 (2011).
The authors of this paper evaluated the impact of NIH grants on publications. They concluded that researchers who did not get an NIH grant but simultaneously applied for others grants saw one more publication (+7%).
-
J. Berg, Productivity metrics and peer review scores: NIGMS feedback loop blog (2011)
This article is a reasonable hypothesis that preliminary data that contribute to receiving an outstanding peer review score likely lead to high visibility publications shortly after the grant is funded.
-
S. Cole, J. R. Cole, G. A. Simon, Science 214, 881–886 (1981).
This article is about one negative effect of peer review, that an individual scientist devotes so much time and energy to getting financial support that it takes away from their science.
Basically, a huge disadvantage of the peer review program is that scientists must spend too much time writing what they intend to research, rather than performing the research.
-
B. Alberts, M. W. Kirschner, S. Tilghman, H. Varmus, Proc. Natl. Acad. Sci. U.S.A. 111, 5773–5777 (2014).
Bruce Alberts, Marc W. Kirschner, Shirley Tilghman and Harold Varmus describe the advances in scientific knowledge and human health that have accrued as a result of the long-standing public investment in biomedical research.
-
we cannot directly assess whether the NIH systematically rejects high-potential applications
Because the authors only looked at projects that received grant funding, their analysis does not take into account how many high-potential projects were rejected by peer review.
-
our estimates are likely downward biased
The authors acknowledge that there is sometimes a long delay between a grant award and patenting, so their analysis may not be a good indicator of how relevant research is to commercial applications.
-
peer reviewers are more likely to reward projects with the potential for a very high-impact publication and have considerable ability to discriminate among strong applications
The authors' findings suggest that peer reviewers are good at identifying innovative and ground-breaking projects.
-
Our final analysis
Finally, the authors wanted to figure out if peer reviewers are good at choosing applicants for their innovation, their practicality, or if they are simply good at weeding out low-quality research.
-
peer reviewers add value by identifying the strongest research proposals
The authors show that peer review scores are good predictors of scientific productivity when differences in field of research, year, and applicant qualifications are removed. This suggests that peer reviewers have the necessary expertise to choose good applicants.
-
These residuals represent the portions of grants’ citations or publications that cannot be explained by applicants’ previous qualifications or by application year or subject are
The authors removed the influence of the grant applicant's background, demographics, and writing skill in order to look at what effect a reviewer's expertise has.
-
nonparametrically
Nonparametric statistical models are often used for data that is ranked.
-
covariates
A covariate is a variable that used in a regression analysis. It is a variable that might be responsible for the outcome of a study, or that might be interfering.
Here, all of the additional variables added in each model were covariates (writing ability, gender, ethnicity, etc.)
-
the grant with a 1-SD worse score is predicted to have 7.3% fewer future publications and 14.8% fewer future citations
The authors conclude here that regardless of gender, ethnicity, or institutional prestige, when the peer-review score lowers by one standard deviation, we can observe a corresponding decrease of the number of publications and citations of an author.
-
including these variables does not substantively affect our findings
The authors concluded that writing skills does not affect the chance of receiving grant funding from the NIH.
-
rewarding an applicant’s grant proposal writing skills
In this model, the authors try to control for the fact that an application could be selected because the applicant writes well, rather than based on the quality of the application
-
Matthew effect
The Matthew Effect can be summarized as "the rich get richer and the poor get poorer." It describes the idea that benefits are distributed unevenly, and that those who already have the benefits will continue to accumulate them while those without will not have the chance.
In scientific publication, the Matthew Effect refers to the phenomenon where researchers who are established publish more often simply because they are established (and regardless of the quality of their work).
-
Controlling for publication history attenuates but does not eliminate the relationship
Again, controlling for the variable of a PI's research background does not eliminate the relationship the authors originally found.
-
adds controls describing a PI’s publication history
The authors control for yet another potential variable, an applicant's research background (they use the PI's publication history to do this).
-
Controlling for cohort and field effects does not attenuate our main finding
The authors' adjustments to control for various external effects did not change their original findings.
-
We also include NIH institute-level fixed effects to control for differences in citation and publication rates by fields
The authors try to remove the effect of an article's field on its impact. For example, a biochemistry article may appear to have a smaller impact because of the high rate of publication and citation in that field, whereas a physics article's impact may be inflated due to a lower publication and citation rate.
-
potential concerns
Several factors can lead to the trends in Figure 1 being misinterpreted, like the age of the grant and the field of study. The authors address these concerns by adjusting their model to account for these effects.
-
Figure 1
Figure 1:
Tab 1 : Axis : The bottom axis is the peer review percentile score that defines how much the committee liked the application (the lower the percentile, the better).
In the first plot, the y axis is the number of citations an author got after receiving a grant.
In the second plot, the y axis is the number of publications an author produced after receiving a grant.
Tab 2 : Description In the first plot, we can see that there are more points in the top left corner than the bottom right corner. This indicates that applications that received better ratings also received more citations.
In the second plot, we see the same trend as in the first. This indicates that applications that received better ratings produced more publications.
-
a 1-SD worse score is associated with a 14.6% decrease in grant-supported research publications and a 18.6% decrease in citations to those publications
Here the authors estimated how much a decrease of one standard deviation on the percentile score affected the number of publications and citations of a grant recipient.
-
regression
Regression is a measure of the relation between the mean value of one variable and corresponding values of other variables. There are different types of regression, all of which are used to identify trends in data.
-
Poisson regressions
A Poisson regression is a form of regression analysis where we have a random variable, which is equal to the number of events over a period of time if these events are independent and occur at a constant speed.
-
This variation in citations underscores the potential gains from being able to accurately screen grant applications on the basis of their research potential
The authors found that there is a lot of variation in the research output of projects that receive grants. They conclude that it would be useful to find a way to accurately screen applications to determine their potential.
-
principal investigator (PI),
A principal investigator (PI) is the holder of an independent grant administered by a university and the lead researcher for the grant project, usually in the sciences.
The phrase is also often used as a synonym for "head of the laboratory" or "research group leader."
-
institutional affiliation
An applicant's institutional affiliation is the organization that has agreed to be the legal recipient of the grant. This organization can be a nonprofit, a university, or an employer.
-
U.S. Patent and Trademark Office (USPTO)
The United States Patent and Trademark Office (USPTO or Office) is an agency of the U.S. Department of Commerce which stores, classifies, and disseminates information on patents and gives grant patents for the protection of inventions and to register trademarks.
-
patents that either directly cite NIH grant support or cite publications acknowledging grant support
The last measure is the number of patents that cite those publications from (i), or acknowledge support from the grant.
-
the total number of citations that those publications receive through 2013
The second measure is the total number of citations the publications from (i) received through 2013.
-
PubMed
PubMed is a database of medical and biological publications, created by the National Center for Biotechnology Information (NCBI). It is the free version of the database MEDLINE.
-
the total number of publications that acknowledge grant support within 5 years of grant approval
The first measure of success is the number of papers a team published during the 5 years after they received the grant.
-
funding is likely to have direct effect on research productivity
The authors considered grants which were already funded and competing for renewal. This makes it easier to attribute differences in research productivity to the peer review process, rather than the amount of funding the project has.
-
standard deviation (SD)
Standard deviation is a statistical measure that is used to describe how much variation there is in a data set. A high standard deviation means that the data is very spread out.
-
percentile score
The percentile score is assigned by the peer review committee. It ranks all authors to determine which was the most favored by the committee. A lower score means the committee liked the application more.
-
NIH is the world’s largest funder of biomedical research (12). With an annual budget of approximately $30 billion, it supports more than 300,000 research personnel at more than 2500 institutions (12, 13). A funding application is assigned by topic to one of approximately 200 peer-review committees (known as study sections).
Based on an analysis conducted by the authors, biomedical research is valued highly by individuals, governments, foundations, and corporations. Research is seen as a source of more effective treatments and preventive measures and as a route to policy, new commercial products, and economic development.
As a result, investments in biomedical research are the highest of all sectors.
-
Because research outcomes are often skewed, with many low-quality or incremental contributions and relatively few ground-breaking discoveries
One critique about peer reviewing is that peer review may not identify pioneering research.
This report highlights the possible disadvantages of peer review:
http://www.theepochtimes.com/n3/1334826-does-peer-review-pick-the-best-science/
-
peer review has high value-added if differences in grants’ scores are predictive of differences in their subsequent research output
If the evaluation by the peer review committee is correlated with the quality of work put out by the research group, then peer review has high value-added (meaning, it is useful for choosing research groups with the highest potential).
-
Because NIH cannot possibly fund every application it receives, the ability to distinguish potential among applications is important for its success.
The outcome of this study could have important implications for how the NIH evaluates and chooses who it gives money to.
-
our paper asks whether NIH selects the most promising projects to support
Previous work has shown that receiving a grant increases scientific productivity. However, the paper authors want to know if the NIH is awarding grants to projects that will make the best use of the money.
-
Whereas previous work has studied the impact of receiving NIH funds on the productivity of awardees
These articles show that receiving a grant for postdoctoral research leads to an increase in productivity.
-
“value-added.”
Value-added is the amount by which the value of the product is increased. Here, it means by how much peer review increases new insights about the quality of grant applications.
-
peer review generates new insights about the scientific quality of grant applications
The effectiveness of peer review has been refuted many times before:
-
applicants from elite institutions
All that separates individual investors is access to the best ideas and powerful research tools.
See this marketwatch article about how social media is present in our daily lives and how much it can create a connection between the ideas and investors: http://www.marketwatch.com/story/social-medias-next-disruption-the-investment-industry-2016-06-09?siteid=rss&rss=1
-
Existing research in this area has focused on understanding whether there is a correlation between good peer-review scores and successful research outcomes and yields mixed results
In these articles, the authors found out that there was no link between higher ratings from the peer-review committee and the number of citations the article eventually got.
-
Disagreement about what constitutes important research may introduce randomness into the process
In this article, the authors showed that getting a research grant depends partially on chance. They reviewed the same proposals with different committees, who each gave different results.
-
Peer-review committees
The aim of the peer-review committees is to both ensure the quality of research and encourage innovation. However, it has been shown that peer review committees can be undermined by various factors.
-
In 2014, the combined budgets of the U.S. National Institutes of Health (NIH), the U.S. National Science Foundation, and the European Research Council totaled almost $40 billion.
In 2016, the combined budgets totaled $41.7 billion. From this we can see that the research budget continues to increase.
https://www.nih.gov/about-nih/what-we-do/budget http://www.nsf.gov/about/budget/fy2016/ https://erc.europa.eu/about-erc/facts-and-figures
-
high-impact
The impact factor (IF) is a numerical indicator of the "importance" of a scientific journal or article, calculated based on the number of citations and published articles.
-
U.S. National Institutes of Health
The National Institutes of Health (NIH) is an agency of the Department of Health and Human Services and the main agency of the U.S. government responsible for biomedical research and healthcare-related research.
-
grants
A grant is a money given to enterprises, organizations, and individuals for research, development, and education. Grants do not have to be returned, but most granting organizations require a report about the progress of the outcome.
-
research project
The Research Project (R01) is a type of grant awarded by the National Institutes of Health (NIH) that provides support for health-related research and development.
-
- Sep 2016
-
scienceintheclassroom.org scienceintheclassroom.org
-
W. R. Kerr, The ethnic composition of US inventors, Working Paper 08-006, Harvard Business School (2008)
This study applies an ethnic-name database to individual patent records granted by the United States Patent and Trademark Office to document these trends with greater detail than previously available.
-
D. F. Horrobin, JAMA 263, 1438–1441 (1990).
The main goal of peer review in the biomedical sciences is to facilitate the introduction into medicine of improved ways of curing, relieving, and comforting patients. The achievement of this aim requires both quality control and the encouragement of innovation. If an appropriate balance between the two is lost, then peer review will fail to reach its purpose.
-
We control for the same variables as described in Model 6 of Table 1.
The patents like the grants are checked according to certain indicators: institutional quality, gender, and ethnicity of applicants.
-
Although our findings show that NIH grants are not awarded purely for previous work or elite affiliations and that reviewers contribute valuable insights about the quality of applications, mistakes and biases may still detract from the quality of funding decisions.
Summing up all results we can say that previous work or elite affiliations do not "close the door" for new ideas in research.
-
probabilistic algorithm
The probabilistic algorithm is algorithm that, providing circulation on the certain stages of its work to the random number generator in order to obtain savings in work time.
-
correlation
A correlation describes how two variables are linked one to the other.
-
Funds of public organisms such as NIH are given through grants after peer-review applications. Although, some voices raised to protest against this system that might be biased for the known applicants who are supposed to be favored.
This paper analyses a big set of data concerning NIH applications to determine if this phenomenon is actually observed or not.
-
Will your ideas or your name bring you more grants ?
-
National Science Foundation
The National Science Foundation (NSF) is an independent agency of the U.S. government that is responsible for the development of science and technology. The Foundation carries out it's mission by providing, in general, temporary grants.
-
Our regression results include separate controls for each type of publication: any authorship position, and first or last author publications
What the authors mean here is that they made statistical computations that allow them to remove the effect that the position of a name in the authors row can have in a publication
-
Fig. 3
Figure 3 :
Tab 1 : Panes : Each of these plots are made with a different set of data. In the first of the plots is made with the data concerning only the applications that published the most (i.e. the 0,1% best). The two last plots are also made only with the applications that published the least (i.e. 50 and 20% worse).
Tab 2 : Axis : The bottom axis of these plots is the percentile score of each application, that defines how the committee like the application (the lower the better). The y axis of these plots represents the relative percentile of citations that an application get per percentile score compared to publications near the 10% best. This means that the higher on the y axis it is, the more (or less) cited it has been compared to the number of citations that the publications around the 10% best got.
Tab 3 : Description : We can see in the 4 first plots that concern the applications that published the most that the more the application got a good rate from the committee, the more likely it will be cited. Although, this phenomenon is more obvious in the top 5 and 10%. In the two last plots, the curve is likely a straight horizontal line. It is especially obvious in the last plot. This means that the least an application publishes after a grant, the less it will be cited regardless of the rate of the committee.
-
Fig. 2
Figure 2 :
Tab 1 : Axis : The bottom axis is for both the plots the percentile score, that defines how the committee like the application (the lower the percentile, the better). For the first plot, the y axis is the residual number of citation an application gets after getting a grant. The residual number of citations is a statistical output that allows to generalise the number of citations by percentile score. For the second plot, the y axis is the residual number of publications a n application gets after getting a grant. The residual number of publications is a statistical output that allows to generalise the number of publications by percentile score.
Tab 2 : Description : On the first plot, we can see that the slope on the left is really steep. That shows that the committees can tell really effectively if an application has some potential. We can also see that the slope is changing around 60% of the percentile score. The authors tell that this is probably due to the ability of the program officers to detect high potential in applications that had a bad percentile score.
-
measure applicant-level characteristics
The authors studied some characteristics such as the grant history or the institutional affiliation to see if the previous work of the applicant has an impact on the result of the grant application.
-
we employ a probabilistic algorithm developed by Kerr to determine applicant gender and ethnicity (Hispanic or Asian)
The algorithm in question was developped by William Kerr to estimate the contribution of Chinese and Indian scientists to the US Patent and Trademark Office.
-