the RRR algorithm become more informative when using the number of citations accumulated for more time after publication

There are three main approaches to quantifying the impact of scientific publications

In other words, if a reviewer has proven to be very (or not very) capable of identifying high-quality papers in the past

Pearson correlation coefficient between the vector composed of all user ratings and the vector composed of the weighted average ratings of the corresponding items.

A reviewer-reputation ranking algorithm to identify high-quality papers during the review process

assuming that the number of citations of a paper is a good proxy for its intrinsic quality

exponential

In particular, network-based ranking algorithms became in the physics community due to their similarities with systems studied in statistical mechanics.

40 million annotations

Software tools, Databases, & Core Facilities RRIDs for software tools and databases, which do not always have a paper about them, are often suggested by SciScore, please verify that the tools listed are what was used and include the RRID in your paper. For shared facilities within your university, RRIDs are a great way to track their use. Please use RRIDs in the methods section or in the acknowledgement section. Authority: SciCrunch Registry (SCR) Homepage https://scicrunch.org/browse/resourcedashboard Submit Data https://scicrunch.org/resources/about/resource?form=Resource&rel=26 Example: PubMed (RRID:SCR_004846)

Plasmids Please add the full citation, including the RRID for each plasmid into your methods section. Authority: Addgene Homepage http://www.addgene.org/ Submit Plasmids http://www.addgene.org/depositing/start-deposit/ Example: pMD2.G plasmid (RRID:Addgene_12259)

Organisms Please check here for latest URLs for data submission (https://scicrunch.org/resources/about/guidelines#organism) Organisms obtained from stock centers should be listed with their stock center generated or approved RRIDs. Mice, authority: Mouse Genome Informatics (MGI) Rats, authority: Rat Genome Database (RGD) Worm, authority: Wormbase Fly, authority: Flybase Zebrafish, authority: Zebrafish Information Network (ZFIN) Xiphophorus, authority: XGSC Frog (Xenopus), authority: Xenbase Salamander (Ambystoma), authority: AGSC Pig, authority: NSRRC Tetrahymena, authority: Tetrahymena Stock Center Please note, each authority may have one or more associated stock centers. Please check with the RRID portal to find the right organism. Example: C57BL/6J mouse (RRID:IMSR_JAX:000664)

Cell Lines All cell lines should be listed with the company or source where they were obtained, the catalog number (if applicable), and the RRID from Cellosaurus. Cellosaurus contains the ICLAC.org list of problematic cell lines, thus looking at the cell line record ensures that authors are made aware of any warnings on cell lines or other known issues. Authority: Cellosaurus Homepage https://web.expasy.org/ Contact Page https://web.expasy.org/contact Example: HEK293 (NCBI_Iran Cat# C497, RRID:CVCL_0045)

Antibodies All antibodies should be listed with the company name, catalog number and RRID. It is also a good practice to add the lot number, just in case there is lot variability. Authority: Antibody Registry (ABR) Homepage https://antibodyregistry.org Submit Data https://antibodyregistry.org/add Example: Burchpilot antibody (DSHB Cat# nc82, RRID:AB_2314866)

Protocol Identifiers MDAR If we find protocol identifiers, then SciScore expects to find the following items (randomization, blinding, power); protocols.io URL or DOI, Nature Protocols DOI, etc.

Data Information: Data Availability MDAR If we find data, then SciScore expects to find the following items (randomization, blinding, power); Data Identifiers MDAR If we find data identifiers, then SciScore expects to find the following items (randomization, blinding, power); dbSNP, dbVar, Sequence Read Archive, BioProject, Protein Circular Dichroism Data Bank, ArrayExpress, GEO, European Genome-phenome Archive, Japanese Genotype-phenotype Archive, MassIVE, MetaboLights, PeptideAtlas, ProteomeXchange, FlowRepository, Image Data Resource, European Nucleotide Archive, UniProt, dbGaP, Biostudies, and ClinVar

Code Information: Code Availability MDAR If we find code information, then SciScore expects to find the following items (randomization, blinding, power); Code Identifiers MDAR URL from github, google code, bitbucket

Cell Line Contamination Check MDAR, NIH A statement addressing the mycoplasma contamination status of the cell lines used. This is only required when cell lines are detected. If we find a statement about contamination of cell lines, then SciScore expects to find the following items (sex, randomization, blinding, power); Example: All cell lines were obtained from ATCC and tested negative for mycoplasma contamination.

Cell Line Authentication MDAR, NIH A statement detailing how the cell lines used were authenticated (e.g. short tandem repeat analysis). This is only required when cell lines are detected. If we find a statement about cell line authentication, then SciScore expects to find the following items (sex, randomization, blinding, power); Example: MOLM-14 cells were authenticated by STR profiling and flow cytometry.

Replication Information Number of replications MDAR If we find a replication information, then SciScore expects to find the following items (randomization, blinding, power); Example: Bioassays were replicated three times.

Power analysis for group size MDAR, NIH, CONSORT, ARRIVE A statement addressing how (and if) an appropriate sample size was computed. If we find a statement about power analysis, then SciScore expects to find the following items (randomization, blinding); Example: Sample size was based on estimations by power analysis with a level of significance of 0.05 and a power of 0.9.

Blinding of investigator or analysis MDAR, NIH, CONSORT, ARRIVE A statement discussing the degree to which experimenters were unaware (or blinded) of group assignment and/or outcome assessment. Blinding in preclinical studies short video (Anita Bandrowski); short video of blinding types in clinical trials (Terry Shaneyfelt); tips for surgical trials Karanicolas et al (2010) If we find a blinding statement, then SciScore expects to find the following items (randomization, power); Example: Responses were then scored by an experimenter blinded to injection condition and experimental cohort.

Randomization of subjects into groups MDAR, NIH, CONSORT, ARRIVE Considered addressed when a statement describing whether randomization was used (e.g. assigning subjects to experimental groups, positions in a multiwell device, processing order, etc.). Good overview of topic Suresh 2011 (doi: 10.4103/0974-1208.82352) If we find a randomization statement, then SciScore expects to find the following items (blinding, power); Example: Animals were assigned to experimental groups using simple randomization.

Weight MDAR, ARRIVE Reporting the weight of any and all organisms and human subjects. If we find a weight statement, then SciScore expects to find the following items (sex, randomization, blinding, power); Example: Six healthy adult rhesus macaques (Macaca mulatta) of Chinese origin (4–8 kg, three males and three females, 4–8 years old) were inoculated intramuscularly (i.m.) with 1,000 pfu of EBOV Makona strain.

Subject Demographics Age MDAR, ARRIVE Reporting the age of any and all organisms and human subjects. For human subjects please see Implementation of the NIH Inclusion Across the Lifespan Policy; for animal subjects please see Training video module for the Vertebrate Animals If we find an age statement, then SciScore expects to find the following items (sex, randomization, blinding, power); Example: Their age varied from 19 to 47 years (mean 26.3 , ssd 6.4) and length of relationship from 4 months to 23 years (mean 3.7, ssd 4.4).

Sex as a biological variable MDAR, NIH, CONSORT, ARRIVE Reporting the sex of any and all organisms, cell lines, and human subjects. NIH Video explaining SABV policy If we find a statement about sex of subjects, then SciScore expects to find the following items (randomization, blinding, power); Example: All females were of reproductive age and none were on progestin.

Attrition MDAR A statement reporting the dropout of any subjects or samples. If we find a statement about attrition, then SciScore expects to find the following items (randomization, blinding, power); Example: Of these, 21 ticks could not be removed from the birds and 162 ticks were lost due to technical problems during nucleic acid extraction, resulting in 1,150 ticks available for analysis.

Inclusion & Exclusion Criteria MDAR, NIH, CONSORT Statement(s) discussing both the inclusion and exclusion criteria of the experiment. If we find a statement about inclusion and exclusion criteria, then SciScore expects to find the following items (randomization, blinding, power); Example: Subjects were eligible for the cross-sectional study if they were fluent in English and had a sexual partner (SP) in the previous 18 months and ineligible if they were post-menopausal or had undergone a sex change.

Euthanasia MDAR A statement addressing the method and/or agent used regarding the euthanasia of organisms. Authoritative source for information: Sivula & Suckow 2018 doi: 10.1201/9781315152189-35 If we find a euthanasia agent, then SciScore expects to find the following items ((iacuc), (ie_criteria or attrition), sex, (age or weight), randomization, blinding, power); Example: Mice were deeply anesthetized by intraperitoneal injection of sodium thiopental before decapitation, followed by brain extraction.

Field Sample Permit MDAR A statement (usually a single sentence) addressing field sampling approval for research (or why field approval was not required). If we find a permit, then SciScore expects to find the following items (randomization, blinding, power); Example: Permission to conduct field surveys on each location was given by the individual landowners concerned, and by the regulatory authority (Natural England) in those situations where the field site was afforded protected status (i.e. Site of Special Scientific Interest).

Institutional Animal Care and Use Committee Statement MDAR, ARRIVE A statement (usually a single sentence) addressing IACUC ethical approval for research involving vertebrate organisms. More information from the NIH Office of Laboratory Animal Welfare If we find an IACUC statement, then SciScore expects to find the following items ((ie_criteria or attrition), sex, (age or weight), randomization, blinding, power); Example: All animal experiments were performed in accordance with relevant guidelines and regulations and were approved by the University of Pennsylvania Institutional Animal Care and Use Committee (IACUC).

Consent Statement MDAR, CONSORT A statement (usually a single sentence) addressing subject/patient consent in human research (or why consent was not required). If we find a consent statement, then SciScore expects to find the following items ((ie_criteria or attrition), sex, (age or weight), randomization, blinding, power); Example: All infants were enrolled with informed parental permission under a protocol that was reviewed and approved by the Institutional Review Boards of the respective study sites.

Institutional Review Board Statement MDAR, CONSORT A statement (usually a single sentence) addressing IRB approval for biomedical research involving human subjects (or why IRB approval was not required). More information from authority: (please see instuctional materials about IRBs and human subject protections from NIH) If we find an IRB statement, then SciScore expects to find the following items ((ie_criteria or attrition), sex, (age or weight), randomization, blinding, power); Example: The trial was approved by the NRES Committee London—South East.

Based on the survey the ISCB membership is diverse in terms of ethnic origin (53% of those responding are of non-European descent) but heavily male biased. 

The 53 papers selected for replication in the RP:CB.

originally purchased from the Mutant Mouse Regional Resource Center

article-specific data

However, this tool does not include indicators of transparency (e.g., data or code sharing)

Resource Identification Initiative To improve the reproducibility of scientific research, the Resource Identification Initiative, aims to provide unique persistent identifiers for key biological resources, including antibodies, cell lines, model organisms and tools. We encourage authors to include unique identifiers - RRIDs- provided by the Resource Identification Portal in the dedicated section of the manuscript. To help authors quickly find the correct identifiers for their materials, there is a single web site where all resource types can be found and a ‘cite this’ button next to each resource, that contains a proper citation text that should be included in the methods section of the manuscript.

RRID:BDSC_50340

RRID:MMRRC_000001-UNC

AB_572208

RRID AB_2565444

AB_2223496

Rattus norvegicus

In stark contrast, the Proceedings of the National Academy of Sciences of the United States of America (PNAS), which put out several reports advocating for the need for increased rigor,22 showed no change in composite score: 3.33 in 2015 to 3.42 in 2019.

Figure

2433S

ATCC, # CRL-11268

A11122

A00702

1010-05

711-035-152

F3165

C57BL/6

β-actin

Cat# JAX_000664, RRID: IMSR_JAX:000664),

SCIBOT_SAYS_UNRESOLVED_CURATOR_SAYS_RRID_AND_UNRECOGNIZED

RRID:SCR_003070

For this study, we downloaded and processed all open access literature available through PubMed Central (PMC, RRID:SCR_004166) in September of 2019. In total, we obtained data from 1,578,964 articles from 4,686 unique journals.

___ Examples of references. Note that when citing RRIDs, the URL must follow the citation format as shown below: Journal Citation: Binoux M, Hossenlopp P. Insulin-like growth factor (IGF) and IGF-binding proteins: comparison of human serum and lymph. J Clin Endocrinol Metab. 1988;67(3):509–514. Abstract Citation: MacLaughlin DT, Cigarros F, Donahoe PK. Mechanism of action of Mullerian inhibiting substance. Program of the 70th Annual Meeting of the Endocrine Society, New Orleans, LA, 1988, p 19 (Abstract P1-21). Book Citation: Bonneville F, Cattin F, Dietemann J-L. Computed tomography of the pituitary gland. Heidelberg: Springer-Verlag; 1986; 15–16. Book Chapter Citation: Burrow GN The Thyroid: nodules and neoplasia. In: Felig P, Baxter JD, Broadus AE, Frohman LA, eds. Endocrinology and metabolism. 2nd ed. New York: McGraw-Hill; 1987:473–507. Repository Citation: Brown C, Jones M, Cohen M. Data from: Medical device-regulation process: review of safety notices and alerts. Dryad Digital Repository 2017. Deposited 2 January 2018. http://doi.org/10.9561/dryad.585t4 Antibody Citation: RRID:AB_2629219 Cell Line Citation: RRID:CVCL_1H60 Organism Citation: RRID:MMRRC_048263-UCD Plasmid Citation: RRID:ADDGENE_104005 Tool Citation: RRID:SCR_007358

recommended name (most frequently, the name provided in the referenced publication) and a list of synonyms;

RRID:MGI:5882597

Early-career researchers do not need to wait passively for coveted improvements. We can create communities and push for bottom-up change.

October

limit

Because SciScore is completely automated, it is able to operate with far more consistency and speed than peer reviewers, while still picking up even the most minute details.

publishers

surprisingly

their

later reproduce the experiment being analyzed.

​SciCrunch Inc. was founded in October of 2015 by Drs. Bandrowski and Martone. They have been focused on reproducibility-based issues in the contemporary sciences over the last 15+ years and have developed and lead hundreds of people across dozens of projects during that time including Force11 and the Resource Identification Initiative, which gave rise to the RRID standard now used throughout thousands of journals.  

founding

Secured NIH funding.

As a result, this limits the time spent on each research paper, increasing a publisher’s efficiency and the overall reproducibility of research published.

submitted to publishers, and increasing reproducibility due to better methods reporting.

Peer reviewers are doing a subpar job in enforcing best practices as they focus their attention more on the science and less on the small yet surprisingly important details of the methods.

despite using datasets identical to the originals.

an attempt was made to contact the original authors, discuss the discrepant findings, exchange reagents and repeat experiments under the authors' direction, occasionally even in the laboratory of the original investigator

downloaded as a zip file

knockdown

For SciScore, we used standard measures for classification performance: precision P, recall R, and harmonic mean of precision and recall F1. These are defined by the following formulas: the number (#) of correctly recognized labels refers to the number of words that are recognized as cell lines, which were also a cell line according to the curators. The number (#) of true good labels refers to the total number of cell lines according to the curators. P, R, and F1 are calculated on each 10% test set. P= (# of correctly recognized good labels) / (# of recognized good labels) R= (# of correctly recognized good labels) / (# of true good labels) F1=(2*P*R) / (P+R)

Related to

SciBot.

while the Resource Identification Initiative portal [13] contains mainly technologies from PubMed that were manually annotated by curators.

Abstract A08: Extensive subclonal mutations in human colorectal cancers detected by Duplex Sequencing

Abstract LB-338: Extensive subclonal mutations in human colorectal cancers detected by duplex sequencing

disorders

Peter's anomaly

sc-138763 from Santa Cruz Biotechnology Inc

for example if a cell line is known to be contaminated

RRID: RRID:SCR_006397

Jax:004453; RRID:IMSR_JAX:004453

CD43BD PharMingenClone S7; RRID: AB_10895376

anti-hCD49b-PEBDCat#555669; RRID: AB_396022

BD BiosciencesCat#565564; RRID:AB_2722547

Resource information A simple three-column table allows full reporting of reagent and resource information, including their source and identifiers.

Mouse Phenome Database: an integrative database and analysis suite for curated empirical phenotype data from laboratory mice.

Mouse Phenotype Database Integration Consortium: integration [corrected] of mouse phenome data resources.

Mouse Phenome Database: an integrative database and analysis suite for curated empirical phenotype data from laboratory mice

Uniform resolution of compact identifiers for biomedical data.

Uniform resolution of compact identifiers for biomedical data

Neuroscience Information Framework (NIF) Encyclopedia of Computational Neuroscience 2015 | other DOI: 10.1007/978-1-4614-6675-8_459

However, Simson (2016) has pointed out that the relabeling of antibodies for commercial sale results in the same antibody being available from multiple sources, negating or diminishing the value of the RRID.

S-2000

The Antibody Registry: Reagent: Antibodies - SciCrunch

The AntibodyRegistry Database SciCrunch 2009 | other RRID: RRID:SCR_006397 http://identifiers.org/rrid/RRID%3ASCR_006397 URL: http://antibodyregistry.org http://antibodyregistry.org Source: AE Bandrowski Preferred source

Research Resource Identifiers. The FASEB Journal encourages the use of Research Resource Identifiers (RRIDs), which have been developed in support of NIH's guidelines for rigor and transparency in biomedical publications. RRIDs may be included with other, required, information in the Materials and Methods section, but may not be used as a substitute. Readers without access to RRID information must have enough information to replicate experiments. For more information about RRIDs, visit https://scicrunch.org/resources.

such as title, abstract or keywords

This list holds 451 cell lines

Eteplirsen for the treatment of Duchenne muscular dystrophy

Scientific rigor: reproducibilityOn April 5, 2017, we added new guidelines for statistical reporting to the JNeurosci Instructions for Authors. While our reviewers have always commented on aspects of experimental design and statistical analyses when evaluating a manuscript, we wanted to create guidelines that would be relevant across the neuroscience field and support effective evaluation by reviewers that was consistent in its rigorous standards for future replication. To create these guidelines, we convened a group of editors with expertise in different subdisciplines. The updated guidelines derived from these discussions focus on including more details of experimental design and making sure that this information is contained within a single section of the Materials and Methods section, entitled Experimental design and statistical analysis, where it can be found more easily.Consistent with these efforts to improve rigor and reproducibility, JNeurosci also encourages authors to include RRIDs for critical reagents used in their studies, including antisera, mouse lines, cell lines, etc.

Spinal cord

cannabinoid

MGC Fully Sequenced Human CEL cDNA

A proposal for validation of antibodies

antibodies: also report source, characteristics, dilutions and how they were validated

Figure 3. Percent correctly reported RRIDs.

Riding on sidewalks should be avoided and may be prohibited in some areas.

Identified Cellular Correlates of Neocortical Ripple and High-Gamma Oscillations during Spindles of Natural Sleep

The Extracellular Matrix Protein Brevican Limits Time-Dependent Enhancement of Cocaine Conditioned Place Preference

Hypoxia stimulates proliferation of rat neural stem cells with influence on the expression of cyclin D1 and c-Jun N-terminal protein kinase signaling pathway in vitro

The following primary antibodies were used and incubated overnight at 4 °C: mouse monoclonal anti-cyclin D1 (1:1000, Neomarker, Fremont, CA, USA)

Expression and purification of the nucleocapsid protein of Schmallenberg virus, and preparation and characterization of a monoclonal antibody against this protein

TNF-alpha neutralizing antibody blocks thermal sensitivity induced by compound 48/80-provoked mast cell degranulation

Fatty acid oxidation in macrophage polarization

RRID: MGI_2178111

Down Syndrome Developmental Brain Transcriptome Reveals Defective Oligodendrocyte Differentiation and Myelination

Peripheral monocytes are functionally altered and invade the CNS in ALS patients

Heterogeneous Optimization Framework: Reproducible Preprocessing of Multi-Spectral Clinical MRI for Neuro-Oncology Imaging Research

Context-Dependent Gait Choice Elicited by EphA4 Mutation in Lbx1 Spinal Interneurons

Experimental febrile seizures impair interastrocytic gap junction coupling in juvenile mice

NMDA Receptor-Dependent LTD Requires Transient Synaptic Incorporation of Ca2+-Permeable AMPARs Mediated by AKAP150-Anchored PKA and Calcineurin

Loss of OMA1 delays neurodegeneration by preventing stress-induced OPA1 processing in mitochondria

Netrin-1LacZ/+ (Ntn1Gt(pGT1.8TM)629Wcs) (referred to as Ntn-1LacZ/+)

α-Axonin-1

α-LH2

α-Sox1

α-Chx10

α-Tag-1

α-Robo3

α-Evx1/2

Mab-Tag-1

Mab-Nkx2.2

Mab-RC2

Mab-Nestin

Chicken α-GFP

α-β-galatosidase

Scalable Brain Atlas

3dBAR

CoCoMac

Tumor necrosis factor-inducible gene 6 protein: A novel neuroprotective factor against inflammation-induced developmental brain injury

Segregated Foxc2, NFATc1 and connexin expression at normal developing venous valves, and connexin-specific differences in the valve phenotypes of Cx37, Cx43, and Cx47 knockout mice

Sharing self-related information is associated with intrinsic functional connectivity of cortical midline brain regions

Psychiatric Risk Gene Transcription Factor 4 Regulates Intrinsic Excitability of Prefrontal Neurons via Repression of SCN10a and KCNQ1

NiftyFit: a Software Package for Multi-parametric Model-Fitting of 4D Magnetic Resonance Imaging Data

Pioneering Activity of the C-Terminal Domain of EBF1 Shapes the Chromatin Landscape for B Cell Programming

Generation of a Xenopus laevis F1 albino J strain by genome editing and oocyte host-transfer

Peripheral injury of pelvic visceral sensory nerves alters GFRα (GDNF family receptor alpha) localization in sensory and autonomic pathways of the sacral spinal cord

s-Adenosylmethionine Levels Govern Innate Immunity through Distinct Methylation-Dependent Pathways

Segregated Glycine-Glutamate Co-transmission from vGluT3 Amacrine Cells to Contrast-Suppressed and Contrast-Enhanced Retinal Circuits

Broadening the spectrum of controls for skin biopsy in painful neuropathies: spondylotic cervical myelopathy patients with painful feet

Sustained release of neurotrophin-3 via calcium phosphate-coated sutures promotes axonal regeneration after spinal cord injury

Individual heritable differences result in unique cell lymphocyte receptor repertoires of naïve and antigen-experienced cells

GABAergic Neuron-Specific Loss of Ube3a Causes Angelman Syndrome-Like EEG Abnormalities and Enhances Seizure Susceptibility

Retinal remodeling in human retinitis pigmentosa

KIF5C S176 Phosphorylation Regulates Microtubule Binding and Transport Efficiency in Mammalian Neurons

Methoxychlor and fenvalerate induce neuronal death by reducing GluR2 expression

Electrochemical detection of protein glycosylation using lectin and protein–gold affinity interactions

A distinct plasmablast and naive B-cell phenotype in primary immune thrombocytopenia

Lymphatic endothelial cells efferent to inflamed joints produce iNOS and inhibit lymphatic vessel contraction and drainage in TNF-induced arthritis in mice

Nanovesicle-targeted Kv1.3 knockdown in memory T cells suppresses CD40L expression and memory phenotype

Physiological function and inflamed-brain migration of mouse monocyte-derived macrophages following cellular uptake of superparamagnetic iron oxide nanoparticles—Implication of macrophage-based drug delivery into the central nervous system

Germline Competent Pluripotent Mouse Stem Cells Generated by Plasmid Vectors