402 Matching Annotations
  1. Jan 2024
  2. Oct 2023
  3. Aug 2023
  4. Mar 2023
    1. German academic publishing in Niklas Luhmann's day was dramatically different from the late 20th/early 21st centuries. There was no peer-review and as a result Luhmann didn't have the level of gatekeeping that academics face today which only served to help increase his academic journal publication record. (28:30)

  5. Nov 2022
    1. not really about the content of the sessions. Or anything you take from it. The most important thing are the relationships, the connections you gain from sharing the things you're passionate about with the people who are interested in it, the momentum you build from working on your project in preparation for a session

      I somewhat disagree - I think this community building is successful precisely because there is a shared interest or goal. It goes hand in hand. If there is no connecting theme or goal, the groups fall apart.

    1. I also think being able to self-host and export parts of your data to share with others would be great.

      This might be achievable through Holochain application framework. One promising project built on Holochain is Neighbourhoods. Their "Social-Sensemaker Architecture" across "neighbourhoods" is intriguing

  6. Sep 2022
    1. social groups create a pressure toward conformity so powerful that it can overcome individual preferences, and by amplifying random early differences, it can cause segregated groups to diverge to extremes.
  7. Jul 2022
    1. Perhaps the most widely recognized failing of peer review is its inability to ensure the identification of high-quality work.

      stakesinscience

  8. May 2022
    1. Studying, done properly, is research,because it is about gaining insight that cannot be anticipated and willbe shared within the scientific community under public scrutiny.

    Tags

    Annotators

    1. or at least they pretend

      I don't think we're pretending. I know I'm not!

    2. Senior colleagues indicate that I should not have to balance out publishing in “traditional, peer-reviewed publications” as well as open, online spaces.

      Do your colleagues who read your work, annotate it, and comment on it not count as peer-review?

      Am I wasting my time by annotating all of this? :) (I don't think so...)

    1. He notes that authors of such projects should consider the return on investment. It take time to go through community feedback, so one needs to determine whether the pay off will be worthwhile. Nevertheless, if his next work is suitable for community review, he’d like to do it again.

      This is an apropos question. It is also somewhat contingent on what sort of platform the author "owns" to be able to do outreach and drive readers and participation.

    2. A short text "interview" with the authors of three works that posted versions of their books online for an open review via annotation.

      These could be added to the example and experience of Kathleen Fitzpatrick.

    1. I returned to another OER Learning Circle and wrote an ebook version of a Modern World History textbook. As I wrote this, I tested it out on my students. I taught them to use the annotation app, Hypothesis, and assigned them to highlight and comment on the chapters each week in preparation for class discussions. This had the dual benefits of engaging them with the content, and also indicating to me which parts of the text were working well and which needed improvement. Since I wasn't telling them what they had to highlight and respond to, I was able to see what elements caught students attention and interest. And possibly more important, I was able to "mind the gaps', and rework parts that were too confusing or too boring to get the attention I thought they deserved.

      This is an intriguing off-label use case for Hypothes.is which is within the realm of peer-review use cases.

      Dan is essentially using the idea of annotation as engagement within a textbook as a means of proactively improving it. He's mentioned it before in Hypothes.is Social (and Private) Annotation.

      Because one can actively see the gaps without readers necessarily being aware of their "review", this may be a far better method than asking for active reviews of materials.

      Reviewers are probably not as likely to actively mark sections they don't find engaging. Has anyone done research on this space for better improving texts? Certainly annotation provides a means for helping to do this.

    1. However, the degraded performance across all groups at 6 weeks suggests that continued engagement with memorised information is required for long-term retention of the information. Thus, students and instructors should exercise caution before employing any of the measured techniques in the hopes of obtaining a ‘silver bullet’ for quick acquisition and effortless recall of important data. Any system of memorization will likely require continued practice and revision in order to be effective.

      Abysmally sad that this is presented without the context of any of the work over the last century and a half of spaced repetition.

      I wonder that this point slipped past the reviewers and isn't at least discussed somewhat narratively here.

  9. Apr 2022
    1. A 2019 study published in the Proceedings of the National Academy ofSciences supports Wieman’s hunch. Tracking the intellectual advancement ofseveral hundred graduate students in the sciences over the course of four years,its authors found that the development of crucial skills such as generatinghypotheses, designing experiments, and analyzing data was closely related to thestudents’ engagement with their peers in the lab, and not to the guidance theyreceived from their faculty mentors.

      Learning has been shown to be linked to engagement with peers in social situations over guidance from faculty mentors.

      Cross reference: David F. Feldon et al., “Postdocs’ Lab Engagement Predicts Trajectories of PhD Students’ Skill Development,” Proceedings of the National Academy of Sciences 116 (October 2019): 20910–16


      Are there areas where this is not the case? Are there areas where this is more the case than not?

      Is it our evolution as social animals that has heightened this effect? How could this be shown? (Link this to prior note about social evolution.)

      Is it the ability to scaffold out questions and answers and find their way by slowly building up experience with each other that facilitates this effect?

      Could this effect be seen in annotating texts as well? If one's annotations become a conversation with the author, is there a learning benefit even when the author can't respond? By trying out writing about one's understanding of a text and seeing where the gaps are and then revisiting the text to fill them in, do we gain this same sort of peer engagement? How can we encourage students to ask questions to the author and/or themselves in the margins? How can we encourage them to further think about and explore these questions? Answer these questions over time?

      A key part of the solution is not just writing the annotations down in the first place, but keeping them, reviewing over them, linking them together, revisiting them and slowly providing answers and building solutions for both themselves and, by writing them down, hopefully for others as well.

  10. Mar 2022
  11. Feb 2022
  12. Jan 2022
  13. Dec 2021
    1. What is P2P (peer-to-peer) and what can you do with it? https://www.itpedia.nl/wp-content/uploads/2018/12/fingerworld.jpg.webp In a sense, Peer to Peer (P2P) networks are the social networks of the Internet. Every peer is equal to the others, and every peer has the same rights and duties as the others. Peers are clients and servers at the same time.

    1. AIMOS. (2021, November 30). How can we connect #metascience to established #science fields? Find out at this afternoon’s session at #aimos2021 Remco Heesen @fallonmody Felipe Romeo will discuss. Come join us. #OpenScience #OpenData #reproducibility https://t.co/dEW2MkGNpx [Tweet]. @aimos_inc. https://twitter.com/aimos_inc/status/1465485732206850054

  14. Nov 2021
    1. Social Annotation

      SA is also known as Collaborative Annotation in writing studies, as it focuses on peer-to-peer annotation of a shared document.

    1. 4 Best Payment Solutions for Online MarketplacesDmitryCEOMarketplaceHomeBlogEntrepreneurship4 Best Payment Solutions for Online MarketplacesPublishedAug 7, 2020UpdatedAug 7, 20209 min readDid you know that payment solutions for online marketplaces can shape your e-commerce business and its success? Thus, Uber succeeded in its global expansion right after it switched to Braintree. In early Uber’s scaling, even a dollar-euro currency conversion wasn’t available. Now, with Braintree, it processes mobile payments in 130 currencies in 80+ countries. Of course, each marketplace faces its own payment challenges. So, you should rely on a payment solution with the features vital right for your e-commerce platform. To identify them, let’s dig deeper into two-sided marketplace payment processing, and analyze the best payment gateways for marketplaces.

      Did you know that payment solutions for online marketplaces can shape your e-commerce business and its success? Thus, Uber succeeded in its global expansion right after it switched to Braintree.

      In early Uber’s scaling, even a dollar-euro currency conversion wasn’t available. Now, with Braintree, it processes mobile payments in 130 currencies in 80+ countries.

      Of course, each marketplace faces its own payment challenges. So, you should rely on a payment solution with the features vital right for your e-commerce platform. To identify them, let’s dig deeper into two-sided marketplace payment processing, and analyze the best payment gateways for marketplaces.

    1. I have no problem with publishers making a profit, or with peer reviewers doing their work for free. The problem I have is when there is such an enormous gap between those two positions.

      If publishers make billions in profit (and they do), while at the same time reviewers are doing a billion dollars worth of work for free, that seems like a broken system.

      I think there are parallels with how users contribute value to social media companies. In both cases, users/reviewers are getting some value in return, but most of the value that's captured goes to the publisher/tech company.

      I'd like to see a system where more of the value accrues to the reviewers. This could be in the form of direct payment, although this is probably less preferable because of the challenges of trying to convert the value of different kinds of peer review into a dollar amount.

      Another problem with simply paying reviewers is that it retains the status quo; we keep the same system with all of it's faults and redistribute profits. This is an OK option as it at least sees some of the value that normally accrues to publishers moving to reviewers.

      I also don’t believe that open access - in it’s current form - is a good option either. There are still enormous costs associated with publishing; the only difference is that those costs are now covered by institutions instead of the reader. The publisher still makes a heart-stopping profit.

      A more elegant solution, although more challenging, would be for academics to step away from publishers altogether and start their own journals, on their own terms.

    1. COVID-19 Living Evidence. (2021, November 12). As of 12.11.2021, we have indexed 257,633 publications: 18,674 pre-prints 238,959 peer-reviewed publications Pre-prints: BioRxiv, MedRxiv Peer-reviewed: PubMed, EMBASE, PsycINFO https://t.co/ytOhLG90Pi [Tweet]. @evidencelive. https://twitter.com/evidencelive/status/1459163720450519042

    1. The censoriousness, the shunning, the ritualized apologies, the public sacrifices—these are rather typical behaviors in illiberal societies with rigid cultural codes, enforced by heavy peer pressure.

      I'd highlighted this from a pull quote earlier, but note that the full context also includes the phrase:

      enforced by heavy peer pressure.

  15. Oct 2021
  16. Aug 2021
    1. 7 Top Strategies To Attract Sellers to Your Peer to Peer MarketplaceDaniil TorkutDeveloper AdvocateMarketplaceHomeBlogEntrepreneurship7 Top Strategies To Attract Sellers to Your Peer to Peer MarketplaceJun 4, 20208 min readUser acquisition is one of the most important aspects to make your marketplace website successful. When you attract sellers and buyers to your platform, you start making a profit and improve your market presence. Our previous article covered the topic of attracting buyers for a peer to peer marketplace. We have considered 4 simple techniques that will help you solve the chicken and egg problem even before you launch your marketplace. These are a referral system, landing page, free offers, and constraints that can help you attract early buyers. Today we want to discuss the second important aspect - acquisition of sellers. We prepared 7 quick ways to attract vendors to your peer to peer marketplace platform. This detailed guide will help you make your e-commerce website profitable and successful.

      User acquisition is one of the most important aspects to make your marketplace website successful. When you attract sellers and buyers to your platform, you start making a profit and improve your market presence.

      We prepared 7 quick ways to attract vendors to your peer to peer marketplace platform.

  17. Jul 2021
  18. Jun 2021
    1. Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J., Thorn, F. S., Vazire, S., Esterling, K. M., & Nosek, B. A. (2021). Initial evidence of research quality of registered reports compared with the standard publishing model. Nature Human Behaviour, 1–8. https://doi.org/10.1038/s41562-021-01142-4

    1. recently published book

      I was honored to interview Remi and Antero (along with other MITP authors) about collaborative community review and how it fit with their traditional peer review experience. The blog post can be found here.

    1. Publisher costs usually include copyediting/formatting and organizing peer review. While these content transformations are fundamental and beneficial, they alone cannot justify the typical APC (Article Publication Charge), especially since peer reviewers are not paid.

      But peer reviewers are largely responsible for generating the assertions you talk about in the next paragraph, and which apparently, justify the cost of publishing.

  19. May 2021
  20. Apr 2021
    1. Note: This rebuttal was posted by the corresponding author to Review Commons. Content has not been altered except for formatting.

      Learn more at Review Commons


      Reviewer #1

      1. One key citation missing from the current manuscript is from Hwang et al. 2014 (PMID 25288734). This study has already described that the isp-1 mutant strain survives longer during P. aeruginosa infection. This citation also describes that the gene expression profile of isp-1 mutants animals includes a considerable number of pathogen-responsive genes that are similarly induced during infection. While the current manuscript does go into the mechanism of this resistance with more detail, they should amend the language to more appropriately reflect previous work, notably the above reference.

      We apologize for the oversight and have added the suggested citation. Hwang et al. show that isp-1 worms have increased resistance to bacterial pathogens that is dependent on HIF-1/HIF1 and AAK- 2/AMPK. In future work, it will be interesting to examine whether HIF-1 and AAK-2 act in concert with, or independently of, ATFS-1 and the p38-mediated innate immune signaling pathway to mediate pathogen resistance and longevity in isp-1 worms. We will add these points to our discussion.

      1. The authors suggest that ROS activation of the p38 MAPK pathway is likely not the mechanism that explains the resistance of long-lived mitochondrial mutant animals due to their reduced food intake. However, is ROS production nonetheless involved? Does antioxidant treatment suppress the increased resistance during infection of isp-1 and/or nuo-6 mutant animals?

      To address this question, we will treat wild-type, isp-1 and nuo-6 worms with antioxidant and then measure resistance to bacterial pathogens using the P. aeruginosa strain PA14 slow kill assay. For the antioxidant treatment, we will use 10 mM Vitamin C as we have previously shown that this concentration is effective at reducing ROS in isp-1 worms to decrease isp-1 lifespan (Van Raamsdonk and Hekimi 2012, PNAS). Although antioxidant treatment can have pleiotropic effects, if this decreases survival of bacterial pathogen exposure, it will suggest that the elevated ROS production in isp-1 and

      nuo-6 worms may contribute to their enhanced bacterial pathogen resistance.

      1. (line 278-282): the authors should elaborate on how the p38 MAPK pathway plays a permissive role. It is intriguing that ATFS-1 and ATF-7 are both bZIP transcription factors that could theoretically heterodimerize and that they share common immune gene targets. The authors do indicate that the binding sites for ATFS-1 and ATF-7 are very different and are likely acting distinctly but some speculation would nonetheless strengthen this statement.

      While ATFS-1 and ATF-7 were shown to bind to the promoter regions of the same innate immunity genes, the apparent consensus binding sites are different suggesting that they bind to different regions of the promoter. One way in which the p38 MAPK pathway may be playing a permissive role is that ATF- 7 binding and relief from its repressor activity is required for any transcription of p38-mediated innate immunity target genes to occur. This is consistent with our data showing that disruption of nsy-1, sek-1, pmk-1 or atf-7 decreases the expression of innate immunity genes in wild-type worms. In contrast, it may be that the role of ATFS-1 is for enhanced expression of innate immunity genes such that when ATFS-1 is bound to the promoter region, or perhaps enhancer elements, the baseline expression of innate immunity genes that results from the binding of ATF-7 is increased. This idea is supported by our data showing that disruption of atfs-1 does not affect the expression of innate immunity genes in wild- type worms but prevents nuo-6 mutants from having increased expression. We will update our manuscript to include these points.

      1. The authors suggest that reduced food consumption of nuo-6 and isp-1 animals may suppress ROS- induced activation of the p38 innate immune pathway. It is intriguing that dietary restriction was previously shown to increase resistance to infection, presumably through p38-independent mechanisms (PMID 30905669). It would be interesting to measure host survival of nuo-6 and isp-1 mutant animals that are dietary-restricted to see if the enhanced survival rates conferred by mitochondrial stress and DR are additive or not.

      According to this suggestion, we will compare the bacterial pathogen resistance of wild-type, isp-1 and nuo-6 worms that have undergone dietary restriction to the same strains under ad libitum conditions. This will determine the extent to which their enhancement of pathogen resistance might be additive.

      1. Figure 2: It is intriguing that loss of p38 signaling appears to have different effects in nuo-6 versus isp-1 animals. Specifically, loss of p38 signaling in isp-1 mutants renders them more sensitive to infection than wild-type, whereas it generally suppresses survival rates back to wild-type levels in the nuo-6 mutant background. Even within the nuo-6 mutant group, loss of SEK-1 has more dramatic effects on nuo-6 mutant animals than does loss of NSY-1, PMK-1 or ATF-7(gf). This is despite the fact that the nsy-1, sek-1, and pmk-1 alleles that are used in this study are all reported to be null. Can the authors speculate on these differences?

      While the isp-1 and nuo-6 mutations both alter mitochondrial function, they affect different components of the electron transport chain. isp-1 mutations affect Complex III (Feng et al. 2001, Dev. Cell), while nuo-6 mutations affect Complex I (Yang and Hekimi 2010, Aging Cell). Although these mutants both have increased lifespan and a similar slowing of physiologic rates, it is not uncommon to observe differences between these mutants. For example, while treatment with the antioxidant NAC completely reverts nuo-6 lifespan to wild-type, it only partially reduces isp-1 lifespan (Yang and Hekimi 2010, PLoS Biology), suggesting that nuo-6 lifespan may be more dependent on ROS than isp-1. We have recently shown that deletion of atfs-1 reduces nuo-6 lifespan, but completely prevents isp-1 worms from developing to adulthood (Wu et al. 2018, BMC Biology), suggesting that isp-1 worms are more dependent on ATFS-1 than nuo-6 worms. Disruption of sek-1 has a greater impact on pathogen resistance than nsy-1 and pmk-1 because SEK-1 is absolutely required for innate immune signaling, while some partial redundancy exists for NSY-1 and PMK-1. We will add these points to our manuscript.

      1. One of the main conclusions from this study is that ATFS-1 likely binds directly to innate immune genes that are in common with ATF-7. Since this is such a pivotal finding, the authors should validate some candidate genes from the referenced ChIP seq datasets using ChIP qPCR. Also, are there predicted ATFS-1 binding sites (PMID 25773600) in these promoters?

      Our data shows that activation of ATFS-1 increases the expression of innate immunity genes without increasing activation of p38. The simplest explanation for this observation is that ATFS-1 can upregulate the same innate immunity genes as ATF-7. Accordingly, we hypothesized that ATFS-1 and ATF-7 can bind to the same promoter. Fortunately, two previous ChIP-Seq studies, from well-established laboratories who have extensive experience studying ATFS-1 and ATF-7, had already determined which genes are bound by these two transcription factors (Nargund et al. 2015, Molecular Cell; Fletcher et al. 2019, PLoS Genetics). Comparing the results of these two published studies confirmed our hypothesis by demonstrating that the same innate immunity genes are bound by both ATF-7 and ATFS-1 in vivo. In order to provide additional support for the conclusion that ATFS-1 and ATF-7 can bind to the same genes, we will examine the genetic sequence of innate immunity genes that were shown to be bound by both ATFS-1 and ATF-7 in the published ChIP-seq studies to identify predicted binding sites for ATFS-1 and ATF-7, while noting that the ATFS-1-associated sequence is an enriched motif and not an established binding site. If we are able to identify the predicted binding sites for these two transcription factors in the same gene, it will provide further support for the conclusion that these transcription factors can both bind to the same innate immunity genes.

      Reviewer #2

      1. The authors state that the p38 MAPK PMK-1 is not activated in the long-lived mitochondrial mutants. However, it might be better to state that there is "no enhanced activation" of PMK-1, since they clearly show in nuo-6 and isp-1 mutants the presence of phosphorylated PMK-1 (Fig. 4A), which would indicate an activated form of PMK-1 in these mutants.**

      According to this suggestion, we will change the text to indicate that there is no enhanced activation of PMK-1 in nuo-6 and isp-1 worms.

      1. Are the food-intake behaviors of all mutants in liquid culture (Fig. 4B-F) the same as their food- intake behaviors on solid agar media, the environment where pathogen resistance was measured?

      We previously compared assays measuring food intake on solid agar media versus the liquid culture approach used in the current study to determine which method is the most robust (Wu et al. 2019, Cell Metabolism). While both assays produced similar results, performing the food intake assay on solid agar plates was much more variable as it is challenging to scrape off all of the uneaten bacteria from solid plates in order to measure it. Since the approach of measuring food intake in liquid media produces more consistent and reliable results, we chose to use this assay for the current study. We will update our manuscript to include this justification.

      1. Does the p38 pathway single mutant nsy-1 or sek-1 live shorter than wild type on dead E. coli OP50 (Fig. S9) than they do on live OP50 (Fig. 3)? If so, what might that mean? These mutants are also living shorter than wild type on PA14 (Fig. 2), but live as long as wild type on OP50 (Fig. 3). What is in the live OP50 that allows these mutants to live like wild type?

      In a previous publication, we found that sek-1 mutants live shorter than wild-type worms, and nsy-1 live slightly shorter than wild-type worms in a lifespan assay performed in liquid medium with dead OP50 bacteria (Wu et al. 2019, Cell Metabolism). In the current study, we performed lifespan assays on solid NGM plates with live OP50 bacteria and observed a wild-type lifespan in sek-1 and nsy-1 worms. Since there are multiple experimental variables that are different between the previous and current study, most notably liquid versus solid media, the lifespan results cannot be directly compared. In the case of measuring survival of these strains on PA14, the simplest explanation is that they are dying sooner because their innate immune signaling pathway is disrupted, and so they are less able to mount an immune response against the pathogenic bacteria. We will update our manuscript to include these points.

      At the same time, wouldn't it be simpler to call the multiple antibiotic-treated OP50 as "dead bacteria", instead of "non-proliferating bacteria"? Some of the antibiotics used to treat OP50 are bactericidal and not bacteriostatic.

      We previously monitored the OD600 of the antibiotic-treated, cold-treated OP50 that we used in our experiment, and found that there is only a very small decrease in OD600 after 10 days (Moroz et al. 2014, Aging Cell). Since dead bacteria are rapidly broken down leading to a decrease in OD600, this result is consistent with the bacteria being alive but not proliferating. We will include this point in our manuscript.

      1. Since nuo-6 and isp-1 do not always behave exactly the same in their dependence on certain genes (e.g., Fig. 2C vs Fig 2D), what happens in isp-1; atfs-1 double mutants? Do these mutants behave in the same manner as nuo-6; atfs-1?

      This is an interesting question. Unfortunately, isp-1;atfs-1 mutants arrest during development (Wu et al. 2018, BMC Biology), which is why we only examined the effect of atfs-1 deletion in nuo-6 mutants. We will update the manuscript to note this point.

      Regarding nuo-6; atfs-1, why does the double mutant live shorter on PA14 than either single mutant (Fig. 6A)? Is this because atfs-1 is needed to activate the p38 MAPK-dependent and -independent pathways?

      It is possible that the nuo-6 mutation makes worms more sensitive to bacterial pathogens, perhaps due to decreased energy production, and that activation of ATFS-1 is required not only to enhance their resistance to pathogens but also to increase their resistance back to wild-type levels. In a previous study, we showed that loss of ATFS-1 slows down the rate of nuclear localization of DAF-16. Thus, loss of atfs-1 may also be decreasing resistance to bacterial pathogens by diminishing the general stress resistance imparted by the DAF-16-mediated stress response pathway. We will update the manuscript to include these points.

      In Fig. 7B, the atfs-1(gof) appears to have slightly more phosphorylated p38 compared to wild type, although it is not statistically significant?

      While there is a trend towards a very modest increase in phosphorylated p38 in the constitutively-active atfs-1 mutant compared to wild-type, quantification of four biological replicates indicated that the difference is not significant. This result is consistent with the fact that the levels of phosphorylated p38 are not significantly increased in nuo-6 or isp-1 mutants, both of which show activation of ATSF-1. We have provided raw images of all of these Western blots in our supplementals. In addition, we will repeat these Western blots to determine if this difference becomes significant with additional replicates.

      In Fig. 6B, the atfs-1 loss-of-function single mutant also increases the expression of Y9C9A.8, but suppresses it in a nuo-6 mutant background? What might that mean?

      It is possible that in wild-type animals disruption of atfs-1 causes a compensatory upregulation of specific stress response genes. We have previously shown that deletion of atfs-1 results in upregulation of chaperone genes involved in the cytoplasmic unfolded protein response (hsp-16.11, hsp-16.2; Wu et al. 2018; BMC Biology). Perhaps Y9C9A.8 is acting in a similar way. In nuo-6, the upregulation of Y9C9A.8 is driven by activation of ATFS-1, and thus is prevented by atfs-1 deletion. We will add these points to the manuscript.

      Reviewer #3

      1. Some studies propose that OP50 offers some toxicity to worms which is not observed in other bacterial strains like HT115. The authors should test the role of the p38-innate immune signaling pathway in nuo-6 and isp-1 lifespan using other non-pathogenic E. coli strains.**

      To determine if the effect of disrupting the p38-mediated innate immune signaling pathway on the lifespan of isp-1 and nuo-6 mutants was simply the result of losing protection against OP50 bacteria, we examined the effect of nsy-1, sek-1 and atf-7(gof) mutations on isp-1 and nuo-6 lifespan using non- proliferating bacteria. We found that even when no proliferating bacteria are present, disruption of the p38-mediated innate immune signaling pathway markedly decreases isp-1 and nuo-6 lifespan. This suggests that the p38-mediated innate immune signaling pathway is required for their long lifespan independently of its ability to protect against bacterial infection. Similarly, we have previously shown that lifespan extension resulting from dietary restriction is dependent on the p38-mediated innate immune signaling pathway even when non-proliferating bacteria are used (Wu et al. 2019, Cell Metabolism). We will clarify this important point in the manuscript.

      1. The authors should measure food intake in worms exposed to pathogenic bacteria, given that reduced bacterial intake may be related to reduced mortality.

      Unfortunately, it is not feasible to perform the food intake assay using the pathogenic bacteria because the bacteria cause death thereby complicating the calculation of food consumed per worm (which requires at least 3 days to assess). As an alternative to measuring food intake, we will attempt to measure intestinal accumulation of P. aeruginosa, which is a balance between food intake and other factors. To do this we will use a P. aeruginosa strain that expresses GFP and quantify the amount of intestinal fluorescence in wild-type, isp-1 and nuo-6 worms that have been grown on the GFP-labelled P. aeruginosa.

      1. The authors should check if ROS is required for the activation of the p38-mediated innate immune signaling pathway and reduction in food intake.

      To determine if the elevated ROS that is present in isp-1 and nuo-6 worms affects activation of the p38- mediated innate immune signaling pathway, we will treat wild-type, isp-1 and nuo-6 worms with Vitamin C and measure the ratio of phosphorylated p38 to total p38 by Western blotting. Similarly, to examine the effect of ROS on food intake, we will treat wild-type, isp-1 and nuo-6 worms with Vitamin C and then quantify its effect on food intake. For these experiments, we will use 10 mM Vitamin C as we have previously shown that this concentration is effective at reducing ROS in isp-1 worms to decrease isp-1 lifespan (Van Raamsdonk and Hekimi 2012, PNAS).

      1. Since ATFS-1 and the p38 pathway control food intake, how related to dietary restriction the phenotypes the authors are studying are?

      While the lifespan extension that results from mild impairment of mitochondrial function and the lifespan extension resulting from dietary restriction are both dependent on the p38-mediated innate immune signaling pathway, these interventions modulate innate immunity gene expression in opposite directions. We previously reported that dietary restriction primarily downregulates innate immunity genes (Wu et al. 2019 Cell Metabolism). Here, we show that mutations in isp-1 or nuo-6 primarily result in upregulation of innate immunity genes. To more globally examine gene expression changes between dietary restriction and mild impairment of mitochondrial function, we compared differentially expressed genes. We found that there was very little overlap of either upregulated or downregulated genes between dietary restriction and isp-1/nuo-6 mutants. We will add a supplementary figure to demonstrate this, and add these points to our manuscript.

      1. Somewhat related to the previous points, I am not so sure whether the changes in food intake are cause or consequence of the alterations in the innate immunity-related genes. Reduced food intake is depicted in Fig. 8 as the cause of the activation of the p38 pathway, but there is not enough evidence to unequivocally prove that. In fact, food intake might be controlled by the p38 or ATFS-1 pathway or by a common regulator such as ROS.

      We apologize that we didn’t make this clearer. In our previous work, we showed that dietary restriction results in decreased activation of the p38 pathway (Wu et al. 2019, Cell Metabolism). Here, we show that activation of ATFS-1 results in decreased food intake. Based on our previous study, this decrease in food intake should similarly decrease p38 pathway activation. In Figure 8, we have depicted ATFS-1 inhibiting food intake, and food intake activating the p38-mediated innate immune signaling pathway. Combined, our model suggests that activation of ATFS-1 should act to decrease p38-mediated innate immune signaling. We will clarify this in the figure legend.

      1. I am not so convinced of the role of DAF-16. In fact, in Fig. 5A daf-16 mutation reduces pathogen resistance and that could represent a toxic effect of the mutation. Furthermore, the results in Fig. 4D do not exclude the possibility that daf-16 and isp-1 act in parallel.

      We agree that the role of DAF-16 could be non-specific. While we show that disruption of daf-16 leads to decreased bacterial pathogen survival in isp-1 worms, it also decreases bacterial pathogen survival in wild-type worms. Since DAF-16 is known to be required for general resistance to stress, the decreased survival when daf-16 is disrupted could be due to a general toxic effect of reducing general stress resistance. This conclusion is consistent with our observation that DAF-16 is not involved in the upregulation of innate immunity genes in isp-1 worms. We will emphasize these points in our manuscript.

      1. Loss of innate immunity related genes may result in toxicity and sensitize worms to pathogenic bacteria. This is further supported by an even lower resistance to pathogens in the double mutants mainly in Fig. 2D.

      We agree. Our data confirms that disruption of the p38-mediated innate immune signaling pathway makes worms more susceptible to bacterial pathogens. We will emphasize this point.

      1. The blots are saturated, particularly in Fig. 4A, and this can be masking the differences in p38 phosphorylation. In fact, the fact that p38 phosphorylation is not changed is contradictory to the other results. How is p38 regulated by mitochondrial mutations then? I am concerned that p38 is actually not altered and the changes in gene expression are exclusively due to ATFS-1. The interaction with the p38 pathway demonstrated genetically could be due to the toxicity elicited by the loss of function mutations in this pathway.

      To address this concern, we will repeat the Western blotting experiment to compare the ratio of phosphorylated p38 to total p38 between wild-type, isp-1 and nuo-6 worms. We will take multiple exposures to ensure that the blots are not over-saturated. Having already completed four replicates, we believe that there is not a major change in p38 activation. Our data suggests that the p38-mediated innate immunity pathway is playing a permissive role such that it is required for baseline expression of innate immunity genes, but that activation of ATFS-1 is driving the enhanced expression of innate immunity genes that we observe in the long-lived mitochondrial mutants and constitutively active atfs-1 mutants. We will update our manuscript to clarify this.

      Minor concerns

      1. Lines 167 and 174: What are these p values referred to?**

      The p-values indicate the significance of the overlap between the two gene sets. Given the size of the two gene sets, this is the probability that the observed number of overlapping genes would result by picking genes at random. We will clarify this in the manuscript.

      1. Line 258: I partially agree with the conclusions, since the functions may not necessarily be associated with innate immune signaling but rather other functions of p38.

      Since isp-1 and nuo-6 worms have extended longevity even when grown on non-proliferating bacteria this indicates that their long life is not dependent on their enhanced resistance to bacterial pathogens. Similarly, since disruption of genes in the p38-mediated innate immune signaling pathway decrease isp- 1 and nuo-6 lifespan even when the worms are grown on non-proliferating bacteria, this suggests that this pathway enhances longevity independently of its ability to increase innate immunity.

      1. Why in figures 4D and E different mutants were used?

      We only used isp-1 mutants to examine the effect of daf-16 because we were unable to generate nuo- 6;daf-16 mutants due to close proximity of the two genes on the same chromosome. We only used nuo- 6 mutants to examine the effect of atfs-1 because isp-1;atfs-1 worms arrest during development. We will include this explanation in our manuscript.

      1. Line 498: revise writing.

      We will rewrite this sentence to improve clarity.

      1. Show blots in Fig. 7B.

      We will provide an image of a representative Western blot in Figure 7, and will provide the raw images for all of Western blots in our supplementals.

      1. It would be interesting to know where the activation of the immune-related genes by the mitochondrial mutations is happening, whether this is a cell autonomous or cell non-autonomous mechanism.

      While it would be interesting to explore whether specific tissues are important in sensing mitochondrial impairment in order to upregulate genes involved in innate immunity, it is beyond the scope of this manuscript. Previous work has shown that knocking down the expression of the cytochrome c oxidase gene cco-1 in neurons can activate the ATFS-1 target gene hsp-6 in the intestine (Durieux et al., 2011). Based on this, one could hypothesize that a similar cell non-autonomous mechanism might be involved. We will note this possible future direction in our discussion.

    1. Die weitestgehende Öffnung liegt bei dieser Variante vor, wenn sowohl Autor*innen- wie auch Gutachter*innen- und Gutachtentransparenz besteht. Offene Review-Verfahren schließen ferner die Option einer nachträglichen Veröffentlichung der Gutachten als Begleittexte einer Publikation mit ein

      Volle Transparenz wäre m.E. erst dann gegeben, wenn auch abgelehente Einreichungen mitsamt der der Gutachten, die zur Ablehnung geführt haben ins Netz gestellt werden. Mir scheint, um Meinungs- oder Zitationskartelle zu verhindern (oder zumindest offensichtlich werden zu lassen), wäre das sogar wichtiger als die Namen der Gutachter anzugeben.

  21. Mar 2021
    1. The elimination of what is arguably the biggest monoculture in the history of software development would mean that we, the community, could finally take charge of both languages and run-times, and start to iterate and grow these independently of browser/server platforms, vendors, and organizations, all pulling in different directions, struggling for control of standards, and (perhaps most importantly) freeing the entire community of developers from the group pressure of One Language To Rule Them All.
  22. Feb 2021
    1. The Rights Retention Strategy provides a challenge to the vital income that is necessary to fund the resources, time, and effort to provide not only the many checks, corrections, and editorial inputs required but also the management and support of a rigorous peer review process

      This is an untested statement and does not take into account the perspectives of those contributing to the publishers' revenue. The Rights Retention Strategy (RRS) relies on the author's accepted manuscript (AAM) and for an AAM to exist and to have the added value from peer-review a Version of Record (VoR) must exist. Libraries recognise this fundamental principle and continue to subscribe to individual journals of merit and support lucrative deals with publishers. From some (not all) librarians' and possibly funders' perspectives these statements could undermine any mutual respect.

  23. Jan 2021
    1. ReconfigBehSci [@SciBeh] (2020-01-27) new post on Scibeh's meta-science reddit describing the new rubric for peer review of preprints aimed at broadening the pool of potential 'reviewers' so that students could provide evaluations as well! https://reddit.com/r/BehSciMeta/comments/l64y1l/reviewing_peer_review_does_the_process_need_to/ please take a look and provide feedback! Twitter. Retrieved from: https://twitter.com/SciBeh/status/1354456393877749763

    1. Mambrini. A. Baronchelli. A. Starnini. M. Marinazzo. D. De Domenico, M. (2020) .PRINCIPIA: a Decentralized Peer-Review Ecosystem. Retrieved from: chrome-extension://bjfhmglciegochdpefhhlphglcehbmek/pdfjs/web/viewer.html?file=https%3A%2F%2Farxiv.org%2Fpdf%2F2008.09011.pdf

  24. Nov 2020
    1. We do not utilize a formulaic or standard, formalized benchmarking level or element in setting our executive compensation relative to that of other companies. 
  25. Oct 2020
    1. An in depth read that has plenty of sources and data to back its findings that peer evaluation has many positive impacts on learning when done the correct way.

      8/10

    1. So that’s already a huge advantage over other platforms due the basic design. And in my opinion it’s got advantages over the other extreme, too, a pure peer-to-peer design, where everyone would have to fend for themselves, without the pooled resources.

      Definitely something the IndieWeb may have to solve for.

  26. Sep 2020
  27. Aug 2020
  28. Jul 2020
    1. Authors should annotate code before the review occurs because annotations guide the reviewer through the changes

      Guide the reviewer during the review process

    2. It´s also useful to watch internal process metrics, including:

      Inspection rate Defect rate Defect density

    3. Before implementing a process, your team should decide how you will measure the effectiveness of peer review and name a few tangible goals.

      Set few tangible goals. Fix more bugs is not a good example.

    4. Code reviews in reasonable quantity, at a slower pace for a limited amount of time results in the most effective code review.

      Only less than 500 LOC per hour

    5. The brain can only effectively process so much information at a time; beyond 400 LOC, the ability to find defects diminishes.

      <400 LOC

  29. Jun 2020
  30. May 2020
  31. Apr 2020
    1. students responded to messages more actively and engaged in more in-depth discussions when discussions were moderated by a peer.

      This could be a good argument to push Hypothes.is to introduce some sort of moderation, in combination with the finding that annotation threads would be rare, and not very deep (Wolfe & Neuwirth, 2001)

    1. There are good preprints and bad preprints, just like there are with journal articles. Overall, do not be afraid to be scooped or plagiarized! Preprints also actually protect against scooping [21,22]. Preprints establish the priority of discovery as a formally published item. Therefore, a preprint acts as proof of provenance for research ideas, data, code, models, and results—all outputs and discoveries.
      • Salah satu alasan untuk tidak mengunggah preprint adalah takut idenya dicuri,

      • Ini adalah faktor budaya yang lain. Ketakutan yang tidak beralasan. Justru dengan mengunggah preprint, peneliti dapat mengklaim ide lebih awal.

      • Preprint ada yang bagus dan ada yang buruk, peninjauan akan ada di tangan pembaca. Ini adalah hambatan budaya berikutnya, ketika mayoritas pembaca ingin melimpahkan tanggungjawab untuk memverifikasi, memeriksa, dan menjamin kualitas suatu makalah kepada para peninjau.

      • Pengalihan tanggungjawab ini sulit dilakukan ketika dokumen PR sendiri tertutup, dan tidak lepas dari bias.

      • Selain itu, dosen akan menyalahi prinsip yang disebarluaskan kepada para mahasiswa, untuk membaca secara kritis.

    2. One of the reasons is the delay in the peer-review process and the subsequent publication
      • Salah satu kritik terbesar terhadap preprint adalah ketidakadaan peninjauan sejawat (Peer Review/PR).

      • Proses PR ini memang menjadi proses sentral dalam publikasi. Di luar manfaatnya, PR juga dapat merugikan, karena memberikan hambatan waktu.

      • Yang unik ada makalah yang memperlihatkan hasil bahwa banyak makalah versi terpublikasi memiliki isi dan tampilan tidak berbeda dengan versi preprintnya.

    1. Someresearch has shown that preprints tend to be of similar quality to their final published versions in journals
      • Salah satu kritik terbesar terhadap preprint adalah ketidakadaan peninjauan sejawat.

      • Proses PR ini memang menjadi proses sentral dalam publikasi. Di luar manfaatnya, PR juga dapat merugikan, karena memberikan hambatan waktu.

      • Yang unik ada makalah yang memperlihatkan hasil bahwa banyak makalah versi terpublikasi memiliki isi dan tampilan tidak berbeda dengan versi preprintnya.

  32. Mar 2020
    1. Chapman describes the inductive approach as a gathering of facts and taking an inventory of visual qualities in a work. After thoroughly taking in the visual elements, the relationships among these visual elements must be compiled into a summary ofthe impressions that “captures the essence of what we have seen” (p. 80). Chapman also warns against premature judgment and emotional reactions. Only after one has described each part of the work, analyzed the relationship among parts, interpreted these relationships, and summarized the recurrent ideas, can one move onto the judgment by citing the information gathered before the judgment stage.

      Chapman approach

    2. develop constructive approachesto critique (formative, peer-to-peer, self-reflective, summative) that are supportive in intent and that offer alternative points of view.” (AI.18). By Art II: Intermediate, students will “use constructive critical approaches to critique (formative, peer-to-peer, self-reflective, summative).
    3. 1) interpret, reflect on, and evaluate the characteristics, purposes, and merits of personal work and the work of others and 2) identify, analyze, and apply criteria for making visual aesthetic judgments of personal work and the work of others.”

      these are good to include in our learning objectives?

    4. Peer critique in the secondary art classroom:Strategies for best practices

      Peer critique secondary (college?) best practices

    1. I give some display guidelines for them to follow so the presentation looks good and work can be seen easily by all participants. Students who are not finished might use the time to keep working to finish. It is sometimes found that students without work on display are not as interested and they are more apt to offer negative comments. 

      provide display guidelines on how to present the art work.

    2. Successful Art Class Critique

      bookmark

  33. Feb 2020
    1. Transparent Review in Preprints (TRiP) — that enables journals and peer review services to post peer reviews of submitted manuscripts on CSHL’s preprint server bioRxiv.

      Incredible use of annotation technology in peer review over preprints! Watch this space! I'm lucky that I get to use annotation in my work at the Knowledge Futures Group.

  34. Jan 2020
    1. Interested authors can select In Review when they submit their manuscript through Editorial Manager. Participating will enable them to track the progress of their manuscript through peer review with immediate access to review reports, share their work to engage a wider community through open annotation using Hypothesis, follow a transparent editorial checklist, and gain early collaboration and citation opportunities.

      Annotation in peer review, whether on preprints or through a more traditional manuscript submission system, offers the option for reviewers, editors, and authors to give and received feedback in context. And I'm super excited about this new project.

  35. Dec 2019
    1. Furthermore, someone who has just learned something is often better at helping someone else learn it, than is someone who learned it long ago. In addition to older students teaching slightly younger ones, peers can learn from each other in collaborative projects, and they can also serve as peer tutors.

      This definitely supports my idea of peer mentors

    1. Supplementary data

      Of special interest is that a reviewer openly discussed in blog his general thoughts about the state of the art in the field based on what he had been looking at in the paper. This blog came out just after he completed his 1st round review, and before an editorial decision was made.

      http://ivory.idyll.org/blog/thoughts-on-assemblathon-2.html

      This spawned additional blogs that broadened the discussion among the community-- again looking toward the future.<br> See: https://www.homolog.us/blogs/genome/2013/02/23/titus-browns-thoughts-on-the-assemblathon-2-paper/

      And

      https://flxlexblog.wordpress.com/2013/02/26/on-assembly-uncertainty-inspired-by-the-assemblathon2-debate/

      Further the authors, now in the process of revising their manuscript, joined in on twitter, reaching out to the community at large for suggestions on revisions, and additional thoughts. Their paper had been posted in arxiv- allowing for this type of commenting and author/reader interaction See: https://arxiv.org/abs/1301.5406

      The Assemblathon.org site collected and presented all the information on the discussion surrounding this article. https://assemblathon.org/page/2

      A blog by the editors followed all this describing this ultra-open peer review, highlighting how these forms of discussions during the peer review process ended up being a very forward-looking discussion about the state of based on what the reviewers were seeing in this paper, and the directions the community should now focus on. This broader open discussion and its very positive nature could only happen in an open, transparent, review process. See: https://blogs.biomedcentral.com/bmcblog/2013/07/23/ultra-open-peer-review/

  36. Oct 2019
    1. A Million Brains in the Cloud

      Arno Klein and Satrajit S. Gosh published this research idea in 2016 and opened it to review. In fact, you could review their abstract directly in RIO, but for the MOOC activity "open peer review" we want you to read and annotate their proposal using this Hypothes.is layer. You can add annotations by simply highlighting a section that you want to comment on or add a page note and say in a few sentences what you think of their ideas. You can also reply to comments that your peers have already made. Please sign up to Hypothes.is and join the conversation!

  37. Sep 2019
    1. Transparent Review in Preprints will allow journals and peer review services to show peer reviews next to the version of the manuscript that was submitted and reviewed.

      A subtle but important point here is that when the manuscript is a preprint then there are two public-facing documents that are being tied together-- the "published" article and the preprint. The review-as-annotation becomes the cross-member in that document association.

    1. I am writing this review for the Drummond and Sauer comment on Mathur and VanderWeele (2019). To note, I am familiar with the original meta-analyses considered (one of which I wrote), the Mathur and VanderWeele (henceforth MV2019) article, and I’ve read both Drummond and Sauer’s comment on MV2019 and Mathur’s review of Drummond and Sauer’s comment on MV2019 (hopefully that wasn’t confusing). On balance, I think Drummond and Sauer’s (henceforth DSComment) comment under review here is a very important contribution to this debate. I tended to find DSComment to be convincing and was comparatively less convinced by Mathur’s review or, indeed, MV2019. I hope my thoughts below are constructive.

      It’s worth noting that MV2019 suffered from several primary weaknesses. Namely:

      1. On one hand, it didn’t really tell us anything we didn’t already know, namely that near-zero effect sizes are common for meta-analyses in violent video game research.
      2. MV2019, aside from one brief statement as DSComment notes, neglected the well-known methodological issues that tend to spuriously increase effect sizes (unstandardized aggression measures, self-ratings of violent game content, identified QRPs in some studies such as the Singapore dataset, etc.) This resulted in a misuse of meta-analytic procedures.
      3. MV2019 naïvely interprets (as does Mathur’s review of DSComment) near-zero effect sizes as meaningful, despite numerous reasons not to do so given concerns of false positives.
      4. MV2019, for an ostensible compilation of meta-analyses, curiously neglect other meta-analyses, such as those by John Sherry or Furuyama-Kanamori & Doi (2016).

      At this juncture, publication bias, particularly for experimental studies, has been demonstrated pretty clearly (e.g. Hilgard et al., 2017). I have two comments here. MV2019 offered a novel and not well-tested alternative approach (highlighted again by Mathur’s review) for bias, however, I did not find the arguments convincing as this approach appears extrapolative and produces results that simply aren’t true. For instance, the argument that 100% of effect sizes in Anderson 2010 are above 0, is quickly falsified merely by looking at the reported effect sizes in the studies included, at least some of which are below .00. Therefore, this would appear to clearly indicate some error in the procedure of MV2019.

      Further, we don't need statistics to speculate about publication bias in Anderson et al. (2010) as there are actual specific examples of published null studies missed by Anderson et al. (see Ferguson & Kilburn, 2010). Further, the publication of null studies in the years immediately following (e.g. von Salisch et al., 2011) indicate that Anderson's search for unpublished studies was clearly biased (indeed, I had unpublished data at that time but was not asked by Anderson and colleagues for it). So there's no need at all for speculation given we have actual examples of missed studies and a fair number of them.

      It might help to highlight also that traditional publication bias techniques probably are only effective with small sample experimental studies. For large sample correlational/longitudinal studies, effect sizes tend to be a bit more homogeneous, hovering closely to zero. In such studies the accumulation of p-values near .05 is unlikely given the power of small studies. Relatively simple QRPs can make p-values jump rapidly from non-significance to something well below.05. Thus, traditional publication bias procedures may return null results for this pool of studies, despite QRPs, and thus, publication bias having taken place.

      It might also help to note that meta-analyses with weak effects are very fragile to unreported null studies, which probably exist in greater numbers (particularly for large n studies) that would be indicated by publication bias techniques.

      I agree with Mathur’s comment about experiments not always offering the best evidence, given lack of generalizability to real-world aggression (indeed, that’s been a long-standing concern). However, it might help DSComment to note that, by this point, probably the pool of evidence least likely to find effects are longitudinal studies. I’ve got two preregistered longitudinal analyses of existing datasets myself (here I want to make clear that citing my work is by no means necessary for my positive evaluation of any revisions on DSComment), and there are other fine studies (such as Lobel et al., 2017, Breuer et al., 2015, Kuhn et al., 2018; von Salisch et al., 2011, etc.) The authors may also want to note Przybylski and Weinstein (2019) which offer an excellent example of a preregistered correlational study.

      Indeed, in a larger sense, as far as evidence goes, DSComment could highlight recent preregistered evidence from multiple sources (McCarthy et al., 2016; Hilgard et al., 2019, Przybylski & Weinstein, 2019, Ferguson & Wang, 2019, etc.) This would seem to be the most crucial evidence and, aside from one excellent correlational study (Ivory et al.) all of the preregistered results have been null. Even if we think the tiny effect sizes in existing metas provide evidence in support of hypotheses (and we shouldn’t), these preregistered studies suggest we shouldn’t trust even those tiny effects to be “true.”

      The weakest aspect of MV2019 was the decision to interpret near-zero effects as meaningful. Mathur, argues that tiny effects can be important once spread over a population. However, this is merely speculation, and there’s no data to support it. It’s kind of a truthy thing scholars tend to say defensively when confronted by the possibility that effect sizes don’t support their hypotheses. By making this argument, Mathur invites an examination of population data where convincing evidence (Markey, Markey & French, 2015; Cunningham et al., 2016; Beerthuizen, Weijters & van der Laan, 2017) shows that violent game consumption is associated with reduced violence in society. Granted, some may express caution about looking at societal-level data, but here is where scholars can’t have it both ways: One can’t make claims about societal-level effects, and then not want to look at the societal data. Such arguments make unfalsifiable claims and are unscientific in nature.

      The other issue is that this line of argument makes effect sizes irrelevant. If we’re going to interpret effect sizes no matter how near to zero as hypothesis supportive, so long as they are “statistically significant” (which, given the power of meta-analyses, they almost always are), then we needn’t bother reporting effect sizes at all. We’re still basically slaves to NHST, just using effect sizes as a kind of fig leaf for the naked bias of how we interpret weak results.

      Also, that’s just not how effect sizes work. They can’t be sprinkled like pixie dust over a population to make them meaningful.

      As DSComment points out, effect sizes that are this small have high potential for Type 1 error. Funder and Ozer (2019) recent contributed to this discussion in a way I think was less than helpful (to be very clear I respect Funder and Ozer greatly, but disagree with many of their comments on this specific issue). Yet, as they note, interpretation of tiny effects is based on such effects being “reliable”, a condition clearly not in evidence for violent game research given the now extensive literature on the systematic methodological flaws in that literature.

      In her comment Dr. Mathur dismisses the comparison with ESP research, but I disagree with (or dismiss?) this dismissal. The fact that effect sizes in meta-analyses for violent game research are identical to those for “magic” is exactly why we should be wary of interpreting such effect sizes as hypothesis supportive. Saying violent game effects are more plausible is irrelevant (and presumably the ESP people would disagree). However, the authors of DSComment might strengthen their argument by noting that some articles have begun examining nonsense outcomes within datasets. For example, in Ferguson and Wang (2019) we show that the (weak and in that case non-significant) effects for violent game playing are no different in predicting aggression than nonsense variables (indeed, the strongest effect was for the age at which one had moved to a new city). Orben and Przybylski (2019) do something similar and very effective with screen time. Point being, we have an expanding literature to suggest that the interpretation of such weak effects is likely to lead us to numerous false positive errors.

      The authors of DSComment might also note that MV2019 commit a fundamental error of meta-analysis, namely assuming that the “average effect size wins!” When effect sizes are heterogeneous (as Mathur appears to acknowledge unless I misunderstood) the pooled average effect size is not a meaningful estimator of the population effect size. That’s particularly true given GIGO (garbage in, garbage out). Where QRPs have been clearly demonstrated for some studies in this realm (see Przybylski & Weinstein, 2019 for some specific examples of documentation involving the Singapore dataset), the pooled average effect size, however it is calculated, is almost certainly a spuriously high estimate of true effects.

      DSComment could note that other issues such as citation bias are known to be associated with spuriously high effect sizes (Ferguson, 2015), another indication that researcher behaviors are likely pulling effect sizes above the actual population effect size.

      Overall, I don’t think MV2019 were very familiar with this field and, appearing unaware of the serious methodological errors endemic in much of the literature which pull effect sizes spuriously high. In the end, they really didn’t say anything we didn’t already know (the effect sizes across metas tend to be near zero), and their interpretation of these near-zero effect sizes was incorrect.

      With that in mind, I do think DSComment is an important part of this debate and is well worth publishing. I hope my comments here are constructive.

      Signed, Chris Ferguson

    2. [This was a peer review for the journal "Meta-Psychology", and I am posting it via hypothes.is at the journal's suggestion.]

      I thank the authors for their response to our article. For full disclosure, I previously reviewed an earlier version of this manuscript. The present version of the manuscript shows improvement, but does not yet address several of my substantial concerns, each of which I believe should be thoroughly addressed if a revision is invited. My concerns are as follows:

      1.) The publication bias corrections still rely on incorrect statistical reasoning, and using more appropriate methods yields quite different conclusions.

      Regarding publication bias, the first analysis of the number of expected versus observed p-values between 0.01 and 0.05 that is presented on page 3 (i.e., “Thirty nine…should be approximately 4%”) cannot be interpreted as a test of publication bias, as described in my previous review. The p-values would only be uniformly distributed if the null were true for every study in the meta-analysis. If the null does not hold for every study in the meta-analysis, then we would of course expect more than 4% of the p-values to fall in [0.01, 0.05], even in the absence of any publication bias. I appreciate that the authors have attempted to address this by additionally assessing the excess of marginal p-values under two non-null distributions. However, these analyses are still not statistically valid in this context ; they assume that every study in the meta-analysis has exactly the same effect size (i.e., that there is no heterogeneity), which is clearly not the case in the present meta-analyses. Effect heterogeneity can substantially affect the distribution and skewness of p-values in a meta-analysis (see Valen & Yuan, 2007). To clarify the second footnote on page 3, I did not suggest this particular analysis in my previous review, but rather described why the analysis assuming uniformly distributed p-values does not serve as a test of publication bias.

      I would instead suggest conducting publication bias corrections using methods that accommodate heterogeneity and allow for a realistic distribution of effects across studies. We did so in the Supplement of our PPS piece (https://journals.sagepub.com/doi/suppl/10.1177/1745691619850104) using a maximum-likelihood selection model that accommodates normally-distributed, heterogeneous true effects and essentially models a discontinuous “jump” in the probability of publication at the alpha threshold of 0.05. These analyses did somewhat attenuate the meta-analyses’ pooled point estimates, but suggested similar conclusions to those presented in our main text. For example, the Anderson (2010) meta-analysis had a corrected point estimate among all studies of 0.14 [95% CI: 0.11, 0.16]. The discrepancy between our findings and Drummond & Sauer’s arises partly because the latter analysis focuses only on pooled point estimates arising from bias correction, not on the heterogeneous effect distribution, which is the very approach that we described as having led to the apparent “conflict” between the meta-analyses in the first place. Indeed, as we described in the Supplement, publication bias correction for the Anderson meta-analyses still yields an estimated 100%, 76%, and 10% of effect sizes above 0, 0.10, and 0.20 respectively. Again, this is because there is substantial heterogeneity. If a revision is invited, I would (still) want the present authors to carefully consider the issue of heterogeneity and its impact on scientific conclusions.

      2.) Experimental studies do not always yield higher-quality evidence than observational studies.

      Additionally, the authors focus only the subset of experimental studies in Hilgard’s analysis. Although I agree that “experimental studies are the best way to completely eliminate uncontrolled confounds”, it is not at all clear that experimental lab studies provide the overall strongest evidence regarding violent video games and aggression. Typical randomized studies in the video game literature consist, for example, of exposing subjects to violent video games for 30 minutes, then immediately having them complete a lab outcome measure operationalizing aggression as the amount of hot sauce a subject chooses to place on another subject’s food. It is unclear to what extent one-time exposures to video games and lab measures of “aggression” have predictive validity for real-world effects of naturalistic exposure to video games. In contrast, a well-conducted case-control study with appropriate confounding control and assessing violent video game exposure in subjects with demonstrated violent behavior versus those without might in fact provide stronger evidence for societally relevant causal effects (e.g., Rothman et al., 2008).

      3.) Effect sizes are inherently contextual.

      Regarding the interpretation of small effect sizes, we did indeed state several times in our paper that the effect sizes are “almost always quite small”. However, to universally dismiss effect sizes of less than d = 0.10 as less than “the smallest effect size of practical importance” is too hasty. Exposures, such as violent video games, that have very broad outreach can have substantial effects at the population level when aggregated across many individuals (VanderWeele et al., 2019). The authors are correct that small effect sizes are in general less robust to potential methodological biases than larger effect sizes, but to reiterate the actual claim we made in our manuscript: “Our claim is not that our re-analyses resolve these methodological problems but rather that widespread perceptions of conflict among the results of these meta-analyses—even when taken at face value without reconciling their substantial methodological differences—may in part be an artifact of statistical reporting practices in meta-analyses.” Additionally, the comparison to effect sizes for psychic phenomena does not strike as particularly damning for the violent video game literature. The prior plausibility that psychic phenomena exist is extremely low, as the authors themselves describe, and it is surely much lower than the prior plausibility that video games might increase aggressive behavior. Extraordinary claims require extraordinary evidence, so any given effect size for psychic phenomena is much less credible than for video games.

      Signed, Maya B. Mathur Department of Epidemiology Harvard University

      References

      Johnson, Valen, and Ying Yuan. "Comments on ‘An exploratory test for an excess of significant findings’ by JPA loannidis and TA Trikalinos." Clinical Trials 4.3 (2007): 254.

      Rothman, K. J., Greenland, S., & Lash, T. L. (2008). Modern epidemiology (Vol. 3). Philadelphia: Wolters Kluwer Health/Lippincott Williams & Wilkins.

      VanderWeele, T. J., Mathur, M. B., & Chen, Y. (2019). Media portrayals and public health implications for suicide and other behaviors. JAMA Psychiatry.

  38. Aug 2019
  39. Jul 2019
  40. Apr 2019
  41. Feb 2019
    1. Interactions of tomato and Botrytis genetic diversity: Parsing the contributions of host differentiation, domestication and pathogen variation

      This article has a Peer Review Report

  42. Jan 2019
    1. Web annotation, for example, is catching on as a new mode of collaboration, peer review, and other research functions.

      And the combination of community feedback on preprints with traditional and post-publication peer review through collaborative annotation is catching on with a variety of publishers. See InReview by BMC and ResearchSquare. Also COS preprint servers such as SocArXiv and Psyarxiv.