Could also refer to an environment in which code is run. For example you could have dev, staging, qa, and production environments. I'll often say, something to the effect of

Is it working on dev?

It's important to note that this is a tool developed by a company. It is not a be-all / end-all definition that is universal.

Can also be referred to as the inspector

The CD can also refer to continuous deployment. It really depends on the context that it's used.

This definition better defines the CI part, and it not related to the CD part

Change text to

The backend typically includes software, one or many databases, physical servers and infrastructure

Probably not needed. It's Github specific and not typically relevant to a non-developer

It is better to link notes than to add to notes. This improves reuse and is easier.

廣泛被稱為 Creative Coding 的電腦藝術風潮，並不是一場預謀規劃好的藝術運動，但 1990 年代由 John Maeda 在 MIT 的 Aesthetics and Computation 小組中發展出的編碼架構「DBN」，意外帶出了後續的發展。

Ben Fry 與 Casey Reas （兩人都是 John Maeda 的學生），正式根據 DBN 的架構發展出 Processing，目前最多藝術家與設計師使用的程式語言，讓大家「用代碼畫素描」。

You could add this line to the code:

m = ceiling( 0.75*( length(FDD)^(1/3) ) )

Funny/rude prank in Windows 3.1

思考总是比实施便宜，在纸上写写画画思考一下项目的结构，局限和重点，这样实施的时候更轻松。

• for : concept - universal anti-app

• for : IndyWeb

instead of bring along (owned data) to new applications*

all IndyWeb Anti-Apps Alter-Apps are instead of decoupling data and apps in the sense that the datamodel is not baked into the app code but have independent existence and the app is built around such a model in a uniform way

applications that anyone can spin up by forking

This error is addressed and corrected in 16.4.2 - maybe worth noting for readers who want to check out solution.

But in the R code above you set it to end in 2002, not in 2012.

the code name for the American-led effort to develop a functional atomic weapon during World War II

This is me trying out the annotation feature using Hypothesis. I can add LaTex in annotation: \(i (0+) = \mathcal{E}/R\)

I can also add images:

Westworld is really on to something here.

Author Response

Reviewer #2: Public review:

This manuscript reports results from a sensitivity analysis done to assess jointly the contribution of various factors to the spread of P. falciparum malaria parasites that are resistant to antimalarial drugs. It also explores how probable parasite genotypes are to establish as a function of their consequent rate of spread.

This manuscript's main contribution is its joint consideration of several factors not considered jointly before. The authors achieve their goal of doing a large joint analysis using computer simulations generated under a model framework that includes a model of malaria transmission and a model called an emulator. The malaria model has new features capturing different drug mechanisms and the capacity to track different degrees and types of drug resistance. It is very sophisticated but computationally expensive. The emulator emulates the input to output relationship of the sophisticated malaria model, thereby enabling the authors to do the large joint analysis, which would be computationally prohibitive using the malaria model alone. This is a practical solution to a computationally expensive problem. It could be applied to other computationally expensive models in epidemiology, if not already done so.

The results are impactful because they reinforce the need for continued surveillance of resistance to so-called partner drugs and they reinforce our understanding of drug properties that best withstand resistance. Three drug profiles were investigated: two monotherapies and a combination therapy that combines the two drugs used as monotherapies. The properties of the drugs mimic the properties of the drugs used in artemisinin-based combination therapies (ACTs). (The drug that is like artemisinin and its derivatives has a short half-life, high maximum kill rate and parasites resistant to can endure longer drug exposure times. The partner-like drug has a short half-life, low maximum killing rate and parasites resistant to it can endure higher drug concentrations.) ACTs are recommended for the treatment of malaria in almost all endemic counties. They include a fast-acting artemisinin derivative and a more slowly acting partner drug to kill residual parasites.

Supported by their simulated data, the authors conclude that partner drug resistance likely promotes the establishment and spread of artemisinin resistance. They then go on to say that their results support the belief that partner drug resistance precedes the evolution of artemisinin resistance. This belief is consistent with the spread of artemisinin resistance in the Greater Mekong Subregion but not in Africa. It cannot be tested directly in this study because the malaria model does not capture the sequential evolution of resistance, but the arguments the authors use to extrapolate from their results are logical.

Almost all the results are intuitive, and support previously published epidemiological and laboratory studies. Among the factors that can be acted upon, drug properties play an important role. Longer half-lives of the artemisinin-like drug hinder the spread of artemisinin-like resistance. Longer half-lives of the partner-like drug promote the spread of partner-like drug resistance but protect the artemisinin-like drug. Despite this protective effect, the authors conclude that "reducing the half-life of the partner drug in an ACT regimen could reduce the spread of resistance". Stated thus, this may seem counterintuitive. However, it is logical: longer half-lives of the partner drug likely compromise the artemisinin derivative in the long run by first promoting the emergence, establishment and spread of resistance to the partner drug. Nonetheless, it cannot be tested directly in this study because the malaria model does not capture the sequential nature of the evolution of resistance.

Although this study makes an important contribution, it has some weaknesses. Firstly, it does not capture the sequential evolution of resistance. Secondly, it is important to note that monotherapies are non-longer recommended for malaria treatment. Looking forward, the authors discuss briefly how their findings might extrapolate to triple combination therapies (TACTs), arguing that the two long-acting drugs of TACTs should ideally have matching half-lives. Although it seems reasonable to make this point based on extrapolation, a TACT-like drug profile merits full investigation. What would happen, for example, if the two long-acting drugs exert inverse drug pressure, selecting complementary mutations? Of course, it is not possible to consider all factors that might impact the spread of antimalarial drug resistance. Some potentially important factors not discussed presently in this manuscript include sub-quality drugs and additional factors that impact coverage, such as absorption (nutritional status). Recombination, an obligate stage of the malaria parasite lifecycle which does not feature in the malaria model, is mentioned briefly. Under a modified malaria model, recombination could affect some of the results at higher entomological inoculation rates (EIRs) because higher EIRs leads to more effective recombination. For example, resistance to the partner-like drug might not spread preferentially in high EIR settings when access to treatment is high. This is because the phenotypes of parasites resistant to partner drugs are typically encoded for by more than one mutation, so can thus be disrupted by recombination. Recombination could also affect the spread of artemisinin resistance. Although artemisinin resistance is typically encoded for by a single mutation, compensatory mutations elsewhere in the genome may play a role in mitigating the fitness cost. If so, recombination might restore the resistance cost in high EIR settings with low access to treatment. On the contrary, recombination could unite multiple mutations that encode drug resistance. In short, recombination could have a complicated and hard-to-intuit effect. It thus merits further investigation using a model.

We thank the reviewer for his/her supportive public review and for sharing his/her expert view on factors not captured by our models. We do identify a typo for non-specialist readers: the “partner-like drug” has a long half-life (we call it a “typo” because subsequent comments show the Reviewer is well aware of this fact).

(1) Sequential resistance and recombination

We recognise that OpenMalaria does not incorporate recombination is a limitation of our study. We have, in the past, seriously investigated whether to re-code our model to incorporate recombination, but for now, it would require fundamental, far-reaching changes to the mosquito model, parasite transmission model, and code. Additionally, to realistically represent recombination would result in significant increases in both memory use and computational time. We have prioritised using existing functionality rather than committing to resource-intensive code revisions, noting potential impacts of recombination on establishment and spread of resistance are addressed below and in our discussion.

Practically, the lack of recombination means that we can investigate the spread of resistance of one mutation at a time. For example, we could not simultaneously simulate the spread of a mutation that confers resistance to drug A and the spread of a mutation that confers resistance to drug B in a drug-sensitive parasite population. However, we could assume that resistance arises first to drug B and gets fixed before the emergence of resistance to drug A. Similarly, we assumed that the resistant genotypes had a fixed degree of resistance (i.e., a fixed number of mutations) and could not acquire a new mutation that could confer higher degrees of resistance across the simulation. However, we assessed how the selection pressure and impact of factors vary for different degrees of resistance (from low to high degrees of resistance). Thus, our model can capture the effect of a changing pattern of selection that occurred with the increasing degrees of resistance due to sequential evolution. However, we acknowledge that we did not dynamically model the sequential evolution of resistance. We added this remark to the discussion (L547-550).

In our paper we highlighted that the consequence of not modelling recombination is that we overestimated the evolution of drug resistance in settings with a high rate of infection when the resistant phenotype involves multiple mutations. The reviewer rightly highlighted that this could also influence the evolution of resistance to artemisinin (despite being caused by one mutation) due to compensatory mutation. We added this point to the revised version of our paper. We have also added that the effect of recombination depends on the frequency of each mutation needed to confer resistance. When these mutations are present at a low frequency (such as during the establishment phase), recombination will have a stronger effect as resistant parasites are more likely to recombine with sensitive parasites. Thus, the resistant phenotype is more likely to be lost. However, the impact of recombination decreases when the frequency of resistant mutations increases because resistant parasites are more likely to recombine with a resistant parasite. Thus, the main consequence of not simulating recombination events was that in high transmission settings, our model overestimated the probability of establishment of resistant parasites that need multiple mutations to be drug-resistant or that require additional mutations to restore the fitness cost. This means that the difference between the probability of establishment in high and low transmission settings is probably larger than reported here (see figure 4). In addition, we overestimated the spread of these resistant parasites when the mutations were present in low frequencies. However, these assumptions likely did not impact the probability of establishment and rate of spread of parasites that only need one mutation to confer resistance or do not have a mutation that reduces the fitness cost associated with resistance.

(2) Modelling Triple drugs (TACTs)

As pointed out by the reviewer, monotherapies are no longer recommended for malaria. Here we assessed the impact of factors on the evolution of parasite-resistance to the short-acting drug and the long-acting drug used separately in (non-recommended) monotherapy to identify determinants specific to each drug profile. This allowed us to identify some determinants that would not have presented themselves if we had only examined drug combination therapy. Once we identified which factors drive resistance for each drug profile, we looked at the combination of these two drug profiles and observed how dynamics changed. As suggested by the reviewer, the next step would be to look at the evolution of resistance under triple combination therapy.

Our study showed that resistance to the partner drug (long-acting, previously referred to as drug B) depends on the length of the selection window. This result supports the evidence that triple artemisinin combination therapies (TACTs) can delay the spread of resistance to partner drugs as it can minimise the selection pressure that occurs during the selection window if the two long-acting drugs have the same half-lives. While we agree future work could focus on selective pressures from different drug profiles in TACT, this would require a very large study to look at different profiles of three drugs etc. and is outside our scope of already a very large study. Even so, we believe no additional analysis is necessarily needed to highlight the points on TACT in the paper as they logically follow from our results.

However, we agree that other factors are likely to play a role in the evolution of resistance under TACTs, such as the inverse selection pressure generated by some drugs, as highlighted by the reviewer. Understanding the impact of factors on the evolution of resistance to TACTs is an important question and should be further investigated. However, this question is outside the scope of our study, as it would require running many more analyses and considering additional factors (such as the inverse selection pressure generated by some drugs, the synergic effect between drugs, or the fact that some mutation can conferee some degree of resistance to multiple drug, etc.), and could be considered in future work.

(3) General comments

As underlined by the reviewer, we did not directly assess the impact of sub-quality drugs and absorption (i.e. from poor nutritional status) on the rate of spread. However, one could extrapolate the impact of sub-quality drugs and absorption by recognising that both lead to a lower Cmax. In our study, we assessed the impact of low Cmax on the rate of spread, and thus one can extrapolate the effect of sub-quality drugs and absorption based on findings from our study. Factors such as poor nutrition could also affect other drug factors such as half-life, which we did not investigate, so we regard this as an important operational point made by the Reviewers which could be addressed in future studies.

code is breaking. curl should appear with the others and note separately.

Ruby sample code is missing from Github repo. Raised a slack thread with the developer on this.

can you check the Java code once please?

https://github.com/razorpay/razorpay-java/blob/master/documents/subscription.md#cancel-a-update

If you run the code like that you will get a consolidated table that does not make any sense. I suggest you run instead the following code:

        stargazer(
                    SR_AR1, 
                    SR_AR2, 
                    SR_AR4,

                     title = "AR Models of Monthly Excess Stock Returns",
          dep.var.caption  = "Dependent Variable: Excess Returns on the CSRP Value-Weighted Index",
    dep.var.labels.include = FALSE,
             model.numbers = FALSE,
             column.labels = c("AR(1)", "AR(2)", "AR(4)"),
                      type = "text",
                        se = rob_se )

There's a problem reading the Excel file directly from the online folder. I prefer to run this alternative code, which reads the ASC file directly from the internet:

SReturns <- read.csv("https://www.princeton.edu/~mwatson/Stock-Watson_3u/Students/EE_Datasets/Stock_Returns_1931_2002.asc", sep = "\t", header = FALSE )

colnames(SReturns) <- c( "Year", "Month", "ExReturn", "ln_DivYield" )

head(SReturns)

StockReturns <- ts( SReturns[, 3:4], start = c(1931, 1), frequency = 12)

head(StockReturns)

Reviewer #2 (Public Review):

Chen and Darst et al present a rigorous evaluation of a previously developed multi-ancestry polygenic risk score (PRS) for prostate cancer in multiple multi-ancestry datasets through meta-analysis. They found that their multi-ancestry PRS for prostate cancer shows strong risk stratification across European, African, and Hispanic populations (i.e., increased estimated associations with prostate cancer in increasing deciles of genetic risk). Consistent with previous literature, the authors showed that the PRS associations with prostate cancer risk attenuated in older men with higher genetic risks; the authors also show that this attenuation occurs in African ancestry men. Lastly, the authors show that men with higher genetic risk, across all three ancestry groups, reach a 5% absolute risk of prostate cancer far earlier than those in the median risk group. Overall, the results of the paper support the conclusions and the main take-home message of increasing sample sizes in non-European populations to fully evaluate the capabilities of this PRS in risk-stratifying during screening is clear.

The main strength of the study is a clear statement of aims and demonstration of conclusions. Applying this multi-ancestry PRS to multiple multi-ancestry datasets shows that the PRS is effective in risk stratification. The methods are well-articulated and the figures are easy to understand. I also commend the authors for providing links to data and sample code.

The main weakness is the lack of some background on polygenic scores. The manuscript is written for a genetic epidemiology and/or clinical audience, where a background understanding of polygenic scores is assumed. Adding context about PRS for a wider audience of life sciences readers would be warranted.

Reviewer #1 (Public Review):

The paper puts a lot of effort into many things that could make this work influential: the assumptions and parameter values under which the results hold are carefully examined, the approximations are difficult and carefully explained, the results are checked by simulations, and the underlying reasons for the results are explained in simple terms. In particular, the "linear" approximation is already enough for a good theoretical paper; the subsequent "nonlinear" approximation (which builds on the linear one) is very impressive. I am not certain precisely which results are new to this paper, but my impression is that it gives a much more complete picture of the details of polygenic adaptation than any previous work. The main limitation of the work is that it describes (and, simulates) a large number of unlinked loci, but this is entirely appropriate and well discussed in the paper.

My first observation is that, despite the author's good attention to detail and effort to explain what's going on, I found this to be a difficult paper, that I had to put a lot of work into to understand. (I did feel that that the work paid off eventually, though.) However, this is not a serious criticism - the topic is complex, and the paper does a good job of explaining the big picture. The hardest thing for me to keep straight was the various layers of approximations (I think: linear Lande, nonlinear Lande, linear non-Lande, and nonlinear non-Lande, each within each of the two phases - plus three different types of simulation). If it were possible to remove discussion of some of these parallel tracks without removing important conceptual results, I think that would help. However, I have no concrete suggestions.

Besides that, I have only one concern. The authors have but a lot of work into the simulations, but all plots show mean values, with no indication of between-simulation stochasticity. This makes sense, because the theory they develop describes mean quantities, but it would still be nice to know how well we expect the theory to predict dynamics of a single given bout of adaptation. For instance, Figure 2 shows the mean trajectories of trait mean, variance, and skew. What is the typical path of these for a single simulation trajectory? Or, Figure 5 shows how alleles of different sizes are expected to contribute to adaptation. How much do typical contributions to adaptation in a single simulation differ? Showing just one or two examples in the supplement could help make things more concrete.

Other comments:

- The github repository that's supposed to contain the code for the paper is empty. ( https://github.com/sellalab/PolygenicAdaptation1D )

- Many of the plots (e.g., Figure 5) show "contributions" to adaptation plotted against S, on a log scale. But, isn't this a density with respect to effect size, and so shouldn't we read these plots as histograms, with relative area under the curves giving the relative contributions to adaptation? If so, the log scale could give a very wrong idea, and changing variables so the curve is a function of log(S) would avoid the problem.

- It was hard for me to figure out a single set of simulation parameters to put into a forward simulator to match what was used in the paper, as the relevant information is scattered throughout (supplemental section 5.2 notwithstanding). To make things concrete, it would be nice to put a self-contained example in somewhere. I think that with a genome of length L, typical parameters were N=1e4, u=0.01/L, V_s=2e4, and an Exponential distribution of effect sizes with mean 4, equal probabilities + or -?

- The agreement between simulations of allele frequencies and the "full" (still unlinked) model is impressive (see Figure C5.1)

Reviewer #2 (Public Review):

Context:<br /> The authors propose a new analysis of an already well-studied conceptual model of adaptation to a new environment. Individual genotypes are characterized by some (breeding value for) phenotype under gaussian stabilizing selection (meaning that fitness is a gaussian function of phenotype, centered around some optimum value). The scenario assumed is that an isolated population of fixed size is initially at equilibrium (between mutation, selection and genetic drift). This population is diploid and sexual with many unlinked loci acting additively on phenotype (across loci and between homologous chromosomes). This view simplifies the analysis but is also not inconsistent with various empirical analysis of locus specific effects on quantitative traits (the empirical support is discussed and reviewed in both introduction and discussion).

Then a change in the environment induces a shift in the optimum without affecting any other parameter (strength of selection, population size, mutation effects, existing phenotypes), see figure 1. We wish to know how the population responds to this change, both in terms of phenotype distributions, and the underlying genetic basis (how alleles of various effects change in frequency and contribute to the phenotypic response).

This process has been at the core of the modelling of adaptation for more than a century, as it is maybe the most natural conceptual framework to describe adaptation to a new environment (a "niche shift" so to speak). It is relevant to both the study of demographic/ecological and phenotypic responses to changing conditions, and to the genomics of the changes associated with this process.<br /> However, in spite of this long history (reviewed in introduction in broad lines), we do not have an exact mathematical description of this process. The reason is that the problem is in fact very complex: the genome is a sea of various genes, each bearing various alleles (depending on the individual), that further interact mutually by selection (even though loci are additive on phenotype), because fitness is not a linear function of phenotype. The simple population genetics with two alleles and one locus seem far away...

I think it is fair to say that the main route to handle this problem, in predominantly sexual species, has been through the approximations of quantitative genetics. There, each locus is assumed of small effect and linkage disequilibrium between them is neglected. This has led to empirically testable, and often quite accurate, predictions on the response to selection in terms of mean phenotypic change. Yet, even under this broad approximation strategy, there are various ways to derive predictions, each neglecting one force or another (genetic drift most of the time), or looking at the process over short or longer timescales.

Aim and achievements:<br /> The authors include their work within this broad framework, but set to derive new approximations that are intended to cover several of the existing approach as subcases, and especially to handle genetic drift effects in finite populations (large ones), and short vs. longer timescales. I believe they succeed quite well in doing so: they provide clear approximation methods (in appendix mostly) and substantial simulations to show their accuracy. The derivations are fairly technical but most of the time they manage to give an intuition of where they come from and illustrate this intuition via figures in the main text. They produce a prediction of two main observable dynamics: that of the (breeding value for) phenotype itself (its mean over time, variance, third moment), and that of the genetic contribution of various loci and alleles along the genome (depending on the allelic effect on phenotype). They also describe two timescales where the dynamics are fairly different, a short timescale where the mean phenotype is shifting (quite rapidly over tens/hundreds generations) towards the new optimum, and a longer timescale where the higher moments and mostly the genetic basis changes while the mean phenotype merely wanders in a narrow vicinity of the new optimum. The connection between the two timescales is important as it is the slight differences in allele fates during the first one that result in differences in long term behavior in the longer one (illustrated in figure 3).

The main achievement on the phenotypic response is mostly to reobtain previous approximations under somewhat different or broader assumptions. This is not useless as it may explain why these known predictions (the "Lande model") are surprisingly robust to deviations from the required conditions (e.g. figure 2). However, I think that some extra exploration of the parameter space (away from the required conditions) would allow to really see when the Lande model does fail on mean phenotype dynamics over short timescales, as anticipated. The question of whether this range is relevant remaining open to empirical measurement.<br /> Therefore, the main contribution of this ms is not on phenotypic responses but on the underlying genetic basis, and what we may expect to observe when measuring QTL's or GWAS between two populations separated by an environmental shift in the past: are there many loci contributing limited difference, or fewer loci contributing most of it. In that respect, eqs 20-21 and 25-26-27, and figures 5 and 6 display the main findings and thei check by simulations. These findings, although stemming from quite elaborate derivations, yield a fairly simple and yet accurate outcome, at least in the parameter range studied. Various other parameter sets are also checked against simulations in the appendix, and the simulation code is made available for any further check (as exploring all the possible parameters is a fairly taunting task, for an article of its own probably).

Limits:<br /> I believe the main limit of this work is fairly explained in the discussion: to achieve mathematical tractability (a full numerical treatment being inherently impossible given the many parameters), many simplifying assumptions must be made (simple fitness landscape, simple effect of the environmental change, simple demography etc.). This means that it is possible that empirical observations will differ from the predictions for various reasons. However, quantitative genetics have already proven reasonably robust and accurate in predicting observed phenotypic dynamics, using comparable approximations so it is not madness to hope that the same will happen concerning the genetic basis of adaptation. Also, I would suspect that the methods proposed in appendix will most likely extend fairly easily to some deviations from the model's assumption: change in phenotypic variance with the new environment (a form of plasticity), or in width of the fitness function, or change the population size, without too much effect on the main conclusions. Still, some other limits may not be overcome as easily (e.g. pleiotropy among multiple traits, or non-stationary optimum), but it seems (a priori) that part of the approach could still be adapted for these situations. The main "wall-hitting" limit of the paper is inherent in the very basis of the approach, namely assuming mild changes occurring in weakly linked polymorphic and numerous loci as opposed to strong changes occurring on more tightly linked and fewer loci. These limits are all fairly described in discussion.

Overall, this paper is not an easy read, but not by lack of clarity, rather because the problem at hand is complex, and there is a lot of material to describe. Each part flows quite well in my opinion, but there are many parts to read.

Potential impact:<br /> I believe that because it yields relatively simple analytic outcomes (at least the predictions in main text), the paper could be useful to data analysis, mostly in the field of genomics of adaptation where it may provide testable predictions for GWAS and QTL data. It could also be used to infer genetic distributions (v(a),f(a)) from observed QTL or GWAS data, if the model is deemed valid.

In the field of theoretical population genetics, it may also provide a methodology to capture sexual adaptation dynamics in other contexts by mixing various approximation methods: connecting distinct timescales, connecting deterministic approximations for phenotype and diffusion approximations for allele frequencies. This may not be the first time of course (see e.g. "stochastic house of cards" and their extensions), but it is here used in the context of adaption dynamics rather than equilibria, for the first time I think.

应该没有问题吧？因为Trichrome已经分开啦！

动态尺寸代码被删除了，因为它很复杂，而且不能正确处理静态共享库APK--实际所需的空间量变化不大，硬编码值就足够了。

Lots of info on form validation

Webflow MultiStep Form - CODE GENERATOR

Live Site: https://advanced-forms.webflow.io/

Github Documentation: https://github.com/brotame/multi-step-form

think I made a mistake earlier. We need to have only one section. However, we will need two responses- one for Standard and one for UPI. This is because there is no difference in the request code for both PL variants.

think I made a mistake earlier. We need to have only one section. However, we will need two responses- one for Standard and one for UPI. This is because there is no difference in the request code for both PL variants.

I think I made a mistake earlier. We need to have only one section. However, we will need two responses- one for Standard and one for UPI. This is because there is no difference in the request code for both PL variants.

Go sample code is missing.

Go sample code is missing. Please check and add

Use this sample code but swap out the card parameters for the token parameter in all languges. https://razorpay.com/docs/payments/payment-gateway/s2s-integration/redirect/build-integration/#sample-code-1

Use this sample code but swap out the card parameters for vpa parameters in all languges. https://razorpay.com/docs/payments/payment-gateway/s2s-integration/redirect/build-integration/#sample-code-1

1.1.5

I guess this shows some simplistic models the interpreter uses to evaluate combinations. Basically it subsitutes the definitions of procedures then evalues it like a combination

Sometimes we use normal order evaluation to better understand teh code or something in whcih we keep subsying until we reach a point where we only have primitive procedures. This can be inefficinet so its not used by interpreter but it is used sometimes

system-level complexity

disconnectedness

product- and app-centric sprawl

de-facto data cartels

!- for : broken paradigm

from the Broken Paradigm to Paradigm Regained

**

What it means to "comprehend" a generative text -- which is not just re-reading, since the object itself is in flux.

It's not recommended to use a function before it's declared, as it may give your code some errors!

This HTML code is very high level for a basics student! Great job! Some pointers here:

1. In this case, I think it's better to separate your css into another file. Link it to your html afterwards.
2. I like how you're using grid! Maybe learn flex boxes too! It could make your css styling easier. Once you've learnt what flex boxes are, you'd never want display: block/inline-block ever again.
3. if you ever want extra challenge, do read up on bootstrap! That would actually make css stylings SUPER easy. You don't even need to define breakpoints for smaller screens too! Bootstrap will take care of everything for you!
4. Also, if you're really keen on doing css-based animations, do read up on animate.css as well! It's something that's pre-defined and it saves you all the hassle of doing great animations on your own.

I'm really excited for your upcoming projects now! You seem to be super comfortable in html/css! If possible, no more pink box and white bg please!

A solid attempt at your first project! Here are a few pointers:

1. I know that you've struggled quite a bit in getting this up and running. I think you just need more practice in JS. Try going back to any exercise that you might not have been super comfortable with (like say, functions in this project) and re-doing them on your own. This should help your understanding of javascript more!
2. Maybe it's easier that you do your pseudocode before you attempt coding this out. It will help you map out what you want and need to be done. You might need to be fairly specific about it. So something like --check whether player win-- is not recommended, and something like --check if player has scissors and computer has paper, player win if condition met--- is more recommended.
3. I like how you want to make it the most comfortable version, I can give you some pointers on how you can make that happen, and if you still want to experiment with this code, feel free to do so!
4. You're very close to implementing the win/draw/lose percentage. What you can do is to add a counter on how many times you've played the game, and divide the win/draw/lose amt to that counter.
5. for reverse game mode / korean sps - You can add more game states, or new win/lose checking for all the reversed / korean sps game states. Granted the korean sps is very hard to implement, only very few people in all of the basics batches successfully do it.
6. you can also add a new game mode for computers! if you want to do so! It's basically just 2 random rolls and check it against one another.

I know that you can do what needs to be done on the next project! It might be hard but keep working on it, if you don't understand a concept, do look at and re-do your pre-class/in-class exercises! I'm looking forward to see your project 2 submission!

Great use of helper functions to spread the logic of your code!

It's good that you're assigning constants on the top of the page! It's a great fail-safe for typos in your app.

Although I don't think you're using the AI variable in any of your code

Great attempt at your first project! A couple of pointers:

1. Usually we remove console logs when we submit projects, because console logs are just to help you debug your own code, and the end user will not see it.
2. Variable names and function names are clear! Nice going!
3. Great job for making the game run!
4. This is personal preference for readability, but there are two varying opinions for returns in a function: single returns (only 1 return per each function) and multiple early returns (you can have a lot of returns in a function). Rocket (and I personally) follow multiple return conventions, but I will not criticize people who does single returns. If - else if - else syntax (lines 42-55) is usually paired with single return convention. if - if - if syntax (lines 16-24) is usually paired with multiple return convention. It's best practice to take one convention and stick with it, for the sake of readability, rather than changing conventions back and forth. This is just a "nice to know" bit, don't need to really stress over this.

Overall, great attempt for this project! I'm looking forward to your submissions for the upcoming projects!

the error code should appear in the right dark pane

the error code should appear in the right dark pane

The sample code Request heading can be just 6-Digit IINs and 9-Digit IINs

The robust standard errors are not correct for model "3" because of the error in the code above.

Reference to Shakespeare's Twelfth Night. Referencing already established popular works as a means of establishing artistic value, as well as situating oneself within the culture and the canon. Brown utilises similar tactics in his evocations of famous artworks in The Davinci Code, and Gay also uses similar strategies in referencing pre-existing Marvel characters to situate the story of Ayo and Aneka.

Author Response

Reviewer #1 (Public Review):

As we lack empirical data of the response of most species to environmental changes, developing predictive tools based on traits that are easier to access or infer may help us developing better management tools. This is the case even for terrestrial mammals, a rather well studied group but with a large study bias towards temperate Europe and North America. This study uses maximum longevity, litter size and body mass to predict the sign and size of the relationships between annual temperature and precipitation anomalies and population growth rates, using the Living Planet database for times series of abundance and Chelsa for weather anomalies. The authors use a Bayesian framework to relate the size and absolute magnitude of the relationships between detrended population growth rates and weather anomalies, the framework accounting for the uncertainty in estimates as well as phylogenetic dependencies. They did not find any systematic effects -- on average the slopes of the relationships were close to 0 -- but the magnitude of the coefficients decreases for species with high maximum longevity and low litter size. Therefore, this study points to possible predictions of the magnitude of the response to weather variability using simple demographic indices such as longevity and litter size. The study has clear limitations that are common to similar "meta-regressions" using publicly available databases, but they are not ignored when discussing the results. One would hope that such limitations would lead to improving the quality of such databases, both in terms of taxonomic and geographic coverage as well as quality of data.

We would like to thank Reviewer 1 for their overall positive feedback and constructive comments on the method and our predictions. We have now included complementary analyses based on high-quality subsets (≥ 20-year records; using life history traits estimated from structured population models), have clarified our set of hypotheses and discussed our results accordingly. Detailed responses are given below.

I would like to challenge the authors in terms of why one would expect relationships of a given sign or magnitude. First with respect to sign of relationships, even for the same species and the same weather parameters, one could expect different signs depending on where the study is done with regards to the climatic niche. If one is close to the warm (or wet) edge, any positive temperature (or precipitation) anomalies would probably have a negative effect, but the reverse would happen when close to the cold or dry edge. There are studies showing such demographic and growth rate variability differences. I find therefore hard to interpret the sign of such weather anomalies and what it tells us about the "effect" of weather variability.

We think that this is an important point to discuss with respect to the importance of within-species variability in population dynamics. Certainly, from the results L203-206 it is clear that populations of the same species can have responses of differing signs. It is also interesting to note that this may be the result of a population’s position in the climatic niche. However, aside from exploring this for species with long-term demographic monitoring across the range, we do not feel that exploring this was in the scope of the current study across species. We agree fully however that adding this perspective to studies of how populations are responding to changing climates is critical. As well as the paper mentioned below by Gaillard et al. (2013), recent work in Plantago lancelota with extensive spatial replication has also begun to reveal these within-range dynamics as a function of latitudinal or climatic gradients (Römer et al. 2021). We have added further discussion of this to the manuscript L330-340. We believe that this point adds to the context of our results highlighting variability within-species. In addition, we have clarified in the introduction that no clear directional responses of populations to weather anomalies was expected among and within species L133-135.

Römer, G., Christiansen, D. M., de Buhr, H., Hylander, K., Jones, O. R., Merinero, S., ... & Dahlgren, J. P. (2021). Drivers of large‐scale spatial demographic variation in a perennial plant. Ecosphere, 12(1), e03356.

Second with regards to the magnitude, it is clear that the maximum growth rate is strongly linked to maximum longevity and litter size -- slow species have a much lower maximum rate of growth than fast species. So, one would expect that variability of population growth rates is larger in fast species than slow species, and therefore the magnitude of their response to environmental variability. Now the question might also be whether weather variability explains a smaller or larger proportion of the variability in population growth rates -- that is, does weather have a relatively larger influence in fast species than slow species? You might have the answer but with the multiple standardizations of the response and predictor variables it is not obvious (that is, when you standardize the response and predictor variables, coefficients are correlations, but this is across species, not for a given population).

The reviewer raises a very interesting and important point on whether the patterns we observe are simply a result of larger variability in growth rates in short-lived species. We have two responses to this point: 1) while there is indeed larger variation in the population growth rates of short-lived species, we believe that this variability is likely an evolved life-history strategy in response to the environment, and thus a key component of patterns we observe, 2) we also feel that our use of models that included annual effects, and state-space models with explicit process-noise terms, account for any confounding effect of this variation.

To address the first point in more detail, we expect that life-histories (and thus population dynamics) are evolved responses to the environment (Stearns, 1992). For ‘fast’ organisms therefore, their intrinsic life-history strategy results in boom-bust population dynamics relative to ‘slow’ species. This is clearly observable in transient or non-asymptotic dynamics, where short-lived species more often have short-term population dynamics with a greater magnitude (Stott et al. 2011). On this point, we therefore argue that this variation in population growth is part of what we are trying to capture. Anomalies in the weather are therefore expected to act more strongly in ‘fast’ species. Following this point and the comments of Reviewer #3, we have now included more explicit hypotheses in terms of life-history L133-144.

For the second point, while we may expect this variability to be the result of dynamics we are trying to capture, this does not preclude other sources of variation in population size confounding the patterns we could observe. For example, hunting pressure may influence both short-term population variability and long-term trends. As a result, we aimed to capture this residual variation using auto-regressive terms for year in our GAMs. While these terms do not explicitly model variability in population growth, they do account for a component of the trend, with variation (error around the trend, which is expected to be larger for fast species), and auto-regressive components of population change. Moreover, we did additional analyses using a state-space modelling approach. In the state-space approach, process noise, which in our case would equate to variability in population growth, is explicitly modelled and accounted for. We therefore believe that our analyses account for residual variability in population growth rates. State space models were also highly correlated with our auto-regressive GAMs, and we can therefore conclude that we do not expect that this variability influences our findings. We have now asserted this in the Methods section L531-535.

Stearns, S.C., 1992. The evolution of life histories (No. 575 S81).

Stott, I., Townley, S. and Hodgson, D.J., 2011. A framework for studying transient dynamics of population projection matrix models. Ecology Letters, 14(9), pp.959-970.

Your analyses remove trends -- that is, climate or other systematic change as opposed to weather anomalies (yearly differences) -- and trends might be the main concerns in terms of conservation. This is made clear in the discussion but perhaps not as much in the introduction where you seem to focus on climate change (the title reflects this well, however, as you mention weather, not climate). This confusion between weather and climate is often made in the literature, when reference is made to climate effects rather than weather effects.

We agree with the reviewer that climate and weather are often conflated in ecological studies. We apologise for this oversight in the introduction, and agree that the narrative and link to weather was not made explicit in the previous version. Following this point and the suggestions of Reviewer #3, we have now restructured large sections of the introduction to improve the clarity of our hypotheses. To address this point, we have now included specific introduction of different components of climate that species populations may respond to, including short-term extreme weather patterns as we explore in this study. Please find this revised section L80-97.

Finally, I would like to see a measure of how good is the prediction you can make using traits. You may have "significant effects" but not helping much in terms of prediction (see PB Adler et al. 2011 in Science, for an example with species richness and productivity).

On this point we disagree with the reviewer. The core of our analysis framework was to examine the predictive performance of models. We do not report any significant effects, and instead use Bayesian inference. Throughout the analysis framework, we used explicit tests of out-of-sample predictive performance with leave-one-out cross validation (Vehtari et al. 2017). This is asserted in the manuscript title and results section when introducing our spatial analysis L188-191. Cross validation was combined with model selection to test the predictive performance of a set of candidate models with respect to base models excluding predictors of interest. This predictive performance framework was not applied to examine the directional effects (question 1), as these models did not contain key predictors. However, model selections using predictive performance were done throughout questions 2 and 3, to explore spatial and life-history effects. We highlight this point in both the results L188-191 and methods sections L608-615. In the case of life-history, we found that relative to the base model, out-of-sample predictions were improved when including univariate life-history traits relative to the base model, and thus life-history traits aid in predicting weather responses.

We did not explore the relative predictive performance of life-history traits with respect to other traits such as dietary specialisation, which have been shown to be important in climate responses (Pacifici et al. 2017). We believe that this would have been out of scope for the purpose of the current study, where we aimed to test specific hypotheses established in life-history theory.

Pacifici, M., Visconti, P., Butchart, S.H., Watson, J.E., Cassola, F.M. and Rondinini, C., 2017. Species’ traits influenced their response to recent climate change. Nature Climate Change, 7(3), pp.205-208.

Vehtari, A., Gelman, A. and Gabry, J., 2017. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and computing, 27(5), pp.1413-1432.

Reviewer #2 (Public Review):

Jackson et al. present a global analysis of the effects of life history on the response of terrestrial mammal populations to weather, showing that litter size and longevity significantly alter how populations respond to anomalies in temperature and rainfall. The topic is highly interesting, as it has implications for what data we should monitor to make more reliable predictions about species' responses to climatic change, and how we should prioritise which species to conserve by identifying those which might be at greatest risk.

The authors comprehensively validate their results with substantial secondary analyses, and I believe that their assertions are supported by the results presented here. Whilst global scale analyses such as this provide useful generalities, they should be taken as that: an investigation of the general trends observed across large spatial scales, and caution should be taken extrapolating too far away from the species which have been analysed for this study.

We thank the reviewer for their positive feedback, and agree with not drawing too many generalities from our findings. In the first paragraph of the discussion L253-262, we now explicitly refer to the results in the context of mammal population-dynamics/conservation.

Reviewer #3 (Public Review):

In this study, the authors aim to investigate how mammalian species are likely to respond to climate change. To this end, they investigate the effects of weather anomalies on the growth rates of mammalian populations. They use long-term population records for 157 terrestrial mammals from the Living Planet database. They explore three different questions using a two-step modelling approach: (1) whether temperature and precipitation anomalies have significant effects on population growth rates across species; (2) whether responses differ among species and biomes; and (3) whether life-history traits explain species responses to weather anomalies.

The work undertaken in this manuscript is of broad appeal in the field and has the potential to inform conservation. Overall, the methodology is sound and the modelling framework robust; the authors took care to test the robustness of their models by fitting alternative sets of models. The two-step design of this study is interesting and the choice of the study system is relevant for the questions the authors aim to tackle. The authors also paid attention to some important points that are at times overlooked such as resolving taxonomy before running their analyses. I also appreciated the fact that the authors made their code available.

We thank the reviewer for their positive feedback on the manuscript, which highlights many of our key goals with the paper.

I nevertheless think that, in its present form, the main weakness of this manuscript is the clarity of the writing, the framing of the study and the overall flow. I found the manuscript at times a bit difficult to follow. That said, I think there is much scope for the authors to improve it. First, I think the work would benefit from better explanation of the underlying hypotheses. Second, in some places I think the authors go into a lot of details at the expense of clarity. As such, I think the authors should strive to better balance clarity with detailed information (notably in the results and methods; adding summary sentences, for example, could help clarify these sections). Third, I think there is room for improvement in the narrative and the flow of the introduction and the discussion. Finally, I think stronger justifications are sometimes required regarding specific points of the analysis.

I believe that the conclusions of this work are supported by the data and the analyses, and think they are of interest and relevant to the field. However, I think the discussion should highlight the main limitations of the study. In particular, I think the biases in the data should be discussed, and notably whether these biases are expected to affect the results (and if so, in what way).

To conclude, I think that beyond the aforementioned weaknesses of this study, the results and the methods are of interest for the field. I think the modelling framework is applicable to other study systems and relevant to the field as well.

We warmly thank the reviewer for their positive words and thorough constructive feedback. We have extensively re-worked large sections of the manuscript (particularly the discussion and introduction) based on these points, and done our best to address all of them. Generally, we have strived to improve the clarity and succinctness of the manuscript.

Author Response

Reviewer #1 (Public Review):

In this study, Guggenmos proposes a process model for predicting confidence reports following perceptual choices, via the evidence available from stimuli of various intensities. The mechanisms proposed are principled, but a number of choices are made that should be better motivated - I develop below a number of concerns by order of importance.

I’d like to thank the reviewer for their thorough and excellent review. It’s no set phrase that this review substantially improved the manuscript.

1) Lack of separability of the two metacognitive modules.

Can the author show that the proposed model can actually discriminate between the noisy readout module and the noisy report module? The two proposed modules have a different psychological meaning, but seem to similarly impact the confidence output. Are these two mutually exclusive (as Fig 1 suggests), or could both sources of noise co-exist? It will be important to show model recovery for introducing readout vs. report at the metacognitive level, e.g., show that a participant best-fitted by a nested model or a subpart of the full model, with a restricted number of modules (some of the parameters set to zero or one), is appropriately recovered? (focusing on these two modules) This raises the question of how the two types of sigma_m are recoverable/separable from each other (and should they both be called sigma_m, even if they both represent a standard deviation)? If they capture independent aspects of noise, one could imagine a model with both modules. More evidence is needed to show that these two capture separate aspects of noise.

Testing the separability of the two noise types (readout, report) is a great idea and I have now performed a corresponding recovery analysis. Specifically, I have simulated data with both noise types for different regimes of sensory and metacognitive noise. As shown in the new Figure 7—figure supplement 6, the noise type can be precisely recovered in the most typical regimes.

I now refer to this analysis in the subsection 2.4 Model recovery (Line 521ff):

“One strength of the present modeling framework is that it allows testing whether inefficiencies of metacognitive reports are better described by metacognitive noise at readout (noisy-readout model) or at report (noisy-report model). To validate this type of application, I performed an additional model recovery analysis which tested whether data simulated by either model are also best fitted by the respective model. Figure 7—figure supplement 6 shows that the recovery probability was close to 1 in most cases, thus demonstrating excellent model identifiability. With fewer trials per observer, recovery probabilities decrease expectedly, but are still at a very good level. The only edge case with poorer recovery was a scenario with low metacognitive noise and high sensory noise. Model identification is particularly hard in this regime because low metacognitive noise reduces the relevance of the metacognitive noise source, while high sensory noise increases the general randomness of responses.”

In principle, both noise modules can co-exist and model inversion should be possible (though mathematically more complicated). On the other hand, I anticipate that parameter recovery would be extremely noisy in such a scenario. For this work, I decided to not test this possibility as it would add a lot of complexity, with a high probability of ultimately being unfeasible.

2) The trade-off between the flexibility of the model (modularity of the metacognitive part, choice of the link functions) and the generalisability of the process proposed seems in favor of the former. Does the current framework really allow to disambiguate between the different models? Or at least, the process modeled is so flexible that I am not sure it allows us to draw general conclusions? Fig 7 and section 3 of the results explain that all models are similar, regardless of module of functions specified; Fig 7 supp shows that half of participants are best fitted by noisy readout, while the other half is best fitted by noisy report; plus, idiosyncrasies across participants are all captured. Does this compromise the generalisability of the modeling of the group as a whole?

This is a fair point and I understand the question has two components: a) is the model too flexible, potentially preventing generalized conclusions? b) is the flexibility of the model recoverable?

Regarding a), I should emphasize that the manuscript (and toolbox) provides a modeling framework, rather than a single specific model. In other words, researchers applying the framework/toolbox must make a number of decisions: which noise type? which metacognitive biases should be considered? which link function? To ensure interpretability / generalizability, researchers have to sufficiently constrain the model. Due to this framework character, it makes sense that the manuscript is submitted under the Tools & Resources Article format rather than the Research Article format.

On the other hand, I agree that it is the duty of the manuscript introducing the framework to provide all necessary information to help the researcher make these decisions. This is where the reviewer’s point b) is critical and I hope that with the new parameter and model recovery analyses in the present revision (see other comments) I meet this requirement to a satisfactory degree.

To clarify the scope and aim of the paper, I now put a new subsection in front of the example application to the data from Shekhar and Rahnev, 2021 (Line 534ff):

“It is important to note that the present work does not propose a single specific model of metacognition, but rather provides a flexible framework of possible models and a toolbox to engage in a metacognitive modeling project. Applying the framework to an empirical dataset thus requires a number of user decisions: which metacognitive noise type is likely more dominant? which metacognitive biases should be considered? which link function should be used? These decisions may be guided either by a priori hypotheses of the researcher or can be informed by running a set of candidate models through a statistical model comparison. As an exemplary workflow, consider a researcher who is interested in quantifying overconfidence in a confidence dataset with a single parameter to perform a brain-behavior correlation analysis. The concept of under/overconfidence already entails the first modeling decision, as only a link function that quantifies probability correct (Equation 6) allows for a meaningful interpretation of metacognitive bias parameters. Moreover, the researcher must decide for a specific metacognitive bias parameter. The researcher may not be interested in biases at the level of the confidence report, but, due to a specific hypothesis, rather at metacognitive biases at the level of readout/evidence, thus leaving a decision between the multiplicative and the additive evidence bias parameter. Also, the researcher may have no idea whether the dominant source of metacognitive noise is at the level of the readout or report. To decide between these options, the researcher computes the evidence (e.g., AIC) for all four combinations and chooses the best-fitting model (ideally, this would be in a dataset independent from the main dataset).”

In addition, the website of the toolbox now provides a lot more information about typical use cases: https://github.com/m-guggenmos/remeta

3) More extensive parameter recovery needs to be done/shown. We would like to see a proper correlation matrix between parameters, and recovery across the parameter space, not only for certain regimes (i.e. more than fig 6 supp 3), that is, the full grid exploration irrespective of how other parameters were set.

The recovery of the three metacognitive bias parameters is displayed in Fig 4, but what about the other parameters? We need to see that they each have a specific role. The point in the Discussion "the calibration curves and the relationships between type 1 performance and confidence biases are quite distinct between the three proposed metacognitive bias parameters may indicate that these are to some degree dissociable" is only very indirect evidence that this may be the case.

A comprehensive parameter recovery analysis is indeed a key analysis that was missing in the first version of the manuscript. I now performed several analyses to address this, rewrote and extended section 2.3 on parameter recovery. The new parameter recovery analysis was performed as follows (Line 455ff):

“To ensure that the model fitting procedure works as expected and that model parameters are distinguishable, I performed a parameter recovery analysis. To this end, I systematically varied each parameter of a model with metacognitive evidence biases and generated data. Specifically, each of the six parameters (σs, ϑs, δs, σm, 𝜑m, δm) was varied in 500 equidistant steps between a sensible lower and upper bound. The model was then fit to each dataset. To assess the relationship between fitted and generative parameters, I computed linear slopes between each generative parameter (as the independent variable) and each fitted parameter (as the dependent variable), resulting in a 6 x 6 slope matrix. Note that I computed (robust) linear slopes instead of correlation coefficients, as correlation coefficients are sample-sizedependent and approach 1 with increasing sample size even for tiny linear dependencies. Thus, as opposed to correlation coefficients, slopes quantify the strength of a relationship. Comparability between the slopes of different parameters is given because i) slopes are – like correlation coefficients – expected to be 1 if the fitted values precisely recover the true parameter values (i.e., the diagonal of the matrix) and ii) all parameters have a similar value range which makes a comparison of off-diagonal slopes likewise meaningful. To test whether parameter recovery was robust against different settings of the respective other parameters, I performed this analysis for a coarse parameter grid consisting of three different values for each of the six parameters except σm, for which five different values were considered. This resulted in 35·51 = 1215 slope matrices for the entire parameter grid.”

In addition, I computed additional supplementary analyses assessing a case with fewer trials, a model with confidence biases, and models with mixed evidence and confidence biases. For details about these analyses, I kindly point the reviewer to section 2.3. Together, these new analyses demonstrate that parameter recovery works extremely well across different regimes and for all model parameters, including the metacognitive bias parameters mentioned in the reviewer’s comment.

1.8: It would be important to report under what regimes of other parameters these simulations were conducted. This is because, even if dependence of Mratio onto type 1 performance is reproduced, and that is not the case for sigma_m, it would be important to know whether that holds true across different combinations of the other parameter values.

I now repeated this analysis for various settings of other parameters and include the results as new Figure 6—figure supplement 2. While the settings of other parameters affect the type 1 performance dependency of Mratio (with some interesting effects such as Mratio > 1), parameter recovery of sigma_m is largely unaffected. The same basic point thus holds: Mratio shows a nonlinear dependency with type 1 performance, but sigma_m can be recovered largely without bias under most regimes (the main exception is a combination of low sensory noise and high metacognitive noise under the noisy-readout model, which is also mentioned in the manuscript).

Is lambda_m meaningfully part of the model, and if so, could it be introduced into the Fig 1 model, and be properly part of the parameter recovery?

I now reworked the part about metacognitive biases to make it more consistent and to introduce lambda_m on equal footing with the other metacognitive bias parameters. I now distinguish between metacognitive evidence biases (the two main bias parameters of the original model, phi_m and theta_m) and metacognitive confidence biases, i.e. lambda_m and a new additive confidence bias parameter kappa_m. The schematic presentation of the model framework in Figure 1 is updated in accordance:

This change also complies with reviewer 2, who rightfully pointed out that the original model framework put much stronger emphasis on bias parameters loading on evidence than on confidence. The metacognitive confidence bias parameters are now also part of the parameter recovery analyses (Figure 7—figure supplement 2).

While it is still feasible to combine the two evidence-related bias parameters and lambda_m – as queried by the reviewer – not all mixed combinations of evidence- and confidence-related bias parameters perform well in terms of model recovery (in particular, combining all four parameters; cf. Figure 7—figure supplement 3). Hence, a decision on the side of the modeler is required. I comment on this important aspect at the end of the section 1.4 about metacognitive biases (Line 276ff):

“Finally, note that the parameter recovery shown in Figure 4 was performed with four separate models, each of which was specified with a single metacognitive bias parameter (i.e., 𝜑m, δm, λm, or Km). Parameter recovery can become unreliable when more than two of these bias parameters are specified in parallel (see section 2.3; in particular, Figure 7—figure supplement 3). In practice, the researcher thus must make an informed decision about which bias parameters to include in a specific model (in most scenarios one or two metacognitive bias parameters are a good choice). While the evidence-related bias parameters 𝜑m and δm have a more principled interpretation (e.g., as an under/overestimation of sensory noise), it is not unlikely that metacognitive biases also emerge at the level of the confidence report (λm, km). The first step thus must always be a process of model specification or a statistical comparison of candidate models to determine the final specification (see also section 3.1).”

4) An important nuance in comparing the present sigma_m to Mratio is that the present model requires that multiple difficulty levels are tested, whereas instead, the Mratio model based on signal detection theory assumes a constant signal strength. How does this impact the (unfair?) comparison of these two metrics on empirical data that varied in difficulty level across trials? Relatedly, the Discussion paragraph that explained how the present model departs from type 2 AUROC analysis similarly omits to account for the fact that studies relying on the latter typically intend to not vary stimulus intensity at the level of the experimenter.

I thank the reviewer for this comment which made me realize that I incorrectly assumed that my model requires multiple stimulus difficulty levels. The only parameter that would require multiple stimulus intensities is the sensory threshold parameter, but for this parameter I already state that it requires additional stimulus difficulties close to threshold (Line 147ff). Otherwise I now made extensive tests that the model works just fine with constant stimuli. My reasoning mistake (iirc) was related to the fact that I fit a metacognitive link function, which I thought would require variance on the x-axis; but of course there is already plenty of variance introduced through noise at the sensory level, so multiple difficulty levels are not required to fit the metacognitive level. I now removed the relevant references to this requirement from the manuscript.

Nevertheless, I agree that it is interesting to perform the comparison between Mratio and sigma_m also for a scenario with constant stimuli. See both the new Figure 6–supplement 1 with constant stimuli, and the (updated) main Figure 6 with multiple stimulus levels for comparison.

The general point still holds also for constant stimuli: Mratio is not independent of type 1 performance. Thus, the observed dependence on type 1 performance is not due to the presence of varying stimulus levels. I now reference this new supplementary figure in Result section 1.8 (Line 389).

5) 'Parameter fitting minimizes the negative log-likelihood of type 1 choices (sensory level) or type 2 confidence ratings (metacognitive level)'. Why not fitting both choices and confidence at the same time instead of one after the other? If I understood correctly, it is an assumption that these are independent, why not allow confidence reports to stem from different sources of choice and metacognitive noise? Is it because sensory level is completely determined by a logistic (but still, it produces the decision values that are taken up to the metacognitive level)?

The decision to separate the two levels during parameter inference was deliberate. I now explain this choice in the beginning of Result section 2 (Line 416ff):

“The reason for the separation of both levels is that choice-based parameter fitting for psychometric curves at the type 1 / sensory level is much more established and robust compared to the metacognitive level for which there are more unknowns (e.g., the type of link function or metacognitive noise distribution). Hence, the current model deliberately precludes the possibility that the estimates of sensory parameters are influenced by confidence ratings.”

Indeed, I would regard it as highly problematic if the estimates of sensory parameters were influenced by confidence ratings, which are shaped by a manifold of interindividual quirks and biases and for which computational models are still in a developmental stage. Yet, from a pure simulation-based parameter recovery perspective, in which the true confidence model is known, using confidence ratings would indeed make sensory parameter estimation more precise (because of the rich information contained in continuous confidence ratings which is lost in the binarization of type 1 choices).

6) Fig 4 left panels: could you clarify the reasoning that due to sensory noise, overconfidence is expected, instead of having objective and subjective probability correct aligning on the diagonal? Shouldn't the effects of sensory noise average out? In other words, why would the presence of sensory noise systematically push towards overconfidence rather than canceling out on average?

As an intuitive explanation consider the case that no signal is present in a stimulus, e.g., a line grating in a clockwise/counterclockwise orientation discrimination task with an angle of 0 degrees. Since there is no true information in the stimulus, type 1 performance will be at chance level irrespective of sensory noise.

However, sensory noise matters for the metacognitive level. Assuming no sensory noise (i.e., sigma_s = 0), the observer’s stimulus/decision variable would be zero and thus confidence would be zero. Thus, confidence would exactly match type 1 performance. Yet, assuming the presence of sensory noise, the stimulus estimate (“decision value”) will be always different from point-zero, if ever so slightly. While the average estimate of the stimulus variable across trials will indeed cancel out to zero, each individual trial will be different from zero (in either direction) and hence also the confidence will be different from zero in each trial. Since confidence is unsigned, the average confidence will be greater than zero and thus give the impression of an overconfident observer.

Note that this explanation was implicitly included in the paragraph on the 0.75 signature of confidence (“When evidence discriminability is zero, an ideal Bayesian metacognitive observer will show an average confidence of 0.75 and thus an apparent (over)confidence bias of 0.25. Intuitively this can be understood from the fact that Bayesian confidence is defined as the area under a probability density in favor of the chosen option. Even in the case of zero evidence discriminability, this area will always be at least 0.5 − otherwise the other choice option would have been selected, but often higher.”, Line 257ff).

7) The same analysis as Fig 6 but for noisy readout instead of noisy reports do not show the same results: both sigma_m and m-ratio vary as a function of type 1 performance. Does this mean that the present model with readout module does not solve the issue of dependency upon type 1 performance?

I refer to this in the Result section: “The exception is a regime with very high metacognitive noise and low sensory noise under the noisy-readout model, in which recovery becomes biased” (Line 391ff). Indeed, the type 1 performance dependency of sigma_m recovery in this edge case is not as good as in the noisyreport model. However, note that recovery is stable across a large range of d’ including the range typical aimed for in metacognition experiments (i.e., medium performance levels to ensure sufficient variance in confidence ratings).

It is also important to point out that a failure to recover true parameters under certain conditions is not a failure of the model, but a reflection of the fact that information can be lost at the level of confidence reports. For example, if sensory noise is very high, the relationship between evidence and confidence becomes essentially flat (Figure 3), producing confidence ratings close to zero irrespective of the level of stimulus evidence. It becomes increasingly impossible to recover any parameters in such a scenario. Vice versa if sensory noise is extremely low, confidence ratings approach a value of 1 irrespective of stimulus evidence, and the same issue arises. In both cases there is no meaningful variance for an inference about latent parameters. This issue is more pronounced in the noisy-readout case because it requires an inversion of precisely the relationship between evidence and confidence.

8) In Eq8, could you explain why only the decision values consistent with the empirical choice are filtered. Is this an explicit modeling of the 'decision-congruence' phenomenon reported elsewhere (eg. Peters et al 2017)? What are the implications of not keeping only the congruent decision values?

I apologize, this was a mistake in the manuscript. The integration is over all decision values, not just those consistent with the choice. I corrected it accordingly.

Reviewer #2 (Public Review):

This paper presents a novel computational model of confidence that parameterises links between sensory evidence, metacognitive sensitivity and metacognitive bias. While there have been a number of models of confidence proposed in the literature, many of these are tailored to bespoke task designs and/or not easily fit to data. The dominant model that sees practical use in deriving metacognitive parameters is the meta-d' framework, which is tailored for inference on metacognitive sensitivity rather than metacognitive biases (over- and underconfidence). This leaves a substantial gap in the literature, especially as in recent years many interesting links between metacognitive bias and mental health have started to be uncovered. In this regard, the ReMeta model and toolbox is likely to have significant impact on the field, and is an excellent example of a linked publication of both paper and code. It's possible that this paper could do for metacognitive bias what the meta-d' model did for metacognitive sensitivity, which is to say have a considerable beneficial impact on the level of sophistication and robustness of empirical work in the field.

The rationale for many of the modelling choices is clearly laid out and justified (such as the careful handling of "flips" in decision evidence). My main concern is that the limits to what can be concluded from the model fits need much clearer delineation to be of use in future empirical work on metacognition. Answering this question may require additional parameter/model recovery analysis to be convincing.

I thank the reviewer for these encouraging and constructive comments!

Specific comments:

• The parameter recovery demonstrated in Figure 4 across range of d's is impressive. But I was left wondering what happens when more than one parameter needs to be inferred, as in real data. These plots don't show what the other parameters are doing when one is being recovered (nor do the plots in the supplement to Figure 6). The key question is whether each parameter is independently identifiable, or whether there are correlations in parameter estimates that might limit the assignment of eg metacognitive bias effects to one parameter rather than another. I can think of several examples where this might be the case, for instance the slope and metacognitive noise may trade off against each other, as might the slope and delta_m. This seems important to establish as a limit of what can be inferred from a ReMeta model fit.

This is an excellent point and was also raised by reviewer #1. See major comment 3 of reviewer #1 for a detailed response. In short, I now provide comprehensive analyses that demonstrate successful parameter recovery across different regimes and both noisy types (noisy-readout, noisy-report). See Figure 7.

Regarding the anticipated trade-offs between the confidence slope (now referred to as multiplicative evidence bias) and metacognitive noise / delta_m (now additive evidence bias), there is a single scenario in which this becomes an issue. I describe this in the Results section as follows (Line 480ff):

“Here, the only marked trade-off emerges between metacognitive noise σm and the metacognitive evidence biases (𝜑m, δm) in the noisy-readout model, under conditions of low sensory noise. In this regime, the multiplicative evidence bias 𝜑m becomes increasingly underestimated and the additive evidence bias δm overestimated with increasing metacognitive noise. Closer inspection shows that this dependency emerges only when metacognitive noise is high – up to σm  0.3 no such dependency exists. It is thus a scenario in which there is little true variance in confidence ratings (due to low sensory noise many confidence ratings would be close to 1 in the absence of metacognitive noise), but a lot of measured variance due to high metacognitive noise. It is likely for this reason that parameter inference is problematic. Overall, except for this arguably rare scenario, all parameters of the model are highly identifiable and separable.” In my experience, certain trade-offs in specific edge cases are almost inescapable for more complex models. Overall, I think it is fair to say that parameter recovery works extremely well, including the ‘trinity’ of metacognitive noise / multiplicative evidence bias / additive evidence bias.

• Along similar lines, can the noisy readout and noisy report models really be distinguished? I appreciate they might return differential AICs. But qualitatively, it seems like the only thing distinguishing them is that the noise is either applied before or after the link function, and it wasn't clear whether this was sufficient to distinguish one from the other. In other words, if you created a 2x2 model confusion matrix from simulated data (see Wilson & Collins, 2019 eLife) would the correct model pathway from Figure 1 be recovered?

Great point. I introduced a new subsection 2.4 “Model recovery”, in which I demonstrate successful recovery of noisy-readout versus noisy-report models. See also my response to the first comment of Reviewer #1, which includes the new model recovery figure and the associated paragraph in the manuscript. The key new figure is Figure 7—figure supplement 6.

• Again on a similar theme: isn't the slope parameter rho_m better considered a parameter governing metacognitive sensitivity, given that it maps the decision values onto confidence? If this parameter approaches zero, the function flattens out which seems equivalent to introducing additional metacognitive noise. Are these parameters distinguishable?

Indeed, the parameter recovery analysis shows a slight negative correlation between the slope parameter (now termed multiplicative evidence bias) and metacognitive noise (Figure 7). As the reviewer mentions, this is likely caused by the fact that both parameters lead to a flattening /steepening of the evidenceconfidence relationship. For reference, in the empirical dataset by Shekhar & Rahnev, the correlation between AUROC2 and the multiplicative evidence bias is almost absent at r = −0.017. Critically, however, while an increase of the metacognitive noise parameter σm will ultimately lead to a truly flat/indifferent relationship between evidence and confidence, the multiplicative evidence parameter 𝜑m only affects the slope (i.e., asymptotically confidence will still reach 1). This is one reason why parameter recovery for both σm and 𝜑m works overall very well. The differential effects of σm and 𝜑m are now better illustrated in the updated Figure 3:

Also conceptually, the multiplicative evidence parameter 𝜑m plausibly represents a metacognitive bias, with either interpretation that I suggest in the manuscript: as a an under/overestimation of the evidence or as a an over/underestimation of one’s own sensory noise, leading to under/overconfidence, respectively. In sum, I think there are strong arguments for the present formalization and interpretation.

• The final paragraph of the discussion was interesting but potentially concerning for a model of metacognition. It explains that data on empirical trial-by-trial accuracy is not used in the model fits. I hadn't appreciated this until this point in the paper. I can see how in a process model that simulates decision and confidence data from stimulus features, accuracy should not be an input into such a model. But in terms of a model fit, it seems odd not to use trial by trial accuracy to constrain the fits at the metacognitive level, given that the hallmark of metacognitive sensitivity is a confidence-accuracy correlation. Is it not possible to create accuracy-conditional likelihood functions when fitting the confidence rating data (similar to how the meta-d' model fit is handled)? Psychologically, this also makes sense given that the observer typically knows their own response when giving a confidence rating.

While I agree of course that metacognitive sensitivity quantifies the relationship confidence-accuracy relationship, a process model is a distinct approach and requires distinct methodology. Briefly, the current model fit cannot be improved upon, as it is based on a precise inversion of the forward model. Computing accuracy-conditional likelihoods would lead to a biased parameter estimates, because it would incorrectly imply that the observer has access to the accuracy of their choice. While the observer knows their choice, as the reviewer correctly notes, they do not know the true stimulus category and hence not their accuracy.

I argue in the manuscript that both approaches (descriptive meta-d’, explanatory process model) have their advantages and disadvantages. The concept of meta-d’ / metacognitive sensitivity does not care why a particular confidence rating is the way it is, or whether an incorrect response is caused by sensory noise or by an attentional lapse. On the one hand, this implies that one cannot draw any conclusions about the causes and mechanisms of metacognitive inefficiency, which could be perceived as a major drawback. In this respect, it is a purely descriptive measure (cf. last comment of Reviewer #1). On the other hand, because it is descriptive, it can simply compare the confidence between correct and incorrect choices and thus, in a sense, capture a more thorough picture of metacognitive sensitivity; that is, being metacognitively aware not only of the consequences one’s own sensory noise (as in typical process models), but also of all other sources of error (attentional lapses, finger errors, etc.). I now added an additional paragraph in which I summarize the comparison of type 2 ROC / meta-d’ and process models along these lines (Line 800ff):

“In sum, while a type 2 ROC analysis, as a descriptive approach, does not allow any conclusions about the causes of metacognitive inefficiency, it is able to capture a more thorough picture of metacognitive sensitivity: that is, it quantifies metacognitive awareness not only about one’s own sensory noise, but also about other potential sources of error (attentional lapses, finger errors, etc.). While it cannot distinguish between these sources, it captures them all. On the other hand, only a process model approach will allow to draw specific conclusions about mechanisms – and pin down sources – of metacognitive inefficiency, which arguably is of major importance in many applications.”

• I found it concerning that all the variability in scale usage were being assumed to load onto evidencerelated parameters (eg delta_m) rather than being something about how subjects report or use an arbitrary confidence scale (eg the "implicit biases" assumed to govern the upper and lower bounds of the link function). It strikes me that you could have a similar notion of offset at the level of report - eg an equivalent parameter to delta_m but now applied to c and not z. Would these be distinguishable? They seem to have quite different interpretations psychologically: one is at the level of a bias in confidence formation, and the other at the level of a public report.

I substantially reworked the section about metacognitive biases, including an additive metacognitive bias (κm) also at the level of confidence. The previous version of the manuscript already included a multiplicative bias parameter loading onto confidence (previously referred to as ‘confidence scaling’ parameter, now multiplicative confidence bias λm), but it was considered optional and e.g. not part of the parameter recovery analyses.

My previous emphasis on biases that load onto evidence-related variables was due to a more principled interpretation (e.g. ‘underestimation of sensory noise’), but I agree that metacognitive biases must not necessarily be principled and may be driven e.g. by the idiosyncratic usage of a particular confidence scale. Updated Figure 1 sketches the new, more complete model.

Is a mix of evidence- and confidence-related metacognitive bias parameters distinguishable? I tested this in Figure 7—figure supplement 3.

The slope matrices show that e.g., the model suggested by the reviewer (two evidence-related bias parameters 𝜑m and δm + an additive confidence-based bias parameter κm) is to some degree dissociable, although slight tradeoffs start to emerge with such a complex model. By contrast, a mix of only one evidence-related and one confidence-related bias parameter is much more robust. In general, I thus recommend using at most two metacognitive bias parameters, which are selected either based on a priori hypotheses or on a model comparison. I comment on the necessity of choosing one’s bias parameters in a new paragraph in section 1.4 about metacognitive biases (Line 276ff):

“Finally, note that the parameter recovery shown in Figure 4 was performed with four separate models, each of which was specified with a single metacognitive bias parameter (i.e., 𝜑m, δm, λm, or m). Parameter recovery is more unreliable when more than two of these bias parameters are specified in parallel (see section 2.3; in particular, Figure 7—figure supplement 3). In practice, the researcher thus must make an informed decision about which bias parameters to include in a specific model (in most scenarios 1 or 2 metacognitive bias parameters is a good choice). While the evidence-related bias parameters 𝜑m and δm have a more principled interpretation (e.g., as an under/overestimation of sensory noise), it is not unlikely that metacognitive biases also emerge at the level of the confidence report (λm, km). The first step thus must always be a process of model specification or a statistical comparison of candidate models to determine the final specification (see also section 3.1).”

I am forcibly reminded of this orange site comment that makeworld had shared though I guess I haven't really gotten it out of my head in the first place.

Everybody knows the Game is Rigged

Not really. https://web.archive.org/web/20150828195814/http://adamspitz.com/avocado/2011/05/05/open-source-is-not-enough/

Here also would be better to have code blocks so the user can copy/paste. A link for an interactive notebook at the beginning and end would again be valuable.

Perhaps it would be better to add a link to a google colabs notebook, which shows the library in action and is interactive. Adding images is frowned upon on the technical writing community since it can become difficult to keep up to date. Code is better because you can have unit tests to make sure your examples are always working or you can at least pinpoint the version of your library and its dependencies in your interactive notebook.

loop unrolling 为什么可以提高 performance？

In documentation (not actual code):

:: -> class methods

# -> instance methods

This is misleading/hyperbolic. The performance characteristics of WASM vs JS are nothing like a native binary vs the Bash interpreter.

a constructor is used to initialize the state variables of a smart contract. constructor code is executed once when a contract is created and it is used to initialize contract state.

Reviewer #3 (Public Review):

Considerable progress has been made in moving to more open and reproducible fMRI research. However, an accessible end-to-end solution that meets these standards has remained elusive, in part because it requires the combination of many tools. Neuroscout aims to try to provide this platform. Key elements of Neuroscout include:

- An easy-to-use web application for designing the GLM analysis of naturalistic experiments;<br /> - Data ingestion server with a growing repository of naturalistic fMRI studies curated and preprocessed for these analyses;<br /> - Feature extraction server for the generation of different regressors for analyses;<br /> - Tooling for implementing these analyses;<br /> - Automated generation of citations for these analyses.

This platform has no clear precedents, is reasonably mature, is easy to use, and has an impressive number of curated datasets. With a focus on large naturalistic datasets, there should be a wide range of legitimately novel analyses that are made easily accessible with this tool, and this will increase as Neuroscout evolves to offer a wider range of datasets and functionality. A key benefit of easy-to-use platforms of this nature is that researchers gain the ability to quickly implement analyses of phenomena and hypotheses generated from their own work, accelerating science. Documentation and data and code accessibility are excellent. The existing analysis examples are interesting, accessible to users, and generally provide good insight into the use and value of the platform for general users.

A weakness of many automated systems of this nature is that users rapidly find limitations in the types of analyses that can be set up. In the worst cases, this leaves the platform providing largely a demonstration. However, here, the well-developed open-science components make this unlikely. The authors have strong records in developing widely used open software for fMRI, and the considerable number of datasets and feature-generation algorithms that have been integrated into the platform already bodes well for uptake. Nevertheless, while described as end-to-end, the current scope for analysis design is somewhat limited, restricted largely to the specification of the GLM design. Furthermore, it was not clear if or how the platform might scale and develop an active community of data, algorithm, and code contributors. Similarly, choices of preprocessing algorithms are not extensively motivated, and how these might evolve with input from a wider community is unclear.

Overall this is a promising tool that develops upon a burgeoning set of open-science tools for functional neuroimaging and presents new strategies for how fMRI analysis can be made more accessible and reproducible. While a software tool's success is ultimately measured by its uptake, Neuroscout presents a successful implementation of a concept that may provide researchers with or without extensive experience of fMRI the ability to efficiently implement novel analyses to a high standard. If Neuroscout is to be a success, it would be expected to evolve considerably from its current state. Determining how to balance the flexibility of the tool with ease of use will be an ongoing challenge.

Reviewer #3 (Public Review):

Sender & Bar-On et al. perform robust analyses of early SARS-CoV-2 line list data from China to estimate the intrinsic generation interval in the absence of interventions. This is an important topic, as most SARS-CoV-2 data are from periods when transmission-reducing interventions are in place, which will lead to underestimation of the potential infectious period.

The authors highlight two shortcomings in previous approaches. First, the distribution of 'observed' serial intervals (the time between symptom onset in the infector and symptom onset in the infectee) depends not only on the timeline of each infector's infection, but also the epidemic growth rate, which weights the proportion of observed short vs. long serial intervals. The authors argue that by accounting for this weighting, more accurate estimates of the intrinsic generation interval - the metric on which isolation policies are based - can be obtained. Second, the authors find that the original SARS-CoV-2 generation interval distribution has both a higher mean and longer tail than previous estimates when using only data prior to the introduction of interventions. Finally, the authors use publicly available data on viral load trajectories to extrapolate their estimates to other SARS-CoV-2 variants, finding that alpha, delta, and omicron may have shorter generation intervals than original SARS-CoV-2. These findings are important, as case isolation policies are based on assumptions for how long individuals remain infectious. More broadly, these methods will be important for future work to correctly estimate generation intervals in other outbreaks.

The conclusions are well supported by the data, and a suite of sensitivity analyses give confidence that the findings are robust to deviations from many of the key assumptions. The code is well documented and publicly available, and thus the findings are easily reproducible. Key strengths of the paper include the clarity and rigor of the modeling methods, and the exhaustive consideration of potential biases and corresponding sensitivity analyses - it is very difficult to think of potential biases that the authors have not already considered! I think this is a well-written and well-executed study. The work is likely to be impactful for reconsidering SARS-CoV-2 isolation policies and revisiting generation interval estimates from other data sources. I also expect this to be a key reference and method for future studies estimating the generation interval.

I have some minor comments on potential weaknesses and interpretation:

1. Uncertainty in early generation interval estimates<br /> One of the conclusions is that the estimated mean generation interval is longer than the observed mean serial interval. However, this conclusion does not seem justified given that the observed mean serial interval (9.1 days) is well within the 95% CI of 8.3-11.2 days for the mean generation interval. The confidence intervals for the serial interval in figure 2 are also wide for pre-Jan 17th (though presumably these would be reduced if all pre-Jan 17th serial intervals were combined). Further, only 77 of the ~1000 transmission pairs are actually from pre-January 17th. The actual sample size used for these estimates is much smaller than suggested by Figure S1 and thus this should be made clear. Therefore, although the intuition for why observed serial intervals may differ from the generation interval is correct, I do not think that the data alone demonstrate this.

A related issue is on ascertainment bias - could the early serial interval data be biased longer because ascertainment is initially poor and thus more intermediate infectors are missed? The authors consider removing particularly long serial intervals to try and account for this, but that does not deal with e.g. chains of multiple short serial intervals being incorrectly recorded as a single long serial interval (but still within 16 days).

2. Frailty of using viral loads to extrapolate generation intervals<br /> The authors take the observation that variants of concern demonstrate faster viral clearance on average to estimate shorter generation intervals for alpha, delta, and omicron. The authors rightly point out in the discussion that using viral load as a proxy for infectiousness has many limitations. I would emphasize even further that it is very difficult to extrapolate from viral load data in this way, as infectiousness appears to vary far more between variants than can be explained by duration positive or peak viral load. Other factors are potentially at play, such as compartmentalization in the respiratory tract, aerosolization, receptor binding, immunity, etc. Further, there is considerable individual-level variation in viral trajectories and thus the use of a population-mean model overlooks a key component of SARS-CoV-2 infection dynamics. An important reference, which came out recently and thus makes sense to have been missed from the initial submission, is Puhach et al. Nature Medicine 2022 https://doi.org/10.1038/s41591-022-01816-0.

3. Lack of validation with other datasets<br /> This study hinges on data from a single setting in a short window of time. Although the data are from multiple publications, the fact that so many reported the same transmission pair data demonstrates that these are overlapping datasets. As the authors note, there are potential biases e.g., ascertainment rates and behavioral changes which will impact the generation interval estimates. Thus, generalizability to other settings is limited.

4. The impact of epidemic dynamics on infector vs. infectee serial intervals<br /> It took me a long time to get my head around the assertion that the forward serial interval distribution will be longer during epidemic growth due to the overrepresentation of short incubation periods among infectors relative to infectees. A supplementary figure, similar to the way Figure 1 is laid out, to illustrate this concept may go a long way to aid the reader's understanding.

5. Simulations to illustrate concepts and power<br /> Given the assertion that observed serial intervals will depend on epidemic growth rates, reporting, and timing of interventions, I think a simple simulation to illustrate some of these ideas would be very helpful. For example, a simple agent-based model with simulated infectivity profiles and incubation periods using the estimated bivariate distribution would be extremely helpful in illustrating how serial intervals and estimates of the generation interval can differ from the true intrinsic generation interval (I coded such a simulation to help me understand this paper in a couple of hours with <100 lines of R code, so I do not think this would be much work). This would also be very helpful for illustrating statistical power re. comment 1.

什么意思？WebView还要浏览器里的库？不是WebView是独立的吗？

代码一样，只不过Google闭源的多了其他功能，不然难道你还一个功能写两份不同的代码。

Reviewer #3 (Public Review):

Zadbood and colleagues investigated the way key information used to update interpretations of events alter patterns of activity in the brain. This was cleverly done by the use of "The Sixth Sense," a film featuring a famous "twist ending," which fundamentally alters the way the events in the film are understood. Participants were assigned to three groups: (1) a Spoiled group, in which the twist was revealed at the outset, (2) a Twist group, who experienced the film as normal, and (3) a No-Twist group, in which the twist was removed. Participants were scanned while watching the movie and while performing cued recall of specific scenes. Verbal recall was scored based on recall success, and evidence for descriptive bias toward two ways of understanding the events (specifically, whether a particular character was or was not a ghost). Importantly, this allowed the authors to show that the Twist group updated their interpretation. The authors focused on regions of the Default Mode Network (DMN) based on prior studies showing responsiveness to naturalistic memory paradigms in these areas and analyzed the fMRI data using intersubject pattern similarity analysis. Regions of the DMN carried patterns indicative of story interpretation. That is, encoding similarity was greater between the Twist and No-Twist groups than in the Spoiled group, and retrieval similarity was greater between the Twist and Spoiled groups than in the No-Twist group. The Spoiled group also showed greater pattern similarity with the Twist group's recall than the No-Twist group's recall. The authors also report a weaker effect of greater pattern similarity between the Spoiled group's encoding and the Twist group's recall than between the Twist group's own encoding and recall. Together, the data all converge on the point that one's interpretation of an event is an important determinant of the way it is represented in the brain.

This is a really nice experiment, with straightforward predictions and analyses that support the claims being made. The results build directly on a prior study by this research group showing how interpretational differences in a narrative drive distinct neural representations (Yeshurun et al., 2017), but extend an understanding of how these interpretational differences might work retrospectively. I do not have any serious concerns or problems with the manuscript, the data, or the analyses. However I have a few points to raise that, if addressed, would make for a stronger paper in my opinion.

1) My most substantive comment is that I did not find the interpretive framework to be very clear with respect to the brain regions involved. The basic effects the authors report strongly support their claims, but the particular contributions to the field might be stronger if the interpretations could be made more strongly or more specifically. In other words: the DMN is involved in updating interpretations, but how should we now think about the role of the DMN and its constituent regions as a result of this study? There are a number of ideas briefly presented about what the DMN might be doing, but it just did not feel very coherent at times. I will break this down into a few more specific points:

While many of us would agree that the DMN is likely to be involved in the phenomena at hand, I did not find that the paper communicated the logic for singularly focusing on this subset of regions very compellingly. The authors note a few studies whose main results are found in DMN regions, but I think that this could stand to be unpacked in a more theoretically interesting way in the Introduction.

Relatedly, I found the summary/description of regional effects in the Discussion to be a bit unsatisfying. The various pattern similarity comparisons yielded results that were actually quite nonoverlapping among DMN regions, which was not really unpacked. To be clear, it is not a 'problem' that the regional effects varied from comparison to comparison, but I do think that a more theoretical exploration of what this could mean would strengthen the paper. To the authors' credit, they describe mPFC effects through the lens of schemas, but this stands in contrast to many other regions which do not receive much consideration.

Finally, although there is evidence that regions of the DMN act in a coordinated way under some circumstances, there is also ample evidence for distinct regional contributions to cognitive processes, memory being just one of them (e.g., Cooper & Ritchey, 2020; Robin & Moscovitch, 2017; Ranganath & Ritchey, 2012). The authors themselves introduce the idea of temporal receptive windows in a cortical hierarchy, and while DMN regions do appear to show slower temporal drift than sensory areas, those studies show regional differences in pattern stability across time even within DMN regions. Simply put, it is worth considering whether it is ideal to treat the DMN as a singular unit.

2) I think that some direct comparison to regions outside the DMN would speak to whether the DMN is truly unique in carrying the key representations being discussed here. I was reluctant to suggest this because I think that the authors are justified in expecting that DMN regions would show the effects in question. However, there really is no "null" comparison here wherein a set of regions not expected to show these effects (e.g., a somatosensory network, or the frontoparietal network) in fact do not show them. There are not really controls or key differences being hypothesized across different conditions or regions. Rather, we have a set of regions that may or may not show pattern similarity differences to varying degrees, which feels very exploratory. The inclusion of some principled control comparisons, etc. would bolster these findings. The authors do include a whole-brain analysis in Supplementary Figure 1, which indeed produced many DMN regions. However, notably, regions outside the DMN such as the primary visual cortex and mid-cingulate cortex appear to show significant effects (which, based on the color bar, might actually be stronger than effects seen in the DMN). Given the specificity of the language in the paper in terms of the DMN, I think that some direct regional or network-level comparison is needed.

3) If I understand correctly, the main analyses of the fMRI data were limited to across-group comparisons of "critical scenes" that were maximally affected by the twist at the end of the movie. In other words, the analyses focused on the scenes whose interpretation hinged on the "doctor" versus "ghost" interpretation. I would be interested in seeing a comparison of "critical" scenes directly against scenes where the interpretation did not change with the twist. This "critical" versus "non-critical" contrast would be a strong confirmatory analysis that could further bolster the authors' claims, but on the other hand, it would be interesting to know whether the overall story interpretation led to any differences in neural patterns assigned to scenes that would not be expected to depend on differences in interpretation. (As a final note, such a comparison might provide additional analytical leverage for exploring the effect described in Figure 3B, which did not survive correction for multiple comparisons.)

4) I appreciate the code being made available and that the neuroimaging data will be made available soon. I would also appreciate it if the authors made the movie stimulus and behavioral data available. The movie stimulus itself is of interest because it was edited down, and it would be nice for readers to be able to see which scenes were included.

To sum up, I think that this is a great experiment with a lot of strengths. The design is fairly clean (especially for a movie stimulus), the analyses are well reasoned, and the data are clear. The only weaknesses I would suggest addressing are with regards to how the DMN is being described and evaluated, and the communication of how this work informs the field on a theoretical level.

Reasons given for not intending to use technology for lecture capture in future:

• personal choice
• lack of interaction
• better technologies available than institutional LC system, (e.g. Kaltura)

And yet this might be glibly and repeatedly recommended by some in v6ops as a way to fix the ULA brokenness problem.

Reviewer #1 (Public Review):

In this manuscript, the authors present a new technique for analysing low complexity regions (LCRs) in proteins- extended stretches of amino acids made up from a small number of distinct residue types. They validate their new approach against a single protein, compare this technique to existing methods, and go on to apply this to the proteomes of several model systems. In this work, they aim to show links between specific LCRs and biological function and subcellular location, and then study conservation in LCRs amongst higher species.

The new method presented is straightforward and clearly described, generating comparable results with existing techniques. The technique can be easily applied to new problems and the authors have made code available.

This paper is less successful in drawing links between their results and the importance biologically. The introduction does not clearly position this work in the context of previous literature, using relatively specialised technical terms without defining them, and leaving the reader unclear about how the results have advanced the field. In terms of their results, the authors further propose interesting links between LCRs and function. However, their analyses for these most exciting results rely heavily on UMAP visualisation and the use of tests with apparently small effect sizes. This is a weakness throughout the paper and reduces the support for strong conclusions.

Additionally, whilst the experimental work is interesting and concerns LCRs, it does not clearly fit into the rest of the body of work focused as it is on a single protein and the importance of its LCRs. It arguably serves as a validation of the method, but if that is the author's intention it needs to be made more clearly as it appears orthogonal to the overall drive of the paper.

Overall I think the ideas presented in the work are interesting, the method is sound, but the data does not clearly support the drawing of strong conclusions. The weakness in the conclusions and the poor description of the wider background lead me to question the impact of this work on the broader field.

Technical weaknesses

In the testing of the dotplot based method, the manuscript presents a FDR rate based on a comparison between real proteome data and a null proteome. This is a sensible approach, but their choice of a uniform random distribution would be expected to mislead. This is because if the distribution is non-uniform, stretches of the most frequent amino will occur more frequently than in the uniform distribution.

More generally I think the results presented suggest that the results dotplot generates are comparable to existing methods, not better and the text would be more accurate if this conclusion was clearer, in the absence of an additional set of data that could be used as a "ground truth".

The authors draw links between protein localisation/function and LCR content. This is done through the use of UMAP visualisation and wilcoxon rank sum tests on the amino acid frequency in different localisations. This is convincing in the case of ECM data, but the arguments are substantially less clear for other localisations/functions. The UMAP graphics show generally that the specific functions are sparsely spread. Moreover when considering the sample size (in the context of the whole proteome) the p-value threshold obscures what appear to be relatively small effect sizes.

Author Response

Reviewer #1 (Public Review):

The authors sought to create a machine learning framework for analyzing video recordings of animal behavior, which is both efficient and runs in an unsupervised fashion. The authors construct Selfee from recent computational neural network codes. As the paper is methodsfocused, the key metrics for success would be (1) whether Selfee performs similarly or more accurately than existing methods, and more importantly (2) whether Selfee uncovers new behavioral features or dynamics otherwise missed by those existing methods.

Weaknesses:

Although the basic schematics of Selfee are laid out, and the code itself is available, I feel that material in between these two levels of description is somewhat lacking. Details of what other previously published machine learning code makes up Selfee, and how those parts work would be helpful. Some of this is in the methods section, but an expanded version aimed at a more general readership would be helpful.

Thanks for the suggestions. We expanded the paragraphs describing training objectives and AR-HMM analysis. We also revised Figure 2C for clarity, and we have added a new figure, Figure 6, to describe how our pipeline works in detail. We also added a detailed instructions for Selfee usage on our GitHub page.

*The paper highlights efficiency as an important aspect of machine learning analysis techniques in the introduction, but there is little follow up with this aspect.

Our model only had a more efficient training process compared with other self-supervised learning methods. We also found our model could perform zero-shot domain transfer, so training may not even be necessary. However, we did not mean that our model was superior in terms of data efficiency or inference speed. We have revised some of the claims in the Discussion.

*In comparing Selfee to other approaches, the paper uses DeepLabCut, but perhaps running other recent methods for more comprehensive comparison would be helpful as well.

We compare Selfee feature extraction with features from FlyTracker or JAABA, two widely used software. We also visualized the tracking results of SLEAP and FlyTracker in complement to the DeepLabCut experiment.

*Using Selfee to investigate courtship behavior and other interactions was nicely demonstrated. Running it on simpler data (say, videos of individual animals walking around or exploring a confined space) might more broadly establish the method's usefulness.

We used Selfee with open field test (OFT) of mice after chronic immobilization stress (CIS) treatment. We demonstrated that our pipeline from data preprocessing to all the data mining algorisms with this experiment, and the results were added to the last section of Results.

Reviewer #2 (Public Review):

Jia et al. present a CNN based tool named "Selfee" for unsupervised quantification of animal behavior that could be used for objectively analyzing animal behavior recorded in relatively simple setups commonly used by various neurobiology/ethology laboratories. This work is very relevant but has some serious unresolved issues for establishing credibility of the method.

Overall Strengths: Jia et al have leveraged a recent development "Simple Siamese CNNs" to work for behavioral segmentation. This is a terrific effort and theoretically very attractive.

Overall Weakness: Unfortunately, the data supporting the method is not as promising. It is also riddled with incomplete information and lack of rationale behind the experiments.

Specific points of concern:

1) No formal comparison with pre-existing methods like JAABA which would work on similar videos as Selfee.

We added some comparisons with JAABA and FlyTracker extracted features, and also visualized FlyTracker and SLEAP tracking results aside from DeepLabCut. This result is now in the new Table 1. To avoid tracking inaccuracy during intensive interactions and potential inappropriately tuned parameters, we used a peer-reviewed dataset focused on wing extension behavior only. Our results showed a competitive performance of Selfee as other methods.

2) For all Drosophila behavior experiments, I'm concerned about the control and test genetic background. Several studies have reported that social behaviors like courtship and aggression are highly visual and sensitive to genetic background and presence of "white" gene. The authors use Canton S (CS) flies as control data. Whereas it is unclear if any or all of the test genotypes have been crossed into this background. It would be helpful if authors provide genotype information for test flies.

We have added a detailed sheet about their genotype in this version. The genetic information of all animals can also be found on the Bloomington fly center by the IDs provided. In brief, five fly lines used in this work are in the CS background: CCHa2-R-RAGal4, CCHa2-R-RBGal4, Dop2RKO, DopEcRGal4 and Tdc2RO54. We did not back cross other flies into the CS background for three reasons. First, most mutant lines are compared with their appropriate control lines. For example, in the original Figure 3B (the new Figure 4B), for CCHa2-R-RBGal4 > Kir2.1 flies contained wildtype white gene, so the comparison with CS flies would not cause any problem. For TrhGal4 flies, they were in white background, and so were other lines that had no phenotype. At the same time, in the original Figure 3G to J (the new Figure 4G to J), we used w1118 as controls for TrhGal4 flies, which were all in mutated white background. Second, in the original Figure 4F and G (the new Figure 5F and G), we admitted that the comparison between NorpA36, in mutated white background, and CS flies was not very convincing. Nevertheless, the delayed dynamic of NorpA mutants was reported before, and our experiment was just a demonstration of the DTW algorithm. Lastly, our method focused on the methodology of animal behavior analysis, and original videos were provided for research replications. Therefore, even if the behavioral difference was due to genetic backgrounds, it would not affect the conclusion that our method could detect the difference

3) Utility of "anomaly score" rests on Fig 3 data. Authors write they screened "neurotransmitter-related mutants or neuron silenced lines" (lines 251-252). Yet Figure 3B lacks some of the most commonly occurring neurotransmitter mutants/neuron labeling lines (e.g. Acetelcholine, GABA, Dopamaine, instead there are some neurotransmitter receptor lines, but then again prominent ones are missing). This reduces the credibility of this data.

First of all, this paper did not intend to conduct new screening assays, rather we used pre-existed data in the lab to demonstrate the application of Selfee. Previous work in our lab focused on the homeostatic control of fly behaviors, so most listed lines used here were originally used to test the roles of neuropeptides or neurons nutrient and metabolism regulation, such as CCHarelated lines, a CNMa mutant, and Taotie neuron silenced flies. There were some other important genes that were not involved in this dataset. Some most common transmitters are not included for two reasons. First, common neurotransmitters usually have a very global and broad effect on animal behaviors, and even if there is any new discovery, it could be difficult to interpret the phenomenon due to a large number of disturbed neurons. Second, most mutants of those common neurotransmitters are not viable, for example, paleGal4 as a mutant for dopamine; Gad1A30 for GABA, and ChATl3 for acetylcholine. However, we did perform experiments on serotonin-related genes (SerT and Trh), octopamine-related genes (Tdc and Oamb), and some other viable dopamine receptor mutants.

4) The utility of AR-HMM following "Selfee" analysis rests on the IR76b mutant experiment (Fig4). This is the most perplexing experiment! There are so many receptors implicated in courtship and IR76b is definitely not among the most well-known. None of the citations for IR76b in this manuscript have anything to do with detection of female pheromones. IR76b is implicated in salt and amino acid sensation. The authors still call this "an extensively studies (co)receptor that is known to detect female pheromones" (lines310-311). Unsurprisingly the AR-HMM analysis doesn't find any difference in modules related to courtship. Unless I'm mistaken the premise for this experiment is wrong and hence not much weight should be given to its results.

We have removed the Ir76b results from the Results. The demonstration of AR-HMM was now done with a mouse open field assay.

Reviewer #3 (Public Review):

This paper is describing a machine learning method applied to videos of animals. The method requires very little pre-processing (end-to-end) such as image segmentation or background subtraction. The input images have three channels, mapping temporal information (liveframes). The architecture is based on tween deep neural networks (Siamese network) and does not require human annotated labels (unsupervised learning). However, labels can still be used if they are produced, as in this case, by the algorithm itself - self-supervised learning. This flavor of machine learning is reflected in the name of the method: "Selfee." The authors are convincingly applying the Selfee to several challenging animal behavior tasks which results in biologically relevant discoveries.

A significant advantage of unsupervised and self-supervised learning is twofold: 1) it allows for discovering new behaviors, and 2) it doesn't require human-produced labels.

In this case of self-supervised learning the features (meta-representations) are learned from two views of the same original image (live-frame), where one of the views is augmented in several different ways, with a hope to let the deep neural network (ResNet-50 architecture in this case) learn to ignore such augmentations, i.e. learn the meta-representations invariant to natural changes in the data similar to the augmentations. This is accomplished by utilizing a Siamese Convolutional Neural Network (CNN) with the ResNet-50 version as a backbone. Siamese networks are composed of tween deep nets, where each member of the pair is trying to predict the output of another. In applications such as face recognition they normally work in the supervised learning setting, by utilizing "triplets" containing "negative samples." These are the labels.

However, in the self-supervised setting, which "Selfee" is implementing, the negative samples are not required. Instead the same image (a positive sample) is viewed twice, as described above. Here the authors use the SimSiam core architecture described by Chen, X. & He, K (reference 29 in the paper). They add Cross-Level Discrimination (CLD) to the SimSiam core. Together these two components provide two Loss functions (Loss 1 and Loss 2). Both are critical for the extraction of useful features. In fact, removing the CLD causes major deterioration of the classification performance (Figure 2-figure supplement 5).

The authors demonstrate the utility of the Selfee by using the learned features (metarepresentations) for classification (supervised learning; with human annotation), discovering short-lasting new behaviors in flies by anomaly detection, long time-scale dynamics by ARHMM, and Dynamic Time Warping (DTW).

For the classification the authors use k-NN (flies) and LightGBM (mice) classifiers and they infer the labels from the Selfee embedding (for each frame), and the temporal context, using the time-windows of 21 frames and 81 frames, for k-NN classification and LightGBM classification, respectively. Accounting for the temporal context is especially important in mice (LightGBM classification) so the authors add additional windowed features, including frequency information. This is a neat approach. They quantify the classification performance by confusion matrices and compute the F1 for each.

Overall, I find these classification results compelling, but one general concern is the criticality of the CLD component for achieving any meaningful classification. I would suggest that the authors discuss in more depth why this component is so critical for the extraction of features (used in supervised classification) and compare their SimSiam architecture to other methods where the CLD component is implemented. In other words, to what degree is the SimSiam implementation an overkill? Could a simpler (and thus faster) method be used - with the CLD component - instead to achieve similar end-to-end classification? The answer would help illuminate the importance of the SimSiam architecture in Selfee.

We added more about the contribution of the CLD loss in the last paragraph of Siamese convolutional neural networks capture discriminative representations of animal posture, the second section of Results. Further optimization of neural network architectures was discussed in the Discussion section. As for why CLD is that important, there are two main reasons. First of all, all behavior photos are so similar that it is not very easy to distinguish them from each other. In the field of so-called self-supervised learning without negative samples, researchers use either batch normalization or similar operations to implicitly utilize negative samples within a minibatch. However, when all samples are quite similar, it might not be enough. CLD uses explicit clusters to utilize negative samples within a minibatch, in the word of the authors “Our key insight is that grouping could result from not just attraction, but also common repulsion”, so that provides more powerful discrimination. The second reason is what the author argued in the CLD paper, CLD is very powerful in processing long-tailed datasets. As shown in the original Figure 2—figure supplement 5 (the new Figure 3—figure supplement 5), behavior data are highly unbalanced. As explained in the CLD paper. CLD fights against long-tailed distribution from two aspects. One is that it scales up the importance of negative samples within a mini-batch from 1/B to 1/K by k-means; another is that cluster operation could relieve the imbalance between the tail and head classes within a mini-batch. Here I quote: “While the distribution of instances in a random mini-batch is long-tailed, it would be more flattened across classes after clustering.” It was also visualized in Fig5 of the CLD paper.

To the best of our knowledge, SimSiam is the simplest method that would work with CLD. In the original CLD paper, they combined CLD method with other popular frameworks including BYOL and Mocov2. However, those popular frameworks are more complicated than SimSiam networks. We have attempted to combine CLD with BarlowTwins but failed. As the author of CLD suggested on Github: “Hi, good to know that you are trying to combine CLD with BarLowTwins! My concern is also on the high feature dimension, which may cause the low clustering quality. Maybe it is necessary to have a projection layer to project the highdimensional feature space to a low-dimensional one.” In terms of speed, there are two major parts. For inference, only one branch is used, so the major contribution of efficiency comes from CNN backbone. In theory, light backbones like MobileNet would work, but ResNet50 is already fast enough on a model GPU. As for training, the major computational cost aside from the CNN backbone is from Siamese branches. Two branches, two times of computation. Nevertheless, CLD relied on this kind of structure, so even if the learning framework is simpler than Simsiam, it is not likely to achieve a faster training speed. As for other structures, I think this new instance learning framework (https://arxiv.org/abs/2201.10728) is possible to achieve a similar result with fewer data and in a shorter time. However, this powerful method could be used with CLD. We might try it in the future.

One potential issue with unsupervised/self-supervised learning is that it "discovers" new classes based, not on behavioral features but rather on some other, irrelevant, properties of the video, e.g. proximity to the edges, a particular camera angle, or a distortion. In supervised learning the algorithm learns the features that are invariant to such properties, because humanmade labels are used and humans are great at finding these invariant features. The authors do mention a potential limitation, related to this issue, in the Discussion ("mode splitting"). One way of getting around this issue, other than providing negative samples, is to use a very homogeneous environment (so that only invariance to orientation, translation, etc, needs to be accomplished). This has worked nicely, for example, with posture embedding (Berman, G. J., et al; reference 19 in the manuscript). Looking at the t-SNE plots in Figure 2 one must wonder how many of the "clusters" present there are the result of such learning of irrelevant (for behavior) features, i.e. how good is the generalization of the meta-representations. The authors should explore the behaviors found in different parts of the t-SNE maps and evaluate the effect of the irrelevant features on their distributions. For example, they may ask: to what extent does the distance of an animal from the nearest wall affect the position in the t-SNE map? It would be nice to see how various simple pre-processing steps might affect the t-SNE maps, as well as the classification performance. Some form of segmentation, even very crude, or simply background subtraction, could go a very long way towards improving the features learned by Selfee.

In the new Figure 3—figure supplement 1, the visualization demonstrates that our features contained a lot of physical information, including wing angles, animal distance and positions in the chamber. “Mode-split” can be partially explained by those features. We actually performed background subtraction and image crop for mice behaviors, where we found them useful.

The anomaly detection is used to find unusual short-lasting events during male-male interaction behavior (Figure 3). The method is explained clearly. The results show how Selfee discovered a mutant line with a particularly high anomaly score. The authors managed to identify this behavior as "brief tussle behavior mixed with copulation attempts." The anomaly detection analyses were also applied to discover another unusual phenotype (close body contact) in another mutant line. Both results are significant when compared to the control groups.

The authors then apply AR-HMM and DTW to study the time dynamics of courtship behavior. Here too, they discover two phenotypes with unusual courtship dynamics, one in an olfactory mutant, and another in flies where the mutation affects visual transduction. Both results are compelling.

The authors explain their usage of DTW clearly, but they should expand the description of the AR-HMM so that the reader doesn't have to study the original sources.

We expanded the section that talks about AR-HMM mechanisms.

Reviewer #1 (Public Review):

The authors sought to create a machine learning framework for analyzing video recordings of animal behavior, which is both efficient and runs in an unsupervised fashion. The authors construct Selfee from recent computational neural network codes. As the paper is methods-focused, the key metrics for success would be (1) whether Selfee performs similarly or more accurately than existing methods, and more importantly (2) whether Selfee uncovers new behavioral features or dynamics otherwise missed by those existing methods.

Strengths:<br /> * The authors put their work in context very well, discussing machine learning approaches to behavior extraction generally, and clearly stating the unique aspects of their own approach. The schematic framework of Selfee is nicely described.<br /> * The authors use their new methods on existing data sets, mostly in adult Drosophila but also in rodents, with the resulting outputs confirming and accurately classifying known behaviors, in agreement with manual annotation.<br /> * The analysis focuses on behavior video that depicts interactions between animals, typically more difficult than either individual animal video or video with noninteracting animals. This adds to the strength of the method.<br /> * Experiments with mutants and Kir-silenced lines were nicely designed, and highlighted Selfee's anomaly detection methods by finding a short-time-scale behavior unlikely to be noticed by manual human observation.<br /> * Similarly, experiments investigating Trh in flies were very thorough and detailed, and illustrate the effectiveness of the machine learning analysis when combined with follow up experiments to investigate Selfee's initial findings.

Weaknesses:<br /> * Although the basic schematics of Selfee are laid out, and the code itself is available, I feel that material in between these two levels of description is somewhat lacking. Details of what other previously published machine learning code makes up Selfee, and how those parts work would be helpful. Some of this is in the methods section, but an expanded version aimed at a more general readership would be helpful.<br /> * The paper highlights efficiency as an important aspect of machine learning analysis techniques in the introduction, but there is little follow up with this aspect.<br /> * In comparing Selfee to other approaches, the paper uses DeepLabCut, but perhaps running other recent methods for more comprehensive comparison would be helpful as well.<br /> * Using Selfee to investigate courtship behavior and other interactions was nicely demonstrated. Running it on simpler data (say, videos of individual animals walking around or exploring a confined space) might more broadly establish the method's usefulness.

Overall, the results of the paper seem to clearly achieve what was set out in the introduction, which was to use an unsupervised machine learning video analysis method to uncover new features of behavior. The experiments establishing the effectiveness seem very sound and reasonable.

For a reader who does not work directly with implementing machine learning, the paper is highly readable and interesting and should generate interest to a wider audience of researchers who would wish to try Selfee on their own data. The paper could have made it a bit more clear how an inexperienced user might deploy the Selfee software, whether with one of the model systems used here or a different one. But there is certainly something very appealing about an unsupervised method, which has the potential to be more accessible to a wider audience of researchers, allowing more people to take advantage of sophisticated behavior analysis.

Reviewer #3 (Public Review):

In this manuscript, Wang et al describe a series of experiments aimed at optimizing the experimental and computational approach to the detection of projection-specific neurons across the entire mouse brain. This work builds on a large body of work that has developed nuclear-fused viral labelling, next-generation fluorophores, tissue clearing, image registration, and automated cell segmentation. They apply their techniques to understand projection-specific patterns of supraspinal neurons to the cervical and lumbar spinal cord, and to reveal brain and brainstem connections that are preferentially spared or lost after spinal cord injury.

Strengths:

Although this work does not put forward any fundamentally new methodologies, their careful optimization of the experimental and quantification process will be appreciated by other laboratories attempting to use these types of methods. Moreover, the observations of topological arrangement of various supraspinal centres are important and I believe will be interesting to others in the field.

The web app provided by the authors provides a nice interface for users to explore these data. I think this will be appreciated by people in the field interested in what happens to their brain or brainstem region of interest.

Weaknesses:

Overall the work is well done; however, some of the novelty claims should be better aligned with the experimental findings. Moreover, the statistical approaches put forward to understand the relationship between spinal cord injury severity and cell counts across the mouse brain needs to be more carefully considered.

The authors state that they provide an experimental platform for these types of analysis to be done. My apologies if I missed it but I could not find anywhere the information on viral construct availability or code availability to reproduce the results. Certainly both of these aspects would be required for people to replicate the pipeline. Moreover, the described methodology for imaging and processing is quite sparse. While I appreciate that this information is widely provided in papers that have developed these methods, I do not think it is appropriate to claim to have provided a platform for people to enable these types of analyses without a more in-depth description of the methods. Alternatively, the authors could instead focus on how they optimized current methodologies and avoid the overstatement that this work provides a tool for users. The exception to this is of course the viral constructs, the plasmids of which should be deposited.

It was not completely to me clear why or when the authors switch back and forth between different resolutions throughout the manuscript. In the abstract it states that 60 regions were examined, but elsewhere the number is as many as 500. My understanding is that current versions of the Allen Brain Annotation include more than 2000 regions. I think it would make things clear for the readers if a single resolution was used throughout, or at least justified narratively throughout the text to avoid confusion.

The others provide an interesting analysis of the difference between cervical and lumbar projections. I think this might be one of the more interesting aspects of the paper - yet I found myself a bit confused by the analysis, and whether any of the differences observed were robust. Just prior to this experiment the authors provide a comparison of the mScarlet vs. the mGL, and demonstrate that mGL may label more cells. Yet, in the cervical vs. lumbar analysis it appears they are being treated 1 to 1. Moreover, I could not find any actual statistical analysis of this data? My impression would be that given the potential difference in labelling efficiency between the mScarlet and mGL this should be done using some kind of count analysis that takes into account the overall number of neurons labelled, such as a Chi-sq test or perhaps something more sophisticated. Then, with this kind of statistical analysis in place, do any of the discussed differences hold up? If not, I do not think this would detract from the interesting topological observations - but would call on the authors to be a bit more conservative about their statements and discussion regarding differences in the proportions of neurons projecting to certain supraspinal centres.

Finally, I do have some concerns about the author's use of linear regression in their analysis of brain regions after varying severities of SCI. First of all, the BMS score is notoriously non-linear. Despite wide use of linear regressions in the field to attempt to associate various outcomes to these kinds of ordinal measures, this is not appropriate. Some have suggested a rank conversion of the BMS prior to linear analyses, but even this comes with its own problems. Ultimately, the authors have here 2-3 clear cohorts of behavioural scores and drawing a linear regression between these is unlikely to be robustly informative. Moreover, it is unclear whether the authors properly adjusted their p-values from running these regressions on 60 (600?) regions. Finally, the statement in the abstract and discussion that the authors "explain more variability" compared to typical lesion severity analysis is also unsupported. My suggestion would be the following:

Remove the linear regression analyses associated with BMS. I do not think these add value to the paper, and if anything provide a large window of false interpretation due to a violation of the assumptions of this test.

Consider adding a more appropriate statistical analysis of the brain regions, such as a non-parametric group analysis. Knowing which brain regions are severity dependent, and which ones are not, would already be an interesting finding. This finding would not be confounded by any attempt to link it to crude measures of behaviour.

If the authors would like to state anything about 'explaining more variability' then the proper statistical analysis should be used, which in this case would be to compare the models using a LRT or equivalent. However, as I mentioned it does not seem to be appropriate to be doing this with linear models so the authors should consider a non-linear equivalent if they choose to proceed with this.

Author Response

Reviewer #2 (Public Review):

The main strength of the paper is the parallel profiling of virus-specific CD4 T cells in different stages of acute and persistent infection, and the ease of publicly accessing the data and source code. These data extend previous studies, such as Khatun et al. JEM 2020 and Cicucci et al. Immunity 2019, by revealing single-cell transcriptome information on virus-specific CD4 T cells at different stages of infection.

The main drawback is the paper's advertised use as a 'comprehensive atlas of virus-specific CD4 T cells'. This study includes virus-specific T cells from a single organ (spleen) during infections with two clones of a single virus (LCMV). Therefore, its use as a reference atlas does not extend to other viruses or T cells from organs other than spleen during LCMV infection. If such samples were integrated with the splenic LCMV atlas, either new unique populations would be found and therefore not meaningfully annotated or they would be force-integrated with one of the splenic subsets, producing a potentially misleading and crude annotation. In this sense, the authors did not construct an atlas but rather a dataset on LCMV-specific splenic CD4 T cells which, like other datasets, can be compared with other single-cell sequencing datasets.

The methodology description does not include convincing evidence that the integration was successful in minimizing batch effects and retaining biological heterogeneity, virtually no data is presented in support of this point. Therefore, the scope of the work should be refined and the methodology significantly improved before this paper becomes acceptable for publication.

We thank the referee for recognizing the strengths of the study, as well as for advising where we had been insufficiently clear in describing the methodology – in particular with regards to data integration and generalizability of our bioinformatics tool. As detailed below, we provide new evidence supporting the quality of data integration, the robustness and replicability of the T cell states defined in our reference map, and its ability to make accurate predictions across multiple tissues (spleen, liver, lung, lymph nodes) and beyond the LCMV infection model. In addition, we conducted several additional analyses demonstrating the robustness of our predictions, and generated a new scRNA-seq dataset of tumor-specific CD4+ T cells, showing how our LCMV-derived reference map can help identify and characterize a novel cell state uniquely acquired by tumor-infiltrating CD4+ T cells.

The machine code - and the disassemble command that allows the introspection of it - is fascinating! We should be able to introspect every aspect of our computers in this fashion.

overall, great great work!! i like how you are usually various array methods such as includes, indexOf. Good that you are using resources outside of the course material. Also good that you extract out most of the code into various helper functions which makes the main code much more readable.

https://gist.github.com/VasylTyshchuk/62adab009eeff3b9e043d88ccb74ef82

Added this code snippet to my /Templates/scripts folder in Obsidian.

To use, copy a book URL from Goodreads, then run the Goodreads template and it will paste the URL from the clipboard to return the metadata of the book into the current note as frontmatter

Overall, very good job. logic is clear and good use of comments. code is readable and consistent naming convention.And also meaningful naming of variables and functions :)

i like the use of switch case statement. makes code cleaner and more readable!! usually i would assign variables e.g. var SCISSORS = 'scissors' to reduce likelihood of making error since u would be referencing a few times following in the switch case statement

SciScore for 10.1101/2022.06.02.22275900: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

NIH rigor criteria are not applicable to paper type.

Table 2: Resources

No key resources detected.

Results from OddPub: Thank you for sharing your code and data.

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a protocol registration statement.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.01.22275778: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

No key resources detected.

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

Strengths and Limitations: The key strength of this study is the use of a territory-wide, random sampling method. Together with a high response rate, we have captured a representative sample of school-aged children and adolescents in Hong Kong. Furthermore, we kept our sampling methodology consistent before and during COVID, making our data from the two periods comparable. The study has several limitations: First, this study was cross-sectional by nature and not longitudinal. Second, the sleep parameters were derived only from self-report or parent-report, without objective measures such as actigraphy. Third, our data did not capture daytime activities, limiting our interpretation of mediating factors of changes.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.01.22275882: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

Despite these promising findings, there are limitations that need to be acknowledged. The sample size of this study is relatively small due to the difficulty in sampling patients longitudinally in a hospital setting. Antibody kinetics (levels and isotype) during the early infection could be influenced by time since infection. Also, the control samples consist of out-patient populations that had no recorded history of respiratory infection during sample collection. Having control samples from patients who are SARS-CoV-2 negative but positive for other respiratory diseases will further strengthen the reliability of pGSN multi-analyte panels. In future studies, patient sample size will be increased while we investigate the prognostic significance of pGSN multi-analyte panel in SARS-CoV-2 variant patients, vaccinated patients as well as convalescent plasma.

Results from TrialIdentifier: We found the following clinical trial numbers in your paper:<br><table><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">Identifier</td><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">Status</td><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">Title</td></tr><tr><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">NCT04358406</td><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">Active, not recruiting</td><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">Rhu-pGSN for Severe Covid-19 Pneumonia</td></tr></table>

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a protocol registration statement.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.01.494385: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

Our study has several caveats, including the use of pseudoviruses instead of authentic SARS-CoV-2 for conducting experiments. However, our findings using pseudoviruses agree with those reported using authentic SARS-CoV-2. For instance, authentic BA.1 /BA.1.1 VOCs were shown to undergo attenuated replication in TMPRSS2-expressing cells compared to ancestral Wuhan-Hu-1, and Alpha, Beta, and Delta VOCs 6,36. These reports also showed greater sensitivity of BA.1 pseudovirus entry to endosomal inhibitor E64d. While we used pseudovirus entry assays to determine Omicron variant usage of ACE2 receptors of various animal species, it remains unknown whether there may be intrinsic and/or innate host-specific factors that might act to inhibit live Omicron VOCs at an entry or post entry step. Furthermore, although we identified RBM substitutions in Omicron spike that conferred the ability to use mouse or horseshoe bat ACE2, we didn’t confirm ACE2 substitutions that permit or prevent Omicron spike binding. For instance, introducing K35E substitution in horseshoe bat ACE2 should permit Omicron variants’ usage. Finally, analysis of a limited number of serum samples and short follow up after the receipt of three doses of the Pfizer/BNT162b2 mRNA vaccine do not give us insights into the durability of the antibody response. While studies of antibody durability are ongoing, our findings indicate that three dose immunization with the Pfizer/BNT162b2 will likely contribute to protection from sever...

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• No funding statement was detected.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.05.31.22274922: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

No key resources detected.

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

Methodological strengths and limitation outlined in our previous 2020-study also apply for the current study. Among the limitations of the first study was the small sample size, therefore we went from 600 to 1000 participants to strengthen the power of the present study. Compared to our first study, we also shortened the exposure period inquired about, aiming to provide more specific estimates of associations. The use of the Danish Vaccination Registry enabled us to swiftly and objectively exclude those who had been vaccinated by the time of the study. A potential bias would arise from systematic differences in behavior between cases and controls. Some persons who recently had been in close contact with a person with known infection, would likely have been in isolation and therefore not exposed in the community. Because we frequently found controls to be more exposed than cases (resulting in OR estimates below 1), we were suspicious of such a bias being at play. To explore this further, we performed a sensitivity analysis, in which we excluded all participants who reported to have been close contacts to infected persons. However, this did not change the results. Another potential concern relates to the selection of controls. We used matched controls sampled from the general population, which was made possible because of our access to the Danish Civil Registration System. A different possibility, which we did not opt for, would have been control selection with recruitment from...

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.02.494502: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.01.22275674: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: Thank you for sharing your code and data.

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

Strengths and weaknesses: The key strengths are the scale and completeness of the underlying raw EHR data, available close to real time, and we engaged with clinicians for added context. All processed data and analytical code is openly available in the Supplementary Materials or Github. We will publish our recommended key measures in a live updating report, and we encourage other groups to use OpenSafely for further exploration. Our data-driven approach is intended to generate an overall picture of primary care clinical activity, and explore high volume areas that might otherwise be missed, for example when not included in manually curated codelists. Despite the strengths we recognise some limitations as previously discussed.11 Our data-driven approach and filtering processes may have omitted some relevant codes; codes do not necessarily indicate unique or new events, and may be affected by changes in coding behaviour. All coded activity for patients registered at the end of the study period were included, and all activity was included under their latest practice. Patients who died or deregistered from TPP practices during the study period were not included. Overall, activity counts were up to 6-8% lower than database totals in the earliest months of the study period. Interpretation and context for each clinical area: Given the diversity of clinical areas covered by this overarching analysis, the clinical advisory group evaluated and interpreted the variation for each clinica...

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

|- gloss : TrailMarks

|- snippet : high resolution addressing in TrailMark - |- for TrailMarks - |- example : clues in markdown - nestrd

earlier used the generic trailmark 'for' to indicate that the current annotation is relevant for the designated/named topic given as the target/object/subject for the TrailMark for.

SciScore for 10.1101/2022.06.01.22275889: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

No key resources detected.

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.02.22275894: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

NIH rigor criteria are not applicable to paper type.

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.02.22275895: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

NIH rigor criteria are not applicable to paper type.

Table 2: Resources

No key resources detected.

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

Our study has some limitations. First, the external test set was sourced from a single institution. It is critical to test the AI algorithm across multiple datasets distributed across geographical regions to ensure that the model’s results are generalizable across cohorts and geographies. Second, we utilized RT-PCR as the gold standard for the diagnosis of COVID-19 infections. However, RT-PCR has a limited sensitivity of approximately 71%, so there may be cases where the person is COVID-19 positive on chest radiographs but negative on RT-PCR results. Third, our study does not incorporate clinical parameters and does not attempt at categorising patients based on COVID-19 severity scores. We avoided providing COVID-19 scores based on chest radiographs as unlike chest CT scans, there is often interobserver disagreement on the extent of lung involvement in radiography for reasons encompassing different acquisition protocols, image quality, and radiologist opinion. The development of an AI system based on the consensus scoring of a few radiologists from an isolated geographical location may not represent the consensus of radiologists globally. However, some researchers did attempt to build such a system, like the one developed by Borghesi & Maroldi (22) on a small dataset of 100 patients. Attempts were also made by Monaco et al. (23) with the dataset of 295 patients and Orsi et al. (24) with the dataset of 155 patients to produce scoring systems for chest radiographs and link them...

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a protocol registration statement.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.05.31.22275835: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: Thank you for sharing your code and data.

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

In general, reported deaths represent an undercount of total COVID-19 deaths, given limitations in testing and data.46 In addition, there is considerable uncertainty associated with some state variant proportions due to insufficient sequencing. Our model also uses variant proportion figures based on clinical testing which may lag behind actual variant circulation. For example, wastewater surveillance suggests that Omicron was circulating in the country before the first case was reported.47 This would also lead to underestimates of deaths caused by variants. Finally, we do not model disease transmission dynamics so our estimates should not be interpreted as the number of deaths that would have been prevented if novel variants did not emerge. Even in the absence of new variants, the non-variant SARS-CoV-2 would have continued to spread, resulting in additional deaths. SARS-CoV-2 variants detected around the world have imposed a significant mortality burden in the U.S. In addition to national public health strategies, greater efforts are needed to reduce the risk of new variants emerging globally. However, U.S. government officials have warned that they are running out of COVID-19 funds.48 Congressional lawmakers recently excluded $5 billion in funding for global treatment and vaccination campaigns from a new spending proposal, and there has been limited investment in next-generation vaccines that may offer greater protection against transmission.49, 50 Our analysis shows the si...

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.01.494461: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• No conflict of interest statement was detected. If there are no conflicts, we encourage authors to explicit state so.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.05.25.22275564: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

However, there are still some limitations in this study. First, a cut-off Ct value of 35 was adopted, however, some cellular experiments have now confirmed that an excreted viral load in the environment with Ct values of less than 35 does not result in effective infection. The relationship between a Ct value of 35 and infectivity needs to be supported by more genuine clinical data. Second, although all patients enrolled in this study had asymptomatic infections and mild disease, the inconsistent test results between sputum sample and pharyngeal swab cannot be ruled out as being partially related to differences in patient disease status and distribution of the virus. Third, sputum samples are relatively more accurate in the determination of Ct values, but they are obtained by induced nebulization. Whether nebulization represents the normal state of virus excretion and its relationship with infectivity needs to be further investigated. Nevertheless, the large sample of cases with the same exposure period and similar clinical characteristics and the analysis of multiple synchronous triple-sample collections of pharyngeal swabs, sputum samples, and anal swabs showed that the combined sputum sample detection has an important role in determining the criteria for release from quarantine.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a protocol registration statement.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.05.31.22275010: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

This is a small study with several limitations as a consequence, but the trends are confirmation of other observations. Several studies have shown that heterologous booster vaccination can heighten and broaden the immune response compared with homologous booster doses [17–19]. We restricted this study to one priming vaccine, ChAdOx1-S, but results need to be confirmed with other vaccines, particularly mRNA and inactivated vaccines. We only assessed the immune responses out to four weeks after the booster vaccination, and persistence of any improved immune responses following the heterologous and homologous boosters will have to be assessed. Finally, we did not assess the efficacy of the booster immunization; although there were several cases of COVID-19 reported in this small study population it was not designed to include an efficacy assessment which would also require a placebo group. Notably, none of these cases were severe and there were no hospitalizations due to COVID-19. In conclusion, the formulation of 30 μg SCB-2019 adjuvanted with CpG-1018 and alum is safe and well tolerated and as a heterologous booster vaccine in those previously primed with ChAdOx1-S and is immunologically more effective than that same vaccine given as a homologous booster.

Results from TrialIdentifier: We found the following clinical trial numbers in your paper:<br><table><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">Identifier</td><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">Status</td><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">Title</td></tr><tr><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">NCT05087368</td><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">Not yet recruiting</td><td style="min-width:95px; border-right:1px solid lightgray; border-bottom:1px solid lightgray">Immunogenicity and Safety of Heterologous and Homologous Boo…</td></tr></table>

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a protocol registration statement.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.01.22275878: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: Thank you for sharing your code and data.

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.01.22275775: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

No key resources detected.

Results from OddPub: Thank you for sharing your code and data.

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

The findings presented in this study should be interpreted with several limitations in mind. First, more than half of our data came from the UK which may have obscured patterns from other countries. Relatedly, for the analysis of COVID-19 vaccination in countries other than the UK, we grouped 18 countries together due to data availability. This approach neglected variations in vaccination schedules and the evolution of vaccine policies on pregnancy in individual countries. Both risk perceptions to COVID-19 and vaccine hesitancy also likely differ across countries and change over time. Future research is highly warranted as more data become available. Second, we did not collect data on gravidity and gestational age from pregnant women, which likely have important relationships with perceived need of social support and vaccine acceptance. Future studies should explore factors associated with subgroups of pregnant women requiring particular attention. Third, pregnant and non-pregnant individuals may have interpreted the questions on isolation and quarantine due to COVID-19 differently. Terminologies such as “isolation”, “quarantine” and “social distancing” are distinctively different yet somewhat similar notions that might be misinterpreted, especially for high-risk people who have been given stricter guidelines to maintain social distancing as a form of preventive measure. Our finding on isolation and quarantine might have over-estimated the proportion of pregnant women who wer...

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.02.22275901: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

This is discordant to our MR results where higher genetically-predicted iron status is related to increased risk of sepsis and being hospitalized due to COVID-19, and could be attributed to differences in the epidemiological methods applied, such as residual confounding, but also limitations with the two-sample MR method used that is restricted to assess linear models (37). Few MR studies have explored iron status and risk of severe infections. An MR-study using iron related SNPs identified in the Genetics of Iron Status-consortia (38) found evidence that higher serum-iron, TSAT and ferritin were related to increased risk of sepsis (21). Using a more updated set of genetic instruments for iron status biomarkers, we replicated these findings for serum iron and TSAT, a tendency for TIBC, but not for ferritin. Another MR study found evidence of increased risk of skin and soft-tissue infections with higher serum iron levels (39). To the best of our knowledge, no previous study has conducted MR analysis to investigate the effect of iron status on incidence or outcome of COVID-19. Observational studies that have investigated iron status at the time of infection and found evidence of low iron status being a risk factor for a severe course of COVID-19 (12). Another study with COVID-19 patients compared to non-COVID-19 patients showed lower serum iron and TSAT levels in patients with COVID-19 independently of severity. Whereas COVID-19 patients defined as severe and critical had subst...

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.01.494373: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

NIH rigor criteria are not applicable to paper type.

Table 2: Resources

Results from OddPub: Thank you for sharing your code.

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

Sensitivity limitations of PhIP-seq to broadly detected polio epitopes have been previously reported (20, 22) and might contribute to the observed differences, similarly affecting detectability of CoV antigens. Such limitations are not observed in our HDPA approach (44) which typically yielded strong polio responsiveness in over 90% of sampled individuals (45). Importantly, the cross-reactivity in identified B cell epitope sites positively relates to previous infections with seasonal common cold hCoVs. This suggests that immune memory conferred by previous seasonal hCoV infections positively influences SARS-CoV-2-specific antibody responses and may explain the large portion of SARS-CoV-2-infected individuals with mild and asymptomatic disease symptoms (4). Notably, there was little to no correlation between cross-reactivity and immune response in COVID-19 negative patients, suggesting that resistance to infection is not easily explained by cross reactivity. However, this molecular cross-reactivity can pose important complications in serological tests, particularly when studying asymptomatic patients. Cross-reactivity in immunodominant epitopes can be molecular determinants of strong immunity in individuals and therefore may serve as the basis for future pan-coronavirus vaccine design strategies. In turn, mutations in these cross-reactive epitopes can potentially breach pre-existing immune protection conferred by previous viral exposures, contributing to viral evolution, immun...

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• No funding statement was detected.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.05.31.22275831: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

No key resources detected.

Results from OddPub: Thank you for sharing your code and data.

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

He wanted to keep in touch with Mrs. Das so badly and was afraid of messing something up. I wonder what's going to come out of this.

SciScore for 10.1101/2022.05.31.22274501: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

No key resources detected.

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• No funding statement was detected.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.05.31.494262: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• No conflict of interest statement was detected. If there are no conflicts, we encourage authors to explicit state so.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.05.31.494211: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.05.31.494147: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.05.31.493843: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• No funding statement was detected.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.06.01.22275786: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Table 2: Resources

No key resources detected.

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.05.31.493925: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

NIH rigor criteria are not applicable to paper type.

Table 2: Resources

Results from OddPub: Thank you for sharing your code.

Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

• Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
• No funding statement was detected.
• No protocol registration statement was detected.

Results from scite Reference Check: We found no unreliable references.

About SciScore

SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

SciScore for 10.1101/2022.05.31.22275746: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

NIH rigor criteria are not applicable to paper type.

Table 2: Resources

No key resources detected.

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

Strengths and limitations: The study used well-established methods to obtain data and had excellent completion rates with the app. Self-report using visual analogue scales on a smartphone app gives participants control over the data they enter and reduces risks of bias when completing scales either at a later time or with a researcher [45]. Importantly, there was patient involvement in the design of the app and in the interpretation of the results. Analysis used a combination of idiographic (within individual) and nomothetic (between individuals) methods including state of the art graphical vector autoregression modelling [41, 46] . The main limitation is that the sample was largely white, female, middle-aged and well-educated. This reflects the opportunistic sample taken from an online panel of research volunteers promoted by peer-support groups, however this has been seen in other studies [8]. This also meant that additional clinical data obtained during routine or specialised care was not available to supplement the data generated in the study. Approximately one third of participants had their initial illness before the widespread availability of PCR testing for SARS-Cov-2 and another third reported that their PCR test had been negative. This raises the possibility that not all participants’ symptoms were sequelae of covid infection [47] however we were unable to test serology in this study nor check records for prior symptoms. Although the number of participants (N=74) wa...

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent: