Reviewer #1 (Public Review):

The manuscript "Diffusive lensing as a mechanism of intracellular transport and compartmentalization", explores the implications of heterogeneous viscosity on the diffusive dynamics of particles. The authors analyze three different scenarios:

(i) diffusion under a gradient of viscosity,

(ii) clustering of interacting particles in a viscosity gradient, and

(iii) diffusive dynamics of non-interacting particles with circular patches of heterogeneous viscous medium.

The implications of a heterogeneous environment on phase separation and reaction kinetics in cells are under-explored. This makes the general theme of this manuscript very relevant and interesting. However, the analysis in the manuscript is not rigorous, and the claims in the abstract are not supported by the analysis in the main text.

Following are my main comments on the work presented in this manuscript:

(a) The central theme of this work is that spatially varying viscosity leads to position-dependent diffusion constant. This, for an overdamped Langevin dynamics with Gaussian white noise, leads to the well-known issue of the interpretation of the noise term. The authors use the Ito interpretation of the noise term because their system is non-equilibrium.

One of the main criticisms I have is on this central point. The issue of interpretation arises only when there are ill-posed stochastic dynamics that do not have the relevant timescales required to analyze the noise term properly. Hence, if the authors want to start with an ill-posed equation it should be mentioned at the start. At least the Langevin dynamics considered should be explicitly mentioned in the main text. Since this work claims to be relevant to biological systems, it is also of significance to highlight the motivation for using the ill-posed equation rather than a well-posed equation. The authors refer to the non-equilibrium nature of the dynamics but it is not mentioned what non-equilibrium dynamics to authors have in mind. To properly analyze an overdamped Langevin dynamics a clear source of integrated timescales must be provided. As an example, one can write the dynamics as<br /> Eq. (1) \dot x = f(x) + g(x) \eta , which is ill-defined if the noise \eta is delta correlated in time but well-defined when \eta is exponentially correlated in time. One can of course look at the limit in which the exponential correlation goes to a delta correlation which leads to Eq. (1) interpreted in Stratonovich convention. The choice to use the Ito convention for Eq. (1) in this case is not justified.

(b) Generally, the manuscript talks of viscosity gradient but the equations deal with diffusion which is a combination of viscosity, temperature, particle size, and particle-medium interaction. There is no clear motivation provided for focus on viscosity (cytoplasm as such is a complex fluid) instead of just saying position-dependent diffusion constant. Maybe authors should use viscosity only when talking of a context where the existence of a viscosity gradient is established either in a real experiment or in a thought experiment.

(c) The section "Viscophoresis drives particle accumulation" seems to not have new results. Fig. 1 verifies the numerical code used to obtain the results in the later sections. If that is the case maybe this section can be moved to supplementary or at least it should be clearly stated that this is to establish the correctness of the simulation method. It would also be nice to comment a bit more on the choice of simulation methods with changing hopping sizes instead of, for example, numerically solving stochastic ODE.

A minor comment, the statement "the physically appropriate convention to use depends upon microscopic parameters and timescale hierarchies not captured in a coarse-grained model of diffusion." is not true as is noted in the references that authors mention, a correct coarse-grained model provides a suitable convention (see also Phys. Rev. E, 70(3), 036120., Phys. Rev. E, 100(6), 062602.).

(d) The section "Interaction-mediated clustering is affected by viscophoresis" makes an interesting statement about the positioning of clusters by a viscous gradient. As a theoretical calculation, the interplay between position-dependent diffusivity and phase separation is indeed interesting, but the problem needs more analysis than that offered in this manuscript. Just a plot showing clustering with and without a gradient of diffusion does not give enough insight into the interplay between density-dependent diffusion and position-dependent diffusion. A phase plot that somehow shows the relative contribution of the two effects would have been nice. Also, it should be emphasized in the main text that the inter-particle interaction is through a density-dependent diffusion constant and not a conservative coupling by an interaction potential.

(e) The section "In silico microrheology shows that viscophoresis manifests as anomalous diffusion" the authors show that the MSD with and without spatial heterogeneity is different. This is not a surprise - as the underlying equations are different the MSD should be different. There are various analogies drawn in this section without any justification:<br /> (i) "the saturation MSD was higher than what was seen in the homogeneous diffusion scenario possibly due to particles robustly populating the bulk milieu followed by directed motion into the viscous zone (similar to that of a Brownian ratchet, (Peskin et al., 1993))."<br /> (ii) "Note that lensing may cause particle displacements to deviate from a Gaussian distribution, which could explain anomalous behaviors observed both in our simulations and in experiments in cells (Parry et al., 2014)."<br /> Since the full trajectory of the particles is available, it can be analyzed to check if this is indeed the case.

(f) The final section "In silico FRAP in a heterogeneously viscous environment ... " studies the MSD of the particles in a medium with heterogeneous viscous patches which I find the most novel section of the work. As with the section on inter-particle interaction, this needs further analysis.

To summarise, as this is a theory paper, just showing MSD or in silico FRAP data is not sufficient. Unlike experiments where one is trying to understand the systems, here one has full access to the dynamics either analytically or in simulation. So just stating that the MSD in heterogeneous and homogeneous environments are not the same is not sufficient. With further analysis, this work can be of theoretical interest. Finally, just as a matter of personal taste, I am not in favor of the analogy with optical lensing. I don't see the connection.

Reviewer #1 (Public Review):

Summary:<br /> Pulfer et al., describe the development and testing of a transformer-based deep learning architecture called ADeS, which the authors use to identify apoptotic events in cultured cells and live animals. The classifier is trained on large datasets and provides robust classification accuracies in test sets that are comparable to and even outperform existing deep learning architectures for apoptosis detection. Following this validation, the authors also design use cases for their technique both in vitro and in vivo, demonstrating the value of ADeS to the apoptosis research space.

Strengths:<br /> ADeS is a powerful tool in the arsenal of cell biologists interested in the spatio-temporal co-ordinates of apoptotic events in vitro, since live cell imaging typically generates densely packed fields of view that are challenging to parse by manual inspection. The authors also integrate ADeS into the analysis of data generated using different types of fluorescent markers in a variety of cell types and imaging modalities, which increases its adaptability by a larger number of researchers. ADeS is an example of the successful deployment of activity recognition (AR) in the automated bioimage analysis space, highlighting the potential benefits of AR to quantifying other intra- and intercellular processes observable using live cell imaging.

Weaknesses:<br /> A major drawback was the lack of access to the ADeS platform for the reviewers; the authors state that the code is available in the code availability section, which is missing from the current version of the manuscript. This prevented an evaluation of the usability of ADeS as a resource for other researchers. The authors also emphasize the need for label-free apoptotic cell detection in both their abstract and their introduction but have not demonstrated the performance of ADeS in a true label-free environment where the cells do not express any fluorescent markers. While Pulfer et al., provide a wealth of information about the generation and validation of their DL classifier for in vitro movies, and the utility of ADeS is obvious in identifying apoptotic events among FOVs containing ~1700 cells, the evidence is not as strong for in vivo use cases. They mention the technical challenges involved in identifying apoptotic events in vivo, and use 3D rotation to generate a larger dataset from their original acquisitions. However, it is not clear how this strategy would provide a suitable training dataset for understanding the duration of apoptotic events in vivo since the temporal information remains the same. The authors also provide examples of in vivo acquisitions in their paper, where the cell density appears to be quite low, questioning the need for automated apoptotic detection in those situations. In the use cases for in vivo apoptotic detection using ADeS (Fig 8), it appears that the location of the apoptotic event itself was obvious and did not need ADeS, as in the case of laser ablation in the spleen and the sparse distribution of GFP labeled neutrophils in the lymph nodes. Finally, the authors also mention that video quality altered the sensitivity of ADeS in vivo (Fig 6L) but fail to provide an example of ADeS implementation on a video of poor quality, which would be useful for end users to assess whether to adopt ADeS for their own live cell movies.

eLife assessment

This manuscript presents a valuable approach to exploring CD4+ T-cell response in mice across stimuli and tissues through the analysis of their T-cell receptor repertoires. The authors use a transgenic mouse model, in which the possible diversity of the T-cell receptor repertoire is reduced, such that each of a diverse set of immune exposures elicits more detectably consistent T-cell responses across different individuals. However, whereas the proposed experimental system could be utilized to study convergent T-cell responses, the analyses done in this manuscript are incomplete and do not support the claims due to limitations in the statistical analyses and lack of data/code access.

Reviewer #3 (Public Review):

Nakonechnaya et al present a valuable and comprehensive exploration of CD4+ T cell response in mice across stimuli and tissues through the analysis of their TCR-alpha repertoires.

The authors compare repertoires by looking at the relative overlap of shared clonotypes and observe that they sometimes cluster by tissue and sometimes by stimulus. They also compare different CD4+ subsets (conventional and Tregs) and find distinct yet convergent responses with occasional plasticity across subsets for some stimuli.

The observed lack of a general behaviour highlights the need for careful comparison of immune repertoires across cell subsets and tissues in order to better understand their role in the adaptive immune response.

In conclusion, this is an important paper to the community as it suggests several future directions of exploration.

Unfortunately, the lack of code and data availability does not allow the reproducibility of the results.

We need to add link to error code mapping

MRPC(The Microsoft Research Paraphrase Corpus，微软研究院释义语料库)，相似性和释义任务，是一个判断0、1的任务

Searched - Vehicle code

[!NOTE] Git 中，credential.helper <val> 的行为具体是怎么样的？

flashcard

• git credential-<val>
• <val=!...> -> eval ...

easiest way is to just download the azure-file-samples using the green Code button and then extract the azfileshybrid from there

Note: This rebuttal was posted by the corresponding author to Review Commons. Content has not been altered except for formatting.

Learn more at Review Commons

Reply to the reviewers

We would like to thank all reviewers for their highly valuable comments, reviewing the article, and suggesting changes to improve its overall structure, clarity, and comprehension. Please find below our point-by-point responses to each reviewer’s comments. Lines, figures, and tables we refer to in the responses correspond to the clean copy of the revised manuscript.

Reviewer #1:

1) In the introduction the authors list 4 databases of ultra-conserved non-coding elements, and choose to work with the UCNEbase that detects UCNE regions by comparing human and chicken, as they state that it is one of the most comprehensive resources. Can the authors comment on the extent of overlap of UCNE regions identified in this database compared to other databases. It would also be helpful to add an explanation in the Introduction on the advantage of comparing the human genome to the chicken genome, e.g., is chicken the most distant vertebrate with a high quality genome, or another reason? And can the authors comment on the extent of conservation of the UCNEs compared to other species - in the UCNEbase paper, orthologues for the UCNEs are identified in 18 vertebrate species including reptiles, amphibians and fish.

We have addressed these comments and incorporated them in the introduction (lines 46-47; 49-52). As stated, it is not straightforward to evaluate to which extent UCNE regions overlap with those collected in other databases due to the different scopes and methods of these resources. We clarified why we selected this database, which was precisely based on the comments mentioned by the reviewer, i.e., sufficient evolutionary distance to identify functional regions confidently and high quality genome assemblies. Regarding the extent to which UCNEs are conserved in other species, the UCNEdatabase indeed provides additional information with respect to UCNE orthologues in other vertebrate species, including reptiles, amphibians, and fish. As we consider that this comparison is beyond the scope of the present study, it was not included in the main manuscript. However, to address this interesting point raised by the reviewer, we evaluated the proportion of UCNEs found in different species with respect to those annotated for human-chicken. As show in the figure below, UCNEs are conserved to a larger extent up to Xenopus tropicalis, for which most of the UCNEs annotated for human-chicken have corresponding orthologues in this species. Although to a minor extent, UCNEs are also conserved across more distant species (e.g., fish), for which approximately half of the UCNEs annotated for human-chicken have orthologues.

2) The authors briefly describe how potential target genes were assigned to UCNEs in the Method section (section begins on page 21, line 407 and on lines 379-381). They note that they used the Genomic Regions Enrichment of Annotations Tool (GREAT). Can the authors provide additional details on what this method does and also provide a high level sentence in the Results section on page 8, lines 144-146, on how target genes were assigned to UCNEs, as well as in the Discussion on page 16, line 289. Based on the Methods description on lines 409-414, a UCNE was associated with any gene within 1Mb of the UNCE that was expressed in retina and whose curated regulatory domains overlapped the UNCE. Is this correct? How were the regulatory domains curated (line 411-412)? Can the authors please clarify this important point.

Additional details about the GREAT algorithm have been included in the Methods section (lines 410-424) as well as a high-level sentences in the Results (lines 140-141) and Discussion (lines 276-280). It is correct that a UCNE was associated with any gene within 1Mb of the UCNE that was expressed in retina and whose curated regulatory domains overlapped the UCNE. As stated now (lines 420-424), these regulatory domains are supported by experimental evidence demonstrating that a gene is directly regulated by an element located beyond of its putative regulatory domain. We specified these domains for the utilized version of GREAT as well.

3) There are other approaches for linking putative transcriptionally active genomic regions with target genes in specific tissues or cell types through correlation analysis of ATAC-seq peaks (chromatin accessibility) and gene expression in RNA-seq data. A commonly used peak-to-gene linkage method is implemented in ArchR (Granja et al, 2021, PMID: 33633365). The authors note that they used scATAC-seq gene scores and scRNA-seq gene expression to further characterize the proposed target genes of UCNEs. It is not clear what the authors mean by "scATAC-seq gene scores". Please define "scATAC-seq gene score" and add a reference. Can the authors compare the target gene mapping to UCNEs using GREAT and gene expression filtering to that using peak-to-gene linkage based on retina tissue or single cell ATAC-seq and RNA-seq data?

We have defined explicitly what scATAC-seq gene scores are in the Methods section (lines 436-437) along with its reference. We have also addressed this important point and compared the overlap between the set of target genes predicted by GREAT and those assigned by the peak-to-gene linkage method implemented by ArchR. Details of this analysis, its results, interpretation are included in the Methods (lines 441-446), Results (lines 158-163; Supplementary Table 4), and Discussion (lines 280-282) sections.

4) For gene expression filtering (line 419) the authors quantify transcript expression of retina from FASTQ samples using the Kallisto method, and then note that they did the filtering on gene expression levels (TPM<0.5). Please add details on how you went from transcript expression levels to gene expression levels for the filtering, or was the filtering performed at the transcript level?

We have added details on how transcript-level quantification estimates were summarized at gene level, for which the filtering was performed (lines 429-430).

5) The authors use the words "active UCNE", first mentioned in the Results on line 144. Can the authors define what they mean by "active UCNE". What information/evidence is used to ascertain that a UCNE is indeed active. Overlap of a UCNE with a chromatin accessibility region from ATAC-seq or DNAase-Seq would only suggest that the UCNE may be active. Intersection with enhancer activity measured with in vivo enhancer reporter assays in transgenic mice from the VISTA enhancer browser provides stronger evidence of transcriptional activity. The authors might want to distinguish between putatively actively and active based on the functional support.

We thank the reviewer for this relevant comment to address the nuances of defining active UCNEs. The reviewer’s assumptions are correct and hence these terms were clarified throughout the entire text. The term functional is now only used when referring to UCNEs for which there is functional support (e.g., PAX6-associated UCNE in line 193) .

6) The authors assessed the significance of depletion of common variation (MAF>1%) in the UCNE regions compared to a background of randomly selected genomic regions. In generating the random distribution of regions, did the authors match on the distribution of distances of the UNCEs to the TSS of genes in the randomly selected regions? This may be a confounder. Also, in the legend of Figure 3, lines 191-192, it is stated: "Variant population frequencies within putative retinal UCNEs normalized to a background composed of randomly selected sequences (see Methods).", but we did not find a description of this analysis in the Methods section.

Evaluating the potential confounding effect of the genomic background was indeed a very important point. We have now incorporated the details showing the suitability of such background well as a detail description of how such background was generated (lines 479-481; Figure S1). Additionally, to further support our analysis demonstrating the depletion of common variation within UCNEs, we have included an evaluation of the distribution of genome-wide residual variation intolerance score (gwRVIS) values (PMID: 33686085) compared to this background of randomly selected genomic regions in a human reference cohort (lines 173-178; 487-495; Fig. 3C).

7) In regards to intersecting UCNEs with epigenetic marks that detect active or repressed enhancers in retina, the authors used data from Aldiri et al 2017 that measured epigenetic changes during retinal development. Did this dataset contain epigenetic measurements in adult retina? The authors might want to consider using the epigenetic marks/ChIP-seq data from adult human retina in Cherry et al. PNAS 2020 (PMID: 32265282)

We have incorporated the adult-stage data suggested by the reviewer to provide a more comprehensive characterization. Details about the integration of this dataset as well as the results and their corresponding interpretation have been incorporated in the Methods (lines 372-374), Results (lines 115-117; Supplementary Table 2), and Discussion (lines 271-273) sections accordingly.

8) With respect to the examination of rare variants that may be associated with rare eye disease in retina active UCNEs, for the interpretation of the results, it would be helpful to get more information on the distribution of rare variants found in UCNEs associated in this study to known IRD genes in all affected individuals in families, if this information is available in the 100,000 Genomes Project.

Although it is indeed a relevant point, this information cannot be retrieved in the 100,000 Genomes Project. As it is a restricted research environment, we are only allowed to query sequencing data corresponding to participants enrolled within the framework of our specific sub-domain, namely “Hearing and Sight”, and therefore evaluating the distribution of rare variants in all affected individuals is not feasible.

9) In the Methods section on lines 450-451, the authors mention that they performed variant screening of retinal disease genes, referencing the Genomics England Retinal Disorder panel and Martin et al., 2019. Can the authors add to the Methods and Results sections how many retinal disease genes were initially tested. Also, to get a sense of the specificity of the overlap of rare variants in the 100k Genome Project cohort with UNCE regions, it would be informative to show a distribution of the number of rare variants <0.5% that passed the filtering in gnomAD per eye disease gene before the overlap with UNCEs.

We specified the number of retinal genes that were tested in the Method section (line 471). In addition, as suggested by the reviewer, we generated allele frequency distributions for all variants retrieved within a selection of 25 disease-gene associated loci and their corresponding UCNEs in order to assess the specificity of the overlap between rare variants and UCNE regions (lines 181-182, 496-501; Figure S2).

10) The authors found "an ultrarare SNV (chr11:31968001T>C) within a candidate cis-regulatory UCNE located ~150kb upstream of PAX6. This variant was found in four individuals of a family segregating autosomal dominant foveal abnormalities". They tested the functional effect of this element with a reporter assay in zebrafish and found that the UNCE affects expression in the eye, forebrain, and neural tube. It would add further value if the authors were to test the effect of this SNV in the UCNE on the reporter expression pattern, using CRISPR/cas9?

That is a very relevant point. We have tested the effect of this SNV in the UCNE on the reported expression pattern using the same experimental setup that we used for testing the wild-type construct, namely transgenic enhancer zebrafish assays. However, we could not obtain conclusive results, most likely due to the limitations posed by testing these regions outside their native genomic context. Therefore, additional experimental work (e.g., CRISPR-based) should be performed to address this question. This is, however, beyond the scope of the present study, for which the main focus was the identification and functional annotation of ultraconserved cCREs. We have incorporated the details, results, and interpretation of the assays performed mutant construct in the Methods (lines 525-527; Supplementary Table 12), Results (lines 235-238; Supplementary Table 10), and Discussion (lines 350-353) sections.

11) The authors found rare variants in UCNEs linked to 45 IRD genes. Can the authors provide additional information on the functional genomic annotations of these UCNEs and distance to the target genes. The UCNEs were characterized with respect to their genic features in the original paper (UCNEbase, Dimitrieva et al., NAR, 2013), e.g., intergenic, intronic and 3'/5' UTR. Also, it would be useful for clinical applications to provide the start and end positions of the UCNEs that contain the rare variants associated with their 45 IRD genes in Supplementary Table 6.

Additional functional genomic annotations, genic features following those of the original UCNE paper, and the distance to the TSS of these 45 target disease-associated genes have been incorporated in (new) Supplementary Table 5. The start and end positions of the UCNEs that contain the rare variants have also been indicated in new Supplementary Table 7.

12) A total of 724 target genes were assigned to 1,487 UCNEs that displayed candidate cis-regulatory activity. Given the interest in using UCNEs to help identify potential pathogenic mutations that lead to IRDs, can the authors note in the Results section how many of the 724 target genes are IRDs.

We thank the reviewer for this important point. From the total of 724, a total of 27 genes are annotated as IRD genes, of which (interestingly) 23 were kept as found to be expressed in the retina. This has been clarified in the Results section (line 166-168).

13) In the Discussion on page 15 line 259, can the authors clarify if variation found in UCNEs were only associated with rare disease or also with common diseases.

We have clarified that variation found in UCNEs has only been associated with rare diseases (line 247).

Minor edits:

1) In abstract, the authors might consider changing the words "rare eye diseases" on lines 20 and 22 to "rare retina degeneration diseases", and on lines 88-89.

We thank the reviewer for this comment. However, we consider that rare eye diseases is a more suitable term for our purpose as it includes diseases primarily characterized by stationary and non-progressive phenotypes such as North Carolina Macular Dystrophy and fovea hypoplasia.

2) In the Introduction on line 49, there seems to be a typo in the number of UCNE regions reported. 4,135 UCNE regions is supposed to be 4,351, based on the original paper (https://academic.oup.com/nar/article/41/D1/D101/1057253).

We have corrected this typo accordingly.

3) In the introduction on lines 75-76, these references: Lyu et al., Cell Reports 2021, PMID: 34788628, and Zhang et al., Trends in Genetics 2023, PMID: 37423870, could be added to the following sentence to provide additional: "This cellular complexity is the result of spatiotemporally controlled gene expression programs during retinal development”.

We have now included these relevant references.

4) On lines 77 and 84, I would write IRD as plural: IRDs.

This has been amended in the new version.

5) In introduction on lines 89-90, it can be further added that you provide experimental support for an ultra rare SNV in a cis regulatory UCNE affecting PAX6.

We have explicitly stated that we provided functional evidence for this UCNE.

6) On line 98, the authors refer to Figure 1A when noting that the integration of UCNEs with multi-omics data in human retina allows to evaluate the regulatory capacity of UCNEs across the major developmental stages of human retina. However panel A in figure 1 does not seem to show this point. It shows the comparison of elements across species. Please make appropriate changes to the main text and figure legend.

We have made the appropriate changes and located the reference to this figure in a more relevant part of the text (line 87).

7) Please explain what the names appended to the gene symbols in the first column "UCNE ID" in Supplementary Tables 1 and 2 refer to.

We have clarified what these refer to.

8) On line 145, can the authors clarify what they mean by "active gene expression in the retina". Is this just another way of referring to genes found to be expressed in retina? If so, it might be clearer just to write: "We annotated the identified active UCNEs to assign them potential target genes and thus assess their association with genes expressed in the retina"

We indeed meant genes found to be expressed in retina. As this phrasing might not be completely clear, we have now changed to the wording suggested by the reviewer.

9) One line 156, I would write "regulation of transcription" as listed in the gene ontology terms in Figure 2C, instead of "regulation of gene expression". The authors might want to add this to the Discussion. Can the authors include the full gene set enrichment results from Enrichr in a supplementary table at Padj<0.05 since only the top gene sets are shown in Fig. 2C (at P<1E-13)?

We changed the term to “regulation of transcription” to keep the nomenclature consistent to that of Figure 2C. We have also provided a full gene set enrichment from Enrichr as well (Supplementary Table 3).

10) On page 12, line 214, what does "EH38E1530321" Stand for? It seems to specify a distal enhancer-like signatures in bipolar neurons, but I couldn't find this ID in any database.

This refers to the identifier of ENCODE:

https://screen-v2.wenglab.org/search/?q=EH38E1530321&assembly=GRCh38

Additionally, when mentioning a specific UCNE, VISTA enhancer, or ENCODE cCRE (as in this case), we have explicitly included its corresponding identifier.

11) In the Methods section on lines 391-392, can the authors give some high-level explanation of the unconstrained integration method: "Single-nucleus RNA-seq of the same tissue and timepoints (GSE183684) were integrated using the unconstrained integration method". Also, can they comment on how retinal cell class identities were assigned (line 393). Was it based on known markers or on previous identification of cell classes and highly variable genes between clusters?

We have included a high-level explanation of the unconstrained method in the Methods section (lines 387-392). We also clarified that the assignment of cell class identities was based on known markers (line 394).

12) In the integration of UCNEs with bulk and single cell ATAC-seq and Dnase hypersentitivity regions, can the authors note in the Methods section (lines 400-404) what peak width was used to test for overlap with the UCNEs.

We have specified the peak widths that were used for the overlap with UCNEs (lines 397 and 403).

13) On line 436, the word 'and' is missing between "(SNVs, and indels < 50bp)" and "large structural variants".

This has been corrected.

14) On lines 443-444, please provide references to the computational tools listed. Please note if default settings/parameters were used.

We have specified that default parameters were used in the analysis (line 464).

15) In the following sentence in the Methods section on lines 447-449, it is not clear in the Results section how this was used in the flow of the analysis, and how many cases showed such a similarity in phenotype: "For each candidate variant, we compared the similarities between the participant phenotype (HPO terms) and the ones known for its target gene through literature search and clinical assessment by the recruiting clinician when possible." Can the authors add more detail to the Results section.

As the evaluation of the candidate variants was essentially performed on a case-by-case basis, we opted to include a rather general description of the workflow, which indeed included a comparison of the reported phenotypes with those associated with the putative target gene. An example of such comparison has been included in the Results (lines 186-187) section regarding the cases for which a NCMD-like phenotype was suspected.

16) It would be helpful to have a table that describes the different omics datasets used in the paper, with some basic annotations (tissue type, sample size, reference).

This has now been incorporated in Supplementary Table 11.

17) Can the authors add references to their sentence in the Discussion on page 17 lines 299-301: "As has been shown before, the phenotype caused by a coding mutation of a developmental gene can be different from the phenotype caused by a mutation in a CRE controlling spatiotemporal expression of this gene."

We clarified that this phrase referred to the case of PRDM13, for which bi-allelic coding pathogenic variants are linked to hypogonadotropic hypogonadism and perinatal brainstem dysfunction in combination with cerebellar hypoplasia (Whittaker et al., 2021), while non-coding variants within its regulatory regions are associated with NCMD.

Reviewer #2:

Minor discretionary suggestions for improving the presentation:

1) Wherever a specific UCNE, Vista enhancer or ENCODE cCRE is mentioned, the element should be identified by name or accession code: For instance (Iine 212): "this variant is located within a UCNE (PAX6_Veronica) that is catalogued as a cCRE in ENCODE (EH38E1530321)". UCNE names are particularly important, because they are systematically used as identifiers in the supplemental Tables and thus would enable the reader to easily find additional information about the element mentioned in the main text.

We have now explicitly included all corresponding identifiers throughout the text.

2) I also recommend inclusion of the UCNE, Vista and ENCODE cCRE tracks in all genome browser screen shots. The UCNE track is currently included only in Figure 1. Vista and ENCODE cCRE tracks are missing in all browser views.

We have now included UCNE, VISTA, and ENCODE cCREs tracks in the main genome browser figures (Figures 1 and 4).

3) Supplementary Table 6: It would be useful to indicate for each variant, the type of ophthalmological disorder (Table S5, column C) it is associated with.

We agree this is a relevant point. However, due to limitations in the (bulk) export of clinical information from the protected Research Environment of Genomics England, inclusion of this type of information is not feasible.

4) Fig S2 and supplementary Table 3 are not referred to in the main Text.

We have corrected this and updated the figure and table accordingly.

5) Supplementary Table 8: The Table caption should be expanded.

The contents of each column should be explained. For instance, column F: what means Homo_sapiens|M01298_1.94d|Zoo_01|2337? Where does this information come from, what data and software resources were used?

We have expanded the caption of this table to clarify this output, which is derived from the QBiC-Pred tool, a software used for predicting quantitative TF binding changes due to nucleotide variants.

6) Line 401 probable typo: 103-105 days (103-125?)

Indeed, this typo has now been corrected.

Reviewer #3:

1) Given that UCNE only accounts for a small fraction of gene regulatory elements, this study is likely with low sensitivity in terms of identifying potential regulatory mutations. Although one would expect that variants in UCNE are more likely to be pathogenic, it is hard to extrapolate from the results to assess the contribution of gene regulatory variant to the disease.

We agree that restricting our analysis to these particular regions is one of the limitations of the study, as stated in the Discussion section (line 304-311). However, one of our main aims was to provide a strategy to reduce the search space for pathogenic variants with a potential regulatory effect. Given the substantial body of literature supporting a regulatory role for these regions and, particularly, the availability of already-existing functional data, we considered that this set of regions could represent a suitable target for such analysis. Indeed, the features evaluated, and the methods presented in this study could be extrapolated and applied in other settings involving other candidate regulatory regions and/or tissues of interest, and their associated disease-phenotypes, for which, in any case, the overall contribution of regulatory pathogenic variation to disease might vary greatly.

2) I am wondering how many UCNE overlaps with open chromatin regions specific to the fetal retina and how many UCNE overlaps with adult only. Are UCNEs enriched for developmental genes? If so, how many patients are due to developmental defect?

We have now integrated into our analysis epigenetic measurements in adult retina, in particular the candidate cis-regulatory elements nominated by Cherry et al. PNAS 2020 (PMID: 32265282) based on accessible chromatin and enrichment for active enhancer-related histone modifications in adult human retina. Details about the integration of this dataset as well as the results and their corresponding interpretation have been incorporated in the Methods (lines 372-374), Results (115-117; Supplementary Table 2), and Discussion (lines 271-273) sections accordingly. In particular, out of the 111 UCNEs identified to display the active enhancer mark H3K27ac, 33 were found to maintain this signature at adult stage. Regarding the specific question from the reviewer, the majority of UCNEs overlapping with open chromatin regions are specific to the fetal retina (1,346), with only 7 UCNEs overlapping with open chromatin regions exclusively in adult state. This indeed further supports the major role of the identified candidate cis-regulatory UCNEs in the regulation of developmental gene expression programs, which was already suggested by the Gene Ontology enrichment analysis performed using the set of UCNE target genes as input (Figure 2C; new Supplementary Table 3). As far as the number of patients with development defects that were included in this study, these included: corneal abnormalities (n=62), Leber congenital amaurosis (n=142), ocular coloboma (n=111), developmental foveal and macular dystrophy (n=230), developmental glaucoma (n=94), anophthalmia or microphthalmia (n=120).

3) I am wondering if the 431 ultrarare variants found in the UCNEs is higher than expected. This can be tested by comparing patients without visual disorders.

Although it is indeed a relevant point, retrieving sequencing data from patients without visual disorders is not feasible for us. As it is a restricted research environment, we are only allowed to query sequencing data corresponding to participants enrolled within the framework of our specific sub-domain, namely “Hearing and Sight”, and therefore evaluating additional patients from other sub-domains is not doable. Based on previous studies and our observations, common variants are precisely the ones depleted within UCNEs, while ultrarare variation seems to occur at levels comparable to those observed elsewhere in the genome. Therefore, it is reasonable to speculate that this amount of ultrarare variants is not higher than expected as compared to patients without visual disorders. To further demonstrate the high intolerance of UCNEs to common variation, we have included an evaluation of the distribution of genome-wide residual variation intolerance score (gwRVIS) values compared to a set of randomly selected genomic regions in a human reference cohort (lines 173-178; 487-495; Fig. 3C). Additionally, to address this question further, we have also generated allele frequency distributions for all variants retrieved within a selection of 25 disease-gene associated loci and their corresponding UCNEs in order to assess the specificity of the overlap between rare variants and UCNE regions (lines 181-182, 496-501; Figure S2).

4) It seems that the ultrarare variants listed in sup table 6 are more abundant in a small number of genes. Is this due to the number/size of UCNEs is larger in these genes?

Indeed, the clustering of UCNEs in genomic regions containing genes coding for transcription factors and developmental regulators (e.g., OTX2, PAX6, ZEB2) seems to be one of their intrinsic properties, hence the observed enrichment for a small number of genes. One reason can be that these neighboring UCNEs cooperate to achieve higher degrees of tissue-specific regulatory accuracy needed for these genes.

5) The variant in the putative Pax6 gene regulatory element is intriguing. It would be much more informative if the enhancer with and without the variant is tested in parallel in fish.

That is a very relevant point. We have now tested the effect of this SNV in the UCNE on the reported expression pattern using the same experimental setup that we used for testing the wild-type construct, namely transgenic enhancer zebrafish assays. However, we could not obtain conclusive results, most likely due to the limitations posed by testing these regions outside their native genomic context. We have incorporated the details, results, and interpretation of the assays performed mutant construct in the Methods (lines 525-527; Supplementary Table 12), Results (lines 235-238; Supplementary Table 10), and Discussion (lines 350-353) sections.

6) (optional) it would be quite interesting to check the phenotype in fish or mice with the element repressed via technique such as CRISPRi.

Indeed, we fully agree that CRISPR-based techniques would be the ideal experimental approaches to further validate the functionality of the PAX6-associated UCNE and the identified variant in their native genomic context. Conducting these detailed and focused mechanistic studies is, however, beyond the scope of the present work, for which the main focus was the identification and functional annotation of ultraconserved cCREs.

Note: This preprint has been reviewed by subject experts for Review Commons. Content has not been altered except for formatting.

Learn more at Review Commons

Referee #2

Evidence, reproducibility and clarity

Starting with a comprehensive analysis of multi-omics data, the authors identify a subset of ultra-conserved non-coding elements (UCNEs) likely to play a role in retinal development. Restricting subsequent analyses to these genomic regions, they identify ultra-rare mutations associated with eye disease, using data from the 100k genome project. They then follow-up on one newly discovered, disease associated UCNE and a presumably causal mutation within this UCNE. The follow-up experiments involve fine mapping of the spatiotemporal expression pattern of a UCNE-driven reporter gene in zebra fish, as well as a re-examination of the disease phenotype in carriers of the mutation.

The computational pipeline used for prioritization of UCNEs is sound. The evidence supporting the claim that the identified ultra-rare SNV is causal is highly convincing. This study also constitutes proof of concept for a novel methodology to search for causal disease associated mutations in the nowadays still under-investigated non-coding part of the genome.

The paper is clearly written. Methods are described in enough detail to allow for reproduction of the results. Overall, this study is of high scientific quality, self-contained, and complete. Publication in a peer-reviewed journal should not be delayed by additional, perhaps interesting but non-essential experiments.

However, if the authors intend to undertake similar studies in the future, I would recommend to carry out the reporter-gene assays in zebrafish with both the wild-type and the mutant version of the UCNE. Comparison of the spatiotemporal expression patterns of the two alleles could provide valuable insights into the mechanism of action of the deleterious mutation under investigation.

Minor discretionary suggestions for improving the presentation:

Wherever a specific UCNE, Vista enhancer or ENCODE cCRE is mentioned, the element should be identified by name or accession code: For instance (Iine 212):

"this variant is located within a UCNE (PAX6_Veronica) that is catalogued as a cCRE in ENCODE (EH38E1530321)"

UCNE names are particularly important, because they are systematically used as identifiers in the supplemental Tables and thus would enable the reader to easily find additional information about the element mentioned in the main text.

I also recommend inclusion of the UCNE, Vista and ENCODE cCRE tracks in all genome browser screen shots. The UCNE track is currently included only in Figure 1. Vista and ENCODE cCRE tracks are missing in all browser views.

Supplementary Table 6: It would be useful to indicate for each variant, the type of ophthalmological disorder (Table S5, column C) it is associated with.

Fig S2 and supplementary Table 3 are not referred to in the main Text.

Supplementary Table 8: The Table caption should be expanded. The contents of each column should be explained. For instance, column F: what means Homo_sapiens|M01298_1.94d|Zoo_01|2337? Where does this information come from, what data and software resources were used?

Line 401 probable typo: 103-105 days (103-125?)

Significance

This work is of interest to different research communities: Biomedical researchers working on neural disorders, human geneticists engaged in GWAS studies, computational biologists trying to make sense out of omics data, molecular biologists exploring the "dark matter" of the genome, and finally the small community tackling the enigma of UCNEs. As with many omics papers, the most valuable parts of this study are in the supplemental tables, in particular tables 1,4, and 6. It can be hoped that some prospective readers will follow up on the leads presented in these tables.

The detailed computational and experimental characterization of a likely causal ultra-rare disease associated mutation may serve as a guiding and motivating example for medical geneticists working on other syndromes.

Back to UCNEs: They are enigmatic entities, which so far have largely resisted molecular and physiological characterization. It took 10 years to finally uncover a phenotype in ko mice missing one or several UCNEs, after the surprising initial observation that such mice were viable and fertile. The difficulties in studying the function of UCNEs may be due to their conjectured pleiotropic activity in different cell types at different developmental stages, their apparent cooperative interactions with many other control elements (limiting the power of reporter gene assays with single elements), and their putative involvement in morphogenetic processes (minimizing the relevance of epigenetic data collected from cell lines). In view of these considerations, I consider UCNE research starting from human disease phenotypes more promising, than ab initio approaches using reverse genetics in model organisms.

The impact of this paper is potentially very high. Note the following statement in the paper:

"For each instance for which only the UCNE variant remained as candidate, we placed a clinical collaboration request with Genomics England."

We thus can expect more exiting stories from the same team. The strategy and computational pipeline introduced here are of course applicable to other congenital diseases, and it can reasonably be hoped that researchers inspired by this study will apply components of the methodology in other contexts. The prioritization of UCNEs in studying the "dark matter" of the genome and the "missing heredity" would likely lead to new insights into the function of these enigmatic elements and the reasons for their extreme conservation.

My background: I'm a bioinformatician with first training in molecular genetics. My research focus is on gene regulation: promoters, enhancers, transcription factor binding sites. I also made some contributions to the UCNE field, having co-developed UNCEbase with Slavica Dimitrieva. On the other hand, I don't claim to be an expert in medical genetics, and more specifically, I know very little about eye diseases and retinal development.

An ABI (Application Binary Interface) defines how data structures or computational routines are accessed in machine code, which is a low-level, hardware-dependent format.

This made me rethink of how necessary we need coding or programming to learn in DH? Sometimes DH students have misconception that they always need coding to work in DH field. The other more important element to work in DH is "the ideas" of what you actually what to build, transform and create. Computational tools or even coding skills will come after that.

format as code

This is interesting to note that it goes all the way back to our genes. When infants receive the care they need, they have the capacity to calmly trust their parents. Free from worrying about getting their needs met, they can show growth throughout their whole lives.

personally poignant detail in a photograph which 'pierces' or 'pricks' a particular viewer, constituting a private meaning unrelated to any cultural code.

When you design by the interface, code duplication is most likely. To avoid this, we use abstract classes which - Implements the interface - This the base case for other class that used to implement this interface - Can contain methods with/without definition that the child classes can reuse

This books, absolutely the worst book, I have ever seen on this topic. Here is a good one, use this https://www.openintro.org/book/os/

The biggest problem with this book, it does not explain why the model is so, then some code is provided and bad libraries of r are used. Don't wast you time on this book

This seems a bit much, it's kind of like asking someone to live a double life

this is definitely something i would do for fun to make some money

To be honest, when the names don't necessarily align with the type of work that a company does, it gives fraudulent vibes. Sounds suspicious at best.

This would be a killer introduction to computing I wonder if there's a demo of this anywhere

Here I want to highlight the form. The form is an example of good accessibility practices. After looking at the HTML code we can see that the label and the corresponding form field are associated with each other because the for attribute and the id attribute are the same. Ex: for="cons_first_name" and id="cons_first_name".

This proper structure will make it easier for screen readers.

Here I want to highlight the image but I can only annotate text so my comments are for the IMAGE. I believe this would fall under the perceivable principle because it has to do with making the components presentable to users in whatever way they can perceive them.

This is an example of bad accessibility practices. If you take a look at the HTML code for the image, the alt code does not describe the image at all. Instead it says "© Staffan Widstrand / WWF". Poor use of the alt text attribute makes it difficult for visually impaired users as they use this to understand the images that they can not see.

This is an example of good accessibility practice for the robust principle. If you look at the HTML code this title has an H2 header. Other section titles on the webpage also have H2 headers. This makes the webpage accessible to people that use screen readers because the common header tags make it easier to jump from section to section.

Compatibility with Browser Zoom

Explanation: The website is compatible with browser zoom functionality. When users zoom in or out using browser controls, all content, including text, scales appropriately without causing layout or readability issues.

weapons of mass destruction #wmd #massdestruction #literallymasstoz

attacker may also hijack packets by injecting malicious code into the packets itself.

I think there are both pros and cons to having open sources but I think there could be some compromise that allows for closed source depending on the task being preformed by the AI

Feature 1: ( good feature): The language of this page is identified and it is Bengali. This can be found from this page's html code as well

Feature 3 ( Good Feature ): A figure is present and also for non text content alternative text is present in the code section.

Sample code

Change this to heading 3

When I executed,

"tox -e dev -- sh bin/hypothesis --dev user add" (I deleted 'q' from the original command)

I got an error as below. how can I fix this?

AttributeError: module 'psycopg2' has no attribute 'paramstyle' ERROR: InvocationError for command /bin/sh bin/hypothesis --dev init (exited with code 1)

Robert Greene used a color code for his index cards which also helped him to realize gaps in certain areas. He also liked them because "It also made the shoebox look pretty cool."

It' is pretty good to see the mapping innovation taking several shapes, from the starting narrative to this one.

Regarding feedback from this one I would make a call out that make more visible where the data and code behind the map is hosted and how to reproduce the results.

On a more general sense, I think is important to see how the different narratives are better connected and which values they embody and make explicit. I would propose this values:

1. Utility:

   • internal: helping us to make short or long lasting peer to peer connections like the one between Copincha (Habana, Cuba) and HackBo/Grafoscopio (Bogotá, Colombia) communities resulting from DOTS 202.
   • external to showcase which innovation, people and communities are doing and how they are connected now or can be in the future.

2. Reproducibility: The data narratives should be able to be reproducible.

3. Portability: Functionality bundles, including data, code, software should be packages to they can be used in local contexts, particularly those with low/intermittent internet connectivity.

4. Recontextualization: Our data narratives should be empowering its reuse, adaptation, and extension by other communities and in other context.

5. Commons/Community oriented: licenses on data/code should be explicit to allow the previous qualities. Some times that would require a copyfarleft license that protect third parties extract value from the data narratives and its bundles against the community interest (cfg current discussion on data collection from IA projects against community of creators).

its so important to understand there is unconsious bias in many of these algorithms due to the way things are set up structurally within the code of system.

This work has been published in GigaByte Journal under a CC-BY 4.0 license (https://doi.org/10.46471/gigabyte.93), and has published the reviews under the same license. These are as follows.

**Reviewer 1. Po-Hsiang Hung **

Are all data available and do they match the descriptions in the paper?

No. The dataset in FTP includes all the Bac sequences and the restriction enzyme recognition sites in csv files. However, I could not find the database of pairs of BACs, which have overlaps generated by restriction enzymes that linearize the BACs. The makePairs function gave me an error when I tried running it locally, so I was not able to verify what is in these datasets. Personally, I find this function to be one of the most useful features described in this manuscript.

Are the data and metadata consistent with relevant minimum information or reporting standards? See GigaDB checklists for examples http://gigadb.org/site/guide

Yes. This manuscript contains the necessary minimal information (Submitting author, Author list, Dataset title, Dataset description, and Funding information)

Is there sufficient detail in the methods and data-processing steps to allow reproduction?

No. The authors provide their code in GitHub such that researchers can download the datasets and analyze the sequences locally. However, I felt that the descriptions in the readme.md file is often insufficient to reproduce the data presented in the manuscript, especially for researchers with little to no programming experience. Detailed information includes examples of how to use each function, the input format, and the location of the output folder/files. I also encountered software version issues during the installation of bacmapping. Please re-test the code in a new environment and describe all the versions of each software. For instance, I found Python version 3.11 is incompatible with this package while Python version 3.7 is compatible.

Is there sufficient data validation and statistical analyses of data quality?

No. The author used the BioRestriction class from Biopython to get the digestion site information. No extra validation is conducted in this manuscript. Due to the errors I encountered in re-running the code (see details in Any Additional Overall Comments to the Author), an independent method for checking several digestion sites in some Bac clones is suggested. The suggested independent method is to do enzyme digestion on some Bac clones or upload some Bac sequences to other software and compare the digestion sites.

In the output files that contain the digestions sites for each enzyme, some of the enzyme digestion sites are either NA or []. What is the difference between the two? If they mean the same thing (no cutting by the enzyme), bugs or other coding errors may cause this inconsistency. Please check the code again and also verify some of them using the independent methods suggested above. Examples of this issue are the files in maps>sequenced>CEPHB. Here I list two enzymes that show different results in each file: 3.csv : Ragl ([]), SchI (NA) 6.csv: EspEI (NA), AccII([]) 13.csv: EcoT22I ([]), Hsp92II (NA) X.csv: PacI ([]), AcIWI (NA)

Is the validation suitable for this type of data?

No. No validation in this manuscript. See the answer above.

Additional Comments: The authors make a database with enzyme digestion site information of Bac clones to help people to use the Bac clones for further usage. I think it is useful to have this information and also have the code to do further analysis locally. Thus, I think providing a very detailed user manual (or readme.md) is very important to help people use this dataset. Below I summarized the issues I encountered in running codes and also some suggestions. Major points: (1) I tested some bacmapping functions, and I discovered that some functions are not working as intended due to typos/bugs - The version of the software is required to help people properly install this package - Refining the code and also providing a better user manual is very helpful for people without a lot of coding experience to use it. The detailed information includes examples of how to use each function, the input format, and the location of the output folder/files. Descriptions for some functions in the readme file are not detailed enough and often do not describe what the input needs to be. For example, getCuts() require ‘row’ as input. But the author never gives a detailed description of what ‘row’ is in the readme file. I had to look in bacmapping.py to understand what ‘row’ is. If a function requires the variable ‘row’, show a few examples of how ‘row’ can be extracted from the proper input file. - mapPlacedClones() requires an input file (‘/home/eamon/BACPlay/longboys.csv’, line 335) that is located in the author’s local computer and is not available through github. - Typo in line 814 in getMap(). Should be: name = cloneLine[‘CloneName’] - Inconsistency in output variable type in getMap() (line 830 and 851). When local == ‘sequenced’, the output variable is a tuple, which causes issues in downstream functions such as getRestrictionMap() (line 869). (2) Add pairs of BACs into the dataset (3) The output file of digestion sites of each enzyme, some of the enzyme digestion sites showed NA or [ ]. Please double-check this and explain the differences (4) Validation of an independent method for the digestion map is suggested

Minor points: (1) Add a title to each column of sequencedStats.csv is useful for understanding the table easier

Re-review:

The authors have addressed majority of my points. The software installation works great after considering version control. The updated read.me provide detailed information for each function and their required input variables, and the examples in jupyter notebook are a great help for running the code. I did, however, encounter two minor errors when I tested the Ch19_bacmapping_example.ipynb on a Mac system. Please check this and update it.

(1)The .DS_store file that is automatically generated on a Mac system in the bacmapping/Examples/Ch19_example/maps/placed folder causes an error when running bmap.mapPlacedClones(cpustouse=cpus, chunk_size=chunksize). The same problem happened when I ran bmap.mapSequencedClones(cpustouse=cpus). After I deleted .DS_store in the folder, the code worked.

Here is the error message when I ran bmap.mapSequencedClones(cpustouse=cpus). NotADirectoryError: [Errno 20] Not a directory: '/Users/user_nsame/bacmapping/Examples/Ch19_example/maps/sequenced/.DS_Store'

(2) The second error is from running bmap.getRestrictionMap(name,enzyme). I got the error message, 'list' object has no attribute 'item'. I was able to run this function after changing maps[enzyme].item() to maps[enzyme] in line 779 of bacmapping.py. I encountered the same error with the drawMap function. I was able to run to run this function after changing line 847 of bacmapping.py from rmap = maps[nenzyme].item() to rmap = maps[nenzyme].item().

Here is the error message

AttributeError Traceback (most recent call last) Cell In[20], line 5 3 maps = bmap.getMaps(name) 4 #print(maps) #this is a big dataframe of all the maps, uncomment to check it out ----> 5 rmap = bmap.getRestrictionMap(name,enzyme) 6 print('Sites in ' + name + ' where ' + enzyme + ' cuts: '+ str(rmap)) 7 plt = bmap.drawMap(name, enzyme)

File ~/miniconda3/envs/bacmapping/lib/python3.11/site-packages/bacmapping/bacmapping.py:779, in getRestrictionMap(name, enzyme) 777 maps = getMaps(name) 778 nenzyme, r = getRightIsoschizomer(enzyme) --> 779 return(maps[nenzyme].item())

AttributeError: 'list' object has no attribute 'item'

**Reviewer 2. Wei Dong **

Is there sufficient data validation and statistical analyses of data quality? Not my area of expertise

Is the validation suitable for this type of data? I am not sure about this.This is not my specialty.

Overall comments: This is a great idea, fully exploring, integrating, and utilizing existing data for new research.

TurboWish是一个用于分析Rust程序的框架，专注于揭示使用async/await编写的面向任务的代码的性能和资源使用情况。

Languages don't fall into the category of being either compiled or not. Implementations do. And the misconception of compiled code being ipso facto faster is a common one, but it's a misconception nonetheless (I suspect most often held by people who've never implemented one).

low pass rate similar to code generation. however, code generation take advantage of pre-training, tool usage is not

Wouldn't Morse code only really be helpful to converting other english based languages as it is based on the english alphabet? Communication between the US and european countries was probably fine but what about the eventual need to communicate with other countries through telegraph? (China, Japan, India, Greece, Egypt, etc...)

It is only the advent of the "new technology" of the telegraph that forced people to create a new sort of language which is wild. Morse Code made a fast process of communicating even faster, which is interesting because today morse code seems like such an irrelevant language.

Is this why eating is causing instant sleepiness? Non-digestive vessels vasoconstrict and shut off too much cerebral blood flow, then nerves instantly have reduced firing/waste and CO2 build up/diminished mitochondria output/oxidative buildup/ &or then resultant inflammatory triggering cytokine increase?

Vessel endothelial enormous surface area, manipulator of blood flow vasoconstrictor system, and cytokine producer/influencer, and high vulnerability sensitivity to viral infection/corruption...and then it's role or adjacent system and the immediate available Google research on COVID affecting> the vascular elastin system and corrupted elevated production of destructive elastases resulting in reduced vascular compliance then resulting too narrow "pulse pressure" band essentially creating arteriosclerosis.

Also, make sure to be thinking of the entire vascular system not as one system, but subsided by dynamic changing gated sections and inspect signaling creating changing locations and amounts of high/low pressure zones. Also, keep in mind 3 things about BP: 1, when taken with a cuff it is only measuring a reading at the elbow. 2, is a reading from the artery and not giving any direct data from vein part of the system. 3, BP is not the same as blood flow. So I conceive that you could read a good BP, but actual flow could be completely inadequate.

Remember analogy, vascular system is just like car AC system, or any pressurized hydraulic system, or even actually electric circuits. Meaning that there is a high pressure side, the load component(s), and a low pressure side. Also remember veins act as the reservoir tank, and when they constrict it is injecting more blood into the system to, if functioning correctly, allow higher performance and meet increase load demand. It also, therefore has less direct effect on whole system BP vs artery constriction because it's downstream of the load. Arterial constriction conversely has immediate direct effect on systolic BP as it is essentially putting a wall directly downstream of the heart. Therefore, regardless if diastolic pressure is zero or high, when the heart contacts, the pressure shoots straight up.

A working theory component: my pulmonary vein is inappropriately constricting too much. That causes high back up pressure at alveoli. Exercise then induces veinous reservoir injection and increased blood volume into the "working system" further increasing pressure. Possibly arterious had already been fully dilated at rest in order to compensate and then when exercise happens, it can't be dilated further to increase blood flow throughout and BP increases further all behind the pulmonary vein "dam". However it doesn't present as right side heart failure like might initially be guessed (with leg and belly edema) because the right side heart is not failing...yet. So it contains any further backflow and the alveoli are the weakest point and taking the most abuse and pressure is relieved as pulmonary edema. And therefore what may be present is if we look for it, we'll find that actual blood throughput output exiting the heart is too low. And this can exist with a normal ejection fraction because the heart is functioning correctly and pumping the right percentage of what is a low starting volume. And also this can support why right ventricle is showing first signs of enlarging because it's being overpressured and stretching out (enlarging). And this can support why normal BP readings are measured at the arm because it can completely handle the abnormally low blood volume being received in the downstream location it's at. And then therefore this further supports why BP is normal but HR is riding the high limit at rest and then instantly jumps on exertion AND why dizziness happens because the artery system was already maxed out dilation at rest and for any amount of exertion, increasing HR because of the immediate too fast rise in tissue hypoxia due to too low blood supply the brain keeps driving up HR to meet demand. Total result upon exercise: supply continues to more and more not meet demand, HR rises faster and faster to try to inadequately compensate, physically become weaker especially after high output anaerobic every supply deleted in 1-3 minutes and there is no aerobic capacity cavalry with it's O2 rushing in to take over and that's when I fall off the cliff> HR spikes even faster, chest pain immediately jumps, lung edema turns on full tilt as the HR spikes and the resulting pressure is forced to "spray out of the gaskets (alveoli), and brain blood O2 supply immediately becomes super inadequate and the dizziness and need to fall over is the instant result effect. And since dysfuntioning cerebral vasoconstriction is likely the cause or highly involved in migraines, this also supports why the headaches come. ... And perhaps this explaining the rest pain and how it increases with dex and exertion because blood flow o2 becomes inadequate. Then causing lactic acid waste and CO2 buildup... (ie pain). And then it, like all body tissues being deprived necessary blood flow trigger cytokine inflammation response. ... And then, fuck it, maybe this IS chronic fatigue syndrome, and IS long covid explaining PEM, explaining why every symptom imaginable in any combination permutation is being shown, is explaining the observed elevated varing soups of cytokinesis, explaining all variety of tissue damage depending on any person's unique amount of total hit and their particular systems vulnerabilities and ultimately how far down they went on the increasing spiralling cascading systems failure towards total shutdown, and explains why measures at addressing the variety of manifestations are all somewhat helpful, but inadequate and varing efficacy from patient to patient because they are all too downstream of the root cause trunk of the symptom tree where the need to relieve vascular over constriction is the root or next to the root of the symptom tree that is common to all patients. If this were all to be accurate, then the seed would be what caused the break in vasoconstrictor system and repairing/killing it, or perhaps it's a PC bootstrap phenomenon where the simple uncomplex virus was just enough bios code to place innocuous wrenches in any of machines of the systems and then those malfunctioning systems took over control in their new malfunctioning patterns and became the new bosses that are infact the disease, you become your disease, and the initial virus seed has long been killed/departed (they're the ultimate down the road end game that is the totally corrupted bcdhhs that will then exist now as a new monstrous organism slowly lingering and depleting itself and eventually all resources at which point it will have finally killed itself after it destroyed the once thriving self sustaining world it lived in. COVID then is the teenage abusive bf or mean drunk father from their past, that put in motion what would become decades and generations of monsters, years and years after they had been long since gone). And maybe this explains the phasing leaving and returning it symptoms. Because when enough if the symptoms start to be reset/repaired, that starts spiraling the spread of the shutdown of the corruption back to health, but if the spiral up isn't strong enough to overcome the consequential reactive spiral down response, the monster returns and the rebellion is quashed. And so explains why the overall, in every system, stronger less vulnerabilities less armor chinks youth are able to quash with ease the spiral down with their incumbent exceptional spiral up response. .... And aside, this explains why dysautonomia has become a top suspect. And explains why POTS has become almost synonymous with long COVID and CFS.

Author Response

The following is the authors’ response to the original reviews.

Reviewer #1

1) Overall, this is a useful tool, the data is well-presented, and the paper is well-written. However, the predictions are only compared with two existing reconstruction tools though more have been recently published

The aim of this work was to facilitate high-throughput generation of strain-specific metabolic models e.g. at the scale of 100s -1000s as indicated throughput the introduction (see lines 74-82, 91-94), and therefore we only compared tools which were capable of high-throughput analysis via command line and excluded others (e.g. merlin). We have now tested this against the other recent command line tool, gapseq which had escaped our gaze. Thank you for bringing this to our attention. Additionally, we have included KBase (ModelSEED, a web-based app that does not support high-throughput analysis) to allow for readers to interpret the results in the context of community standard approaches, since KBase is a popular tool.

We have added an explicit statement about the choice of approaches, now at lines 194-199 as follows:

“We compared the output and performance of Bactabolize to the two previously published tools that can support high-throughput analyses i.e. CarveMe (30) and gapseq (31). To aid interpretation in the context of community standard approaches, we also include a comparison to the popular web-based reconstruction tool, KBase (ModelSEED), and a manually curated metabolic reconstruction of K. pneumoniae strain KPPR1 (also known as VK055 and ATCC 43816, metabolic model named iKp1289) (15).”

The methods section was updated accordingly (now lines 552-558):

“A draft model was generated using gapseq version 1.2 with the ‘doall’ command using the unannotated genome (as gapseq does not take annotated input files). Gap-filling was subsequently performed using the ‘fill’ command and a custom M9 media file to match the nutrient list found in Bactabolize (https://github.com/kelwyres/Bactabolize/blob/main/data/media_definitions/m9 _media.json).

Finally, a draft model was constructed using the annotated genbank K. pneumoniae KPPR1 file and the KBase narrative (15) (https://narrative.kbase.us/narrative/ws.14145.obj.1).”

The results section (now lines 193-308) and Figure 2 have also been updated / restructured to reflect the new analyses, and include a comparison of the relative compute times for the construction of models (lines 281-291) as follows:

“While model features and accuracy are essential metrics for comparison, computation time is also a key consideration for high-throughput analyses. We recorded the time required for each tool to build draft models for 10 of the completed KpSC genomes used in the quality control framework (see below) on a high-performance computing cluster (Intel Xeon Gold 6150 CPU @ 2.70GHz and 155 GB of requested memory on a CentOS Linux release 7.9.2009 environment. CarveMe KpSC pan was the fastest with a mean of 20.04 (range 19.90 - 20.18) seconds, followed by CarveMe universal at 30.28 (range 29.20 - 31.80) seconds, then Bactabolize KpSC pan at 98.05 (range 92.19 - 100.4) seconds. KBase took 183.50 (range 120.00 - 338.00) seconds per genome via batch analysis, including genome upload time and queuing. gapseq took 5.46 (range 4.55 - 6.28) hours to produce draft models (not including the required gap-filling), consistent with previous reports (37).”

Finally, the whole discussion has been updated and substantially restructured (lines 472-474, 475-493, 494-512). Specific mentions to the new analyses are at lines:

472-474: “Consistent with this assertion, our draft KPPR1 model constructed with KBase (without manual curation) was an outlier in terms of the very low number of genes, reactions and metabolites that were included.”

475-493: “CarveMe with universal model (30) and gapseq (31) are the current gold standard automated approaches for model reconstruction, and we show that a draft KpSC model generated by Bactabolize with the KpSC pan v1 reference resulted in similar or better accuracy for phenotype prediction (Figure 2). Both the CarveMe universal and gapseq models resulted in high numbers of true-positive and true-negative growth predictions. However, these were also accompanied by comparatively higher numbers of false-positive predictions that resulted in a lower overall accuracy for substrate usage analysis compared to Bactabolize with the KpSC-pan v1 reference (Figure 2), and comparatively lower precision and specificity for the gene essentiality analysis. False-positive predictions may indicate that the relevant metabolic machinery are present in the cell but were not active during the growth experiments (e.g. due to lack of gene expression). In this regard, false-positives are not always a sign of model inaccuracy. However, false-positive predictions can also occur from incorrect gene annotations e.g. due to reduced specificity of ortholog assignment resulting from the use of the universal model without manual curation. Given a key objective here is to facilitate high-throughput analysis for large numbers of genomes, it is not feasible to expect that all models will be manually curated, and therefore we believe that identifying fewer genes with lower overall error rates provides greater confidence in the resulting draft models. We also note that the BiGG universal reference model which CarveMe leverages is no longer being actively maintained. In contrast, user defined reference models can be iteratively curated and updated to incorporate new knowledge and data as they become available.”

510-512: “However, gapseq’s long compute time makes it inappropriate for application to datasets comprising 100s-1000s of genomes (such as have become increasingly common in the bacterial population biology literature).”

2) My understanding is that the tool requires a set of reference reconstructions for other strains of the target species. If no reference reconstruction is available for another strain of the target species, can this species not be reconstructed?

Any input reference can be used to generate models however, single strain models matching the target species, or ideally a species-specific panreference, are recommended for best results. We have added a discussion on these points at lines 128-133:

“For optimum results we suggest using a pan-model that captures as much diversity as possible for the target species or group of interest, because Bactabolize’s reconstruction method is reductive i.e. each output strainspecific model will include only genes, reactions and metabolites that are present in the reference or a subset thereof (although novel genes, reactions and metabolites can be added via manual curation).”

We expand on these points further in the discussion:

494-512: “Bactabolize’s reference-based reconstruction approach is reductive, meaning the resultant draft models will comprise only the genes, reactions and metabolites present in the reference, or a subset thereof, and will not include novel reactions unless they are manually identified and curated by the user. This is an important caveat that should be considered carefully for application of Bactabolize to large genome data sets, particularly for genetically diverse organisms such as those in the KpSC. For optimum results we suggest using a curated pan-model that captures as much diversity as possible for the target species or group of interest. While we acknowledge that a reasonable resource investment is required to generate a high-quality reference, we have shown that a pan-model derived from just 37 representative strains can be sufficient to support the generation of highlyaccurate draft models (Figure 2 and 5). Additionally, we note that it is possible to use a single strain reference model, which should ideally represent the same or closely related species to that of the input genome assemblies, in order to facilitate accurate identification of gene orthologs. It is technically possible to use an unrelated reference model, but this is expected to result in inaccurate and/or incomplete outputs and has not been tested in this study. In circumstances were no high quality closely-related reference model is available, we recommend alternative reconstruction approaches that leverage universal databases e.g. CarveMe (30) or gapseq (31). However, gapseq’s long compute time makes it inappropriate for application to datasets comprising 100s-1000s of genomes (such as have become increasingly common in the bacterial population biology literature).”

3) How do the reconstructions generated by Bactabolize compare to those generated by other reconstruction tools besides CarveMe and ModelSEED, e.g., gapseq (Zimmermann et al, Genome Biology 2021. 22:81) or merlin (Capela et al, Nucleic Acids Res 2022, 50(11):6052-6066?

See response to rev 1 point 1.

4) How are the accuracy, specificity, and sensitivity of the pan-models calculated? Is the compared experimental data on the species level?

We used the pan-model as a reference from which we generated a strain-specific model for K. pneumoniae KPPR1 (using Bactabolize and CarveMe). This strain-specific metabolic model was then used to simulate growth phenotypes and compared to published experimental data for KPPR1. This was described in the methods section, including the calculations for the metrics (lines 589-593); however, we have also expanded the description within the results section to clarify the approach (lines 201-209):

“De novo draft models for strain KPPR1 were built using; i) Bactabolize with the KpSC pan v1 reference; ii) CarveMe, with its universal reference model (CarveMe universal); iii) CarveMe, with KpSC-pan v1 reference (CarveMe KpSC pan); iv) gapseq; and v) KBase (ModelSEED). ….. Subsequently, each model was used to predict growth phenotypes; i) in M9 minimal media with different sole sources of carbon, nitrogen, phosphorus and sulfur; and ii) for all possible single gene knockouts in LB under aerobic conditions. The predicted phenotypes were compared directly to the published phenotype data.” [Note the published data are cited in the previous manuscript sentence, not shown here].

5) The link https://github.com/rrwick/GFA-dead-end-counter, in line 286 does not work.

Link regenerated – now at line 451-452 and 604

Reviewer #2

1) KpSC pan-metabolic reference model is provided. Are they required as input for Bactabolize? Are the gene, metabolite information open accessible by users? o See response to reviewer 1 point 2 above and;

All data for the KpSC pan-model described in this work are accessible in the model files and amino acid + nucleotide files + data table at https://github.com/kelwyres/KpSC-pan-metabolic-model. This is also linked in the manuscript at line 631 and in the Data availability statement at line 661.

2) In the results section "description of Bactabolize", the authors present technical details on how to generate a metabolic model. For the input and output, please provide concrete examples to show the functionality of Bactabolize.

Detailed instructions, example code and example input/output files are available via the Bactabolize GitHub repository: https://github.com/kelwyres/Bactabolize.<br /> Instructions and example code can be found on the wiki: https://github.com/kelwyres/Bactabolize/wiki Test data and example files are at: https://github.com/kelwyres/Bactabolize/tree/main/data/test_data

The Github repository is linked in the manuscript at lines 95, 124, 552, and 667, and we have added a further reference at line 124, which mentions the example code/data: “Full documentation, including example code and test data are available at the Bactabolize code repository (https://github.com/kelwyres/Bactabolize).”

3) To generate metabolic models, the authors present comparison results with other methods. However, the authors only present the numbers in genes, metabolites and substrates. Since the interactions between gene, metabolite, and substrate are also critical, if possible, please provide the coverage details about these interactions. Venn diagram is recommended to compare these coverage differences.

Two additional supplementary figures have been generated (Figures S5 and 6) showing Venn diagrams of metabolites and reactions for the highthroughput analysis approaches that are most relevant to this work (see also response to rev 1, point 1). These are discussed at lines 224-237:

“Figures S5 and S6 show the overlaps of metabolites and reactions between the high-throughput reconstruction methods after processing with MetaNetX (59) to standardise the reaction and metabolite nomenclatures (excluding CarveMe pan for simplicity and given the likely problems of reaction oversubscription). The majority of the reactions included in the Bactabolize model were conserved in either the CarveMe universal model (n = 1225, 53.2%), gapseq model (n = 54, 2.3%) or both (n = 665, 28.9%). The reaction overlap was skewed to the CarveMe universal model which shared 1225 reactions that were conserved in the Bactabolize model but absent from the gapseq model. Notably, the gapseq model contained a large number (2200) of unique reactions (70.4% of those in the model). Similarly, the vast majority of metabolites in the Bactabolize model were conserved in one or both of the other models (n = 917, 85.6%). However, it is likely that true overlaps between methods are underrepresented due to the different reaction identifiers and chemical synonyms used within the BiGG (Bactabolize, CarveMe) vs ModelSEED nomenclatures (gapseq), which are difficult to harmonise in an automated manner even after the application of MetaNetX.”

Figure 2 shows not only the model numbers but also includes benchmarking to real phenotypic data in 2DEFG as the key mode of comparison between models. This encompasses meaningful interactions between gene, metabolic and substrate. The results are discussed at length in text at lines 253-271:

“We assessed the performance of each model for in silico prediction of growth phenotypes compared to the previously published experimental data (15). Accuracy, sensitivity, specificity, precision and F1 scores were calculated (60). Note that the specific set of growth substrates and gene knockouts that can be simulated is determined by the sets of genes and metabolites captured by each model and is therefore model-dependent (Data S1 and S2). Among those with matched experimental phenotype data, the Bactabolize and CarveMe universal models were able to predict growth for a greater number of carbon, nitrogen, phosphorous and sulfur substrates than gapseq, CarveMe KpSC pan, KBase and iKp1289 models (Figure 2C, Data S1). While the CarveMe universal model had the highest number of truepositive growth predictions overall (n = 132 of 617 total predictions), it also had a comparably high number of false-positive predictions (n = 39 of 617 total predictions, Figure 2D). Similarly, the gapseq and iKp1289 models resulted in 31 (262 total predictions) and 50 (513 total predictions) falsepositive predictions, respectively. In contrast, the Bactabolize model had fewer false-positive predictions (n = 21 of 505 total predictions) alongside a high number of true-positive predictions (n = 117 of 505 total predictions), resulting in the highest overall accuracy metrics (Figure 2E, Data S1). The KBase model was a notable outlier, associated with a high number of falsenegative predictions (n = 31 of 103 total predictions) and low false-positive predictions (n = 3 of 103 total predictions), presumably resulting from the very low number of genes and reactions included in the model, driving low sensitivity and accuracy.”

Lines 272-280:

“The gene essentiality results showed that gapseq produced the highest absolute number of true-positive gene essentiality predictions (n = 79 of), followed by Bactabolize KpSC pan (n = 44 of 1220 total predictions), then CarveMe universal (n = 39 of 1951 total predictions). CarveMe universal had the largest number of true-negatives by a wide margin (n = 1599 of 1951 total predictions), followed by gapseq (n = 1085 of 1403 total predictions), then Bactabolize KpSC pan (n = 939 of 1220 total predictions), driving their high accuracies (83.96%, 82.96% and 80.57%, respectively). The Bactabolize model was associated with the greatest overall precision and specificity (Figures 2F & 2G) while the gapseq model resulted in the highest F1-score and sensitivity.”

4) Are quality control and gap-filling needed to be processed when constructing a new metabolic model?

Our goal here was to implement an approach to support high-throughput analyses (see response to rev 1 point 1), including leveraging draft genome assemblies as the bases for the construction of strain-specific metabolic models. As part of this work, we have described a robust quality control (QC) framework for screening draft K. pneumoniae genomes i.e. to identify genome assemblies that should not be used. We developed this framework by comparison to models generated for matched completed genomes. Our analyses demonstrate the importance of applying QC to the input draft genome assemblies. When appropriate QC is applied to the input genomes, the resultant draft models show a high degree of completeness compared to the matched models derived from complete genomes. The draft models can also be used to simulate growth phenotypes with high accuracy as compared to those simulated for the matched complete genome models.

No specific QC was applied to the draft models themselves, other than confirmation of positive growth prediction in m9 minimal media plus glucose (which is expected to support growth of all K. pneumoniae). In cases where the input assembly passed our QC criteria but the resultant model was unable to simulate growth in m9 minimal media plus glucose, gap-filling may be optionally applied. Again, by comparison to the simulated phenotypes from matched complete genome models, we show that these gap-filled draft models can produce accurate phenotype predictions. See lines 396-404:

“Of the 901 draft genome assemblies which passed our QC criteria (≤200 assembly graph dead ends), 23 of the resulting draft models failed to simulate growth in M9 minimal media with glucose (despite capturing ≥99% of the genes and reactions in the corresponding complete models). It is expected that all KpSC models should be able to simulate growth on M9 media with glucose as a sole carbon source, as this central metabolism is universal amongst KpSC. To replace missing, critical reactions required for growth on M9 with glucose, we investigated model gap-filling using the patch_model command of Bactabolize. We then assessed the accuracy of the gap-filled models for prediction of growth on the full range of substrates, as compared to the predictions from the corresponding complete models.” Lines 409-413: “Substrate usage predictions from the 21 successfully gap-filled models were highly accurate, with 18/21 having a prediction concordance of ≥99% across all 846 growth conditions (12/21 had 100% concordance) (Figure S9). We therefore conclude that models generated for genome assemblies passing our QC criteria, which have been gap-filled to successfully simulate growth on minimal media plus glucose, are suitable for the prediction of growth across a range of substrates.”

5) Are there any visualization results to check the status of the generated draft model?

No. This is a tool for large-scale and rapid production of metabolic models, and phenotype prediction and we have not included visualisation tools. Third party tools are available e.g. https://fluxer.umbc.edu/. We do provide optional generation of MEMOTE reports at lines 136-138:

“Draft genome-scale metabolic models are output in both SMBL v3.1 (41) and JSON formats (one pair of files for each independent strain-specific model), along with an optional MEMOTE quality report (42)”.

Reviewer #3

1) The justification and evaluation of the generated models are inadequate and onedimensional. The authors only focus on statistics such as the number of reactions and genes in the models, which does not accurately depict the completeness of the model.

The reviewer has misunderstood how we have used ‘completeness’ in this manuscript. In the section describing our novel QC framework, we use this term to refer to the relative completeness of draft models generated from draft genome assemblies as compared to curated models generated from complete genome assemblies for the same strains. The latter were considered as the ‘complete’ models for this purpose. We are not referring to any measure of network or metabolic pathway completeness. We specifically refer to gene and reaction capture compared to the ‘complete’ models because these features directly reflect the problem we are trying to address i.e. that draft genome assemblies may not contain the complete set of genes that are truly present in the underlying genome. We have updated the manuscript text to further clarify the problem we aim to address in this section and justify the use of gene and reaction capture metrics:

Lines 310-319: “There are now thousands of bacterial genomes available in public databases, the majority of which are in draft form, comprising 10s to 1000s of assembly contigs. This fragmentation of the genome is caused by repetitive sequences that cannot be resolved by the assembly algorithm and/or sequence drop-out. The latter can result in the loss of genetic information such that some portion of genes present in the underlying genome are lost from the genome assembly (either completely or partially). This in turn, poses a limitation for the reconstruction of metabolic models using these assemblies, since most published approaches use sequence searches to predict the presence/absence of genes and their associated enzymatic reactions. Therefore, if we are to use public genome data for high-throughput metabolic modelling studies, it is essential to evaluate the expected model accuracies and understand the minimum input genome quality requirements.”

The biological accuracy of the curated ‘complete’ models has been described previously, and this is now noted in the text at lines 320-324:

“Here we performed a systematic analysis leveraging our published curated KpSC models (n=37, (14)), which were generated using completed genome sequences and were therefore considered to represent ‘complete’ models for which the underlying genome sequence contains all genes that are truly present in the genome (note the biological accuracy of these models was reported previously (14) and is not the subject of the current study).”

Throughput the manuscript we not only compare models in terms of the numbers of genes and reactions, but through comparison of binary growth predictions. Specifically, in the Performance Comparison section (Bactabolize vs other approaches) we use comparison of predicted to experimental phenotypes for strain KPPR1 (see response to rev 1 point 4 for details). In the QC Framework section we compare the predictions derived from draft models generated from draft genome assemblies to those derived from the matched ‘complete’ models, and report the concordance as a measure of impact of input assembly quality (lines 309-394). In the final results section (Predictive accuracy of draft models), we generate 10 additional models and compare the growth predictions to matched experimental data (lines 414-433). We view these phenotype prediction comparisons as the ultimate measure of ‘completeness’ with which to assess our models, because these data have direct biological meaning.

2) The authors have not provided evidence or discussion on the accuracy of any metabolic fluxes, which are considered to be crucial for reconstructing metabolic models. Additionally, the authors have not mentioned the importance of non-growth associated maintenance and the criticality of biomass composition analysis, both of which significantly determine the fluxes in the system.

We acknowledge the importance of flux calculations and accurate biomass compositions when using genome-scale models to quantitatively predict growth rates. However, at this stage, the reconstructions developed using Bactabolize are intended for binary predictions and comparisons of growth capabilities on various substrates. The accuracies we report are based on measures of network completion (presence/absence of relevant reactions leading to growth or no-growth phenotypes) rather than specific growth rates. Thus, the models generated by Bactabolize can be used to explore diversity at the strain level in terms of growth capabilities and can serve as a scaffold for building detailed (customized biomass), strain-specific models. Measuring biomass composition and metabolic flux analysis require significant experimental comparisons that are outside the scope of the current study but could be performed for target strains based on reconstructions developed using Bactabolize.

3) It would be interesting to compare the accuracy of the models generated using Bactabolize with those manually curated.

We did exactly this. We compared the manually curated model iKp1289 as part of our benchmarking. Lines 194 – 199:

“We compared the output and performance of Bactabolize to the two previously published tools that can support high-throughput analyses i.e. CarveMe (30) and gapseq (31). To aid interpretation in the context of community standard approaches, we also include a comparison to the popular web-based reconstruction tool, KBase (ModelSEED), and a manually curated metabolic reconstruction of K. pneumoniae strain KPPR1 (also known as VK055 and ATCC 43816, metabolic model named iKp1289) (15).”

Unfortunately, as far as we aware there are currently no other published manually curated models for strains with matched phenotype data that are also not included as part of our pan-reference model (the latter is a key point to ensure a fair comparison of models generated using our pan-reference vs those generated with a universal reference).

4) The authors have not provided evidence or discussion on the accuracy of any metabolic fluxes, which are considered to be crucial for reconstructing metabolic models.

See response to rev 3, point 2.

5) The justification regarding the completeness of the models requires further discussion.

See response to rev 3, point 1.

6) A detailed discussion on the importance of manually curated models would significantly enhance the quality of the manuscript.

This has been added at lines 458-474:

“Traditionally, genome-scale metabolic reconstruction approaches have relied upon significant manual curation efforts. While there will always remain a need for high quality curated models, such resource intensive approaches preclude their application at scale, and have therefore limited analyses to small numbers of individual strains (15, 16). However, automated reconstruction approaches can support the generation and comparison of multiple strain-specific draft models from which meaningful biological insights can be derived (61). Additionally, the quality of curated models is likely to vary depending on their age, level and type of curation, as well as the approach used for preliminary drafting. Indeed it is possible for automated approaches to outperform manually curated models; a draft model for K. pneumoniae KPPR1 generated using Bactabolize with the KpSC pan-v1 reference model outperformed the manually curated iKp1289 model representing the same strain (15). iKp1289 was published in 2017 (6 years prior to this study) and was initially drafted via the KBase pipeline (33), which uses RAST to annotate the sequences with Enzyme Commission numbers. It has been demonstrated several times that the Enzyme Commission scheme has systematic errors (62, 63), leading to a loss in accuracy when compared to the ortholog identification methods used by automated approaches. Consistent with this assertion, our draft KPPR1 model constructed with KBase (without manual curation) was an outlier in terms of the very low number of genes, reactions and metabolites that were included.”

Reviewer #2 (Public Review):

Summary:

Molecular dynamics (MD) data is deposited in public, non-specialist repositories. This work starts from the premise that these data are a valuable resource as they could be used by other researchers to extract additional insights from these simulations; it could also potentially be used as training data for ML/AI approaches. The problem is that mining these data is difficult because they are not easy to find and work with. The primary goal of the authors was to discover and index these difficult-to-find MD datasets, which they call the "dark matter of the MD universe" (in contrast to data sets held in specialist databases).

The authors developed a search strategy that avoided the use of ill-defined metadata but instead relied on the knowledge of the restricted set of file formats used in MD simulations as a true marker for the data they were looking for. Detection of MD data marked a data set as relevant with a follow-up indexing strategy of all associated content. This "explore-and-expand" strategy allowed the authors for the first time to provide a realistic census of the MD data in non-specialist repositories.

As a proof of principle, they analyzed a subset of the data (primarily related to simulations with the popular Gromacs MD package) to summarize the types of simulated systems (primarily biomolecular systems) and commonly used simulation settings.

Based on their experience they propose best practices for metadata provision to make MD data FAIR (findable, accessible, interoperable, reusable).

A prototype search engine that works on the indexed datasets is made publicly available. All data and code are made freely available as open source/open data.

Strengths:

- The novel search strategy is based on relevant data to identify full datasets instead of relying on metadata and thus is likely to have many true positives and few false positives.

- The paper provides a first glimpse at the potential hidden treasures of MD simulations and force field parametrizations of molecules.

- Analysis of parameter settings of MD simulations from how researchers *actually* run simulations can provide valuable feedback to MD code developers for how to document/educate users. This approach is much better than analyzing what authors write in the Methods sections.

- The authors make a prototype search engine available.

- The guidelines for FAIR MD data are based on experience gained from trying to make sense of the data.

Weaknesses:

- So far the work is a proof-of-concept that focuses on MD data produced by Gromacs (which was prevalent under all indexed and identified packages).

As discussed in the manuscript, some types of biomolecules are likely underrepresented because different communities have different preferences for force fields/MD codes (for example: carbohydrates with AMBER/GLYCAM using AMBER MD instead of Gromacs).

- Materials sciences seem to be severely under-represented --- commonly used codes in this area such as LAMMPS are not even detected, and only very few examples could be identified. As it is, the paper primarily provides an insight into the *biomolecular* MD simulation world.

The authors succeed in providing a first realistic view on what MD data is available in public repositories. In particular, their explore-expand approach has the potential to be customized for all kinds of specialist simulation data, whereby specific artifacts are<br /> used as fiducial markers instead of metadata. The more detailed analysis is limited to Gromacs simulations and primarily biomolecular simulations (even though MD is also widely used in other fields such as the materials sciences). This restricted view may simply be correlated with the user community of Gromacs and hopefully, follow-up studies from this work will shed more light on this shortcoming.

The study quantified the number of trajectories currently held in structured databases as ~10k vs ~30k in generalist repositories. To go beyond the proof-of-principle analysis it would be interesting to analyze the data in specialist repositories in the same way as the one in the generalist ones, especially as there are now efforts underway to create a database for MD simulations (Grant 'Molecular dynamics simulation for biology and chemistry research' to establish MDDB' DOI 10.3030/101094651). One should note that structured databases do not invalidate the approach pioneered in this work; if anything they are orthogonal to each other and both will likely play an important role in growing the usefulness of MD simulations in the future.

Currently launching diagnostics on shiny app is not a part of the Cohort Diagnostics step, do we consider adding that part of the code as well?

how are you doing this

This is another use of navigational aid. They list out the steps in order to help viewers find more information about what is being stated.

https://en.wikipedia.org/wiki/Code_coverage https://saucelabs.com/resources/blog/code-coverage-vs-test-coverage

As We May Think

From The Atlantic Monthly, July 1945: 101-108. Reprinted with permission. (c)1945, V. Bush.

As Director of the Office of Scientific Research and Development, Dr. Vannevar Bush has coördinated the activities of some six thousand leading American scientists in the application of science to warfare. In this significant article he holds up an incentive for scientists when the fighting has ceased. He urges that men of science should then turn to the massive task of making more accessible our bewildering store of knowledge. For many years inventions have extended man's physical powers rather than the powers of his mind. Trip hammers that multiply the fists, microscopes that sharpen the eye, and engines of destruction and detection are new results, but the end results, of modern science. Now, says Dr. Bush, instruments are at hand which, if properly developed, will give man access to and command over the inherited knowledge of the ages. The perfection of these pacific instruments should be the first objective of our scientists as they emerge from their war work. Like Emerson's famous address of 1837 on "The American Scholar," this paper by Dr. Bush calls for a new relationship between thinking man and the sum of our knowledge. - The Editor

This has not been a scientist's war; it has been a war in which all have had a part. The scientists, burying their old professional competition in the demand of a common cause, have shared greatly and learned much. It has been exhilarating to work in effective partnership. Now, for many, this appears to be approaching an end. What are the scientists to do next?

For the biologists, and particularly for the medical scientists, there can be little indecision, for their war work has hardly required them to leave the old paths. Many indeed have been able to carry on their war research in their familiar peacetime laboratories. Their objectives remain much the same.

It is the physicists who have been thrown most violently off stride, who have left academic pursuits for the making of strange destructive gadgets, who have had to devise new methods for their unanticipated assignments. They have done their part on the devices that made it possible to turn back the enemy. They have worked in combined effort with the physicists of our allies. They have felt within themselves the stir of achievement. They have been part of a great team. Now, as peace approaches, one asks where they will find objectives worthy of their best.

I

Of what lasting benefit has been man's use of science and of the new instruments which his research brought into existence? First, they have increased his control of his material environment. They have improved his food, his clothing, his shelter; they have increased his security and released him partly from the bondage of bare existence. They have given him increased knowledge of his own biological processes so that he has had a progressive freedom from disease and an increased span of life. They are illuminating the interactions of his physiological and psychological functions, giving the promise of an improved mental health.

Science has provided the swiftest communication between individuals; it has provided a record of ideas and has enabled man to manipulate and to make extracts from that record so that knowledge evolves and endures throughout the life of a race rather than that of an individual.

There is a growing mountain of research. But there is increased evidence that we are being bogged down today as specialization extends. The investigator is staggered by the findings and conclusions of thousands of other workers--conclusions which he cannot find time to grasp, much less to remember, as they appear. Yet specialization becomes increasingly necessary for progress, and the effort to bridge between disciplines is correspondingly superficial.

Professionally our methods of transmitting and reviewing the results of research are generations old and by now are totally inadequate for their purpose. If the aggregate time spent in writing scholarly works and in reading them could be evaluated, the ratio between these amounts of time might well be startling. Those who conscientiously attempt to keep abreast of current thought, even in restricted fields, by close and continuous reading might well shy away from an examination calculated to show how much of the previous month's efforts could be produced on call. Mendel's concept of the laws of genetics was lost to the world for a generation because his publication did not reach the few who were capable of grasping and extending it; and this sort of catastrophe is undoubtedly being repeated all about us, as truly significant attainments become lost in the mass of the inconsequential.

The difficulty seems to be, not so much that we publish unduly in view of the extent and variety of present-day interests, but rather that publication has been extended far beyond our present ability to make real use of the record. The summation of human experience is being expanded at a prodigious rate, and the means we use for threading through the consequent maze to the momentarily important item is the same as was used in the days of square-rigged ships.

But there are signs of a change as new and powerful instrumentalities come into use. Photocells capable of seeing things in a physical sense, advanced photography which can record what is seen or even what is not, thermionic tubes capable of controlling potent forces under the guidance of less power than a mosquito uses to vibrate his wings, cathode ray tubes rendering visible an occurrence so brief that by comparison a microsecond is a long time, relay combinations which will carry out involved sequences of movements more reliably than any human operator and thousands of times as fast-- there are plenty of mechanical aids with which to effect a transformation in scientific records.

Two centuries ago Leibnitz invented a calculating machine which embodied most of the essential features of recent keyboard devices, but it could not then come into use. The economics of the situation were against it: the labor involved in constructing it, before the days of mass production, exceeded the labor to be saved by its use, since all it could accomplish could be duplicated by sufficient use of pencil and paper. Moreover, it would have been subject to frequent breakdown, so that it could not have been depended upon; for at that time and long after, complexity and unreliability were synonymous.

Babbage, even with remarkably generous support for his time, could not produce his great arithmetical machine. His idea was sound enough, but construction and maintenance costs were then too heavy. Had a Pharaoh been given detailed and explicit designs of an automobile, and had he understood them completely, it would have taxed the resources of his kingdom to have fashioned the thousands of parts for a single car, and that car would have broken down on the first trip to Giza.

Machines with interchangeable parts can now be constructed with great economy of effort. In spite of much complexity, they perform reliably. Witness the humble typewriter, or the movie camera, or the automobile. Electrical contacts have ceased to stick when thoroughly understood. Note the automatic telephone exchange, which has hundreds of thousands of such contacts, and yet is reliable. A spider web of metal, sealed in a thin glass container, a wire heated to brilliant glow, in short, the thermionic tube of radio sets, is made by the hundred million, tossed about in packages, plugged into sockets--and it works! Its gossamer parts, the precise location and alignment involved in its construction, would have occupied a master craftsman of the guild for months; now it is built for thirty cents. The world has arrived at an age of cheap complex devices of great reliability; and something is bound to come of it.

II

A record, if it is to be useful to science, must be continuously extended, it must be stored, and above all it must be consulted. Today we make the record conventionally by writing and photography, followed by printing; but we also record on film, on wax disks, and on magnetic wires. Even if utterly new recording procedures do not appear, these present ones are certainly in the process of modification and extension.

Certainly progress in photography is not going to stop. Faster material and lenses, more automatic cameras, finer-grained sensitive compounds to allow an extension of the minicamera idea, are all imminent. Let us project this trend ahead to a logical, if not inevitable, outcome. The camera hound of the future wears on his forehead a lump a little larger than a walnut. It takes pictures 3 millimeters square, later to be projected or enlarged, which after all involves only a factor of 10 beyond present practice. The lens is of universal focus, down to any distance accommodated by the unaided eye, simply because it is of short focal length. There is a built-in photocell on the walnut such as we now have on at least one camera, which automatically adjusts exposure for a wide range of illumination. There is film in the walnut for a hundred exposure, and the spring for operating its shutter and shifting its film is wound once for all when the film clip is inserted. It produces its result in full color. It may well be stereoscopic, and record with spaced glass eyes, for striking improvements in stereoscopic technique are just around the corner.

The cord which trips its shutter may reach down a man's sleeve within easy reach of his fingers. A quick squeeze, and the picture is taken. On a pair of ordinary glasses is a square of fine lines near the top of one lens, where it is out of the way of ordinary vision. When an object appears in that square, it is lined up for its j picture. As the scientist of the future moves about the laboratory or the field, every time he looks at something worthy of the record, he trips the shutter and in it goes, without even an audible click. Is this all fantastic? The only fantastic thing about it is the idea of making as many pictures as would result from its use.

Will there be dry photography? It is already here in two forms. When Brady made his Civil War pictures, the plate had to be wet at the time of exposure. Now it has to be wet during development instead. In the future perhaps it need not be wetted at all. There have long been films impregnated with diazo dyes which form a picture without development, so that it is already there as soon as the camera has been operated. An exposure to ammonia gas destroys the unexposed dye, and the picture can then be taken out into the light and examined. The process is now slow, but someone may speed it up, and it has no grain difficulties such as now keep photographic researchers busy. Often it would be advantageous to be able to snap the camera and to look at the picture immediately.

Another process now in use is also slow, and more or less clumsy. For fifty years impregnated papers have been used which turn dark at every point where an electrical contact touches them, by reason of the chemical change thus produced in an iodine compound included in the paper. They have been used to make records, for a pointer moving across them can leave a trail behind. If the electrical potential on the pointer is varied as it moves, the line becomes light or dark in accordance with the potential.

This scheme is now used in facsimile transmission. The pointer draws a set of closely spaced lines across the paper one after another. As it moves, its potential is varied in accordance with a varying current received over wires from a distant station, where these variations are produced by a photocell which is similarly scanning a picture. At every instant the darkness of the line being drawn is made equal to the darkness of the point on the picture being observed by the photocell. Thus, when the whole picture has been covered, a replica appears at the receiving end.

A scene itself can be just as well looked over line by line by the photocell in this way as can a photograph of the scene. This whole apparatus constitutes a camera, with the added feature, which can be dispensed with if desired, of making its picture at a distance. It is slow, and the picture is poor in detail. Still, it does give another process of dry photography, in which the picture is finished as soon as it is taken.

It would be a brave man who would predict that such a process will always remain clumsy, slow, and faulty in detail. Television equipment today transmits sixteen reasonably good pictures a second, and it involves only two essential differences from the process described above. For one, the record is made by a moving beam of electrons rather than a moving pointer, for the reason that an electron beam can sweep across the picture very rapidly indeed. The other difference involves merely the use of a screen which glows momentarily when the electrons hit, rather than a chemically treated paper or film which is permanently altered. This speed is necessary in television, for motion pictures rather than stills are the object.

Use chemically treated film in place of the glowing screen, allow the apparatus to transmit one picture only rather than a succession, and a rapid camera for dry photography results. The treated film needs to be far faster in action than present examples, but it probably could be. More serious is the objection that this scheme would involve putting the film inside a vacuum chamber, for electron beams behave normally only in such a rarefied environment. This difficulty could be avoided by allowing the electron beam to play on one side of a partition, and by pressing the film against the other side, if this partition were such as to allow the electrons to go through perpendicular to its surface, and to prevent them from spreading out sideways. Such partitions, in crude form, could certainly be constructed, and they will hardly hold up the general development.

Like dry photography, microphotography still has a long way to go. The basic scheme of reducing the size of the record, and examining it by projection rather than directly, has possibilities too great to be ignored. The combination of optical projection and photographic reduction is already producing some results in microfilm for scholarly purposes, and the potentialities are highly suggestive. Today, with microfilm, reductions by a linear factor of 20 can be employed and still produce full clarity when the material is re-enlarged for examination. The limits are set by the graininess of the film, the excellence of the optical system, and the efficiency of the light sources employed. All of these are rapidly improving .

Assume a linear ratio of 100 for future use. Consider film of the same thickness as paper, although thinner film will certainly be usable. Even under these conditions there would be a total factor of 10,000 between the bulk of the ordinary record on books, and its microfilm replica. The Encyclopedia Britannica could be reduced to the volume of a matchbox. A library of a million volumes could be compressed into one end of a desk. If the human race has produced since the invention of movable type a total record, in the form of magazines, newspapers, books, tracts, advertising blurbs, correspondence, having a volume corresponding to a billion books, the whole affair, assembled and compressed, could be lugged off in a moving van. Mere compression, of course, is not enough; one needs not only to make and store a record but also be able to consult it, and this aspect of the matter comes later. Even the modern great library is not generally consulted; it is nibbled at by a few.

Compression is important, however, when it comes to costs. The material for the microfilm Britannica would cost a nickel, and it could be mailed anywhere for a cent. What would it cost to print a million copies? To print a sheet of newspaper, in a large edition, costs a small fraction of a cent. The entire material of the Britannica in reduced microfilm form would go on a sheet eight and one-half by eleven inches. Once it is available, with the photographic reproduction methods of the future, duplicates in large quantities could probably be turned out for a cent apiece beyond the cost of materials. The preparation of the original copy? That introduces the next aspect of the subject.

III

To make the record, we now push a pencil or tap a typewriter. Then comes the process of digestion and correction, followed by an intricate process of typesetting, printing, and distribution. To consider the first stage of the procedure, will the author of the future cease writing by hand or typewriter and talk directly to the record? He does so indirectly, by talking to a stenographer or a wax cylinder; but the elements are all present if he wishes to have his talk directly produce a typed record. All he needs to do is to take advantage of existing mechanisms and to alter his language .

At a recent World Fair a machine called a Voder was shown. A girl stroked its keys and it emitted recognizable speech. No human vocal chords entered into the procedure at any point; the keys simply combined some electrically produced vibrations and passed these on to a loudspeaker. In the Bell Laboratories there is the converse of this machine, called a Vocoder. The loud-speaker is replaced by a microphone, which picks up sound. Speak to it, and the corresponding keys move. This may be one element of the postulated system.

The other element is found in the stenotype, that somewhat disconcerting device encountered usually at public meetings. A girl strokes its keys languidly and looks about the room and sometimes at the speaker with a disquieting gaze. From it emerges a typed strip which records in a phonetically simplified language a record of what the speaker is supposed to have said. Later this strip is retyped into ordinary language, for in its nascent form it is intelligible only to the initiated. Combine these two elements, let the Vocoder run the stenotype, and the result is a machine which types when talked to.

Our present languages are not especially adapted to this sort of mechanization, it is true. It is strange that the inventors of universal languages have not seized upon the idea of producing one which better fitted the technique for transmitting and recording speech. Mechanization may yet force the issue, especially in the scientific field; whereupon scientific jargon would become still less intelligible to the layman.

One can now picture a future investigator in his laboratory. His hands are free, and he is not anchored. As he moves about and observes, he photographs and comments. Time is automatically recorded to tie the two records together. If he goes into the field, he may be connected by radio to his recorder. As he ponders over his notes in the evening, he again talks his comments into the record. His typed record, as well as his photographs, may both be in miniature, so that he projects them for examination.

Much needs to occur, however, between the collection of data and observations, the extraction of parallel material from the existing record, and the final insertion of new material into the general body of the common record. For mature thought there is no mechanical substitute. But creative thought and essentially repetitive thought are very different things. For the latter there are, and may be, powerful mechanical aids.

Adding a column of figures is a repetitive thought process, and it was long ago properly relegated to the machine. True, the machine is sometimes controlled by a keyboard, and thought of a sort enters in reading the figures and poking the corresponding keys, but even this is avoidable. Machines have been made which will read typed figures by photocells and then depress the corresponding keys; these are combinations of photocells for scanning the type, electric circuits for sorting the consequent variations, and relay circuits for interpreting the result into the action of solenoids to pull the keys down.

All this complication is needed because of the clumsy way in which we have learned to write figures. If we recorded them positionally, simply by the configuration of a set of dots on a card, the automatic reading mechanism would become comparatively simple. In fact, if the dots are holes, we have the punched-card machine long ago produced by Hollorith for the purposes of the census, and now used throughout business. Some types of complex businesses could hardly operate without these machines.

Adding is only one operation. To perform arithmetical computation involves also subtraction, multiplication, and division, and in addition some method for temporary storage of results, removal from storage for further manipulation, and recording of final results by printing. Machines for these purposes are now of two types: keyboard machines for accounting and the like, manually controlled for the insertion of data, and usually automatically controlled as far as the sequence of operations is concerned; and punched-card machines in which separate operations are usually delegated to a series of machines, and the cards then transferred bodily from one to another. Both forms are very useful; but as far as complex computations are concerned, both are still in embryo.

Rapid electrical counting appeared soon after the physicists found it desirable to count cosmic rays. For their own purposes the physicists promptly constructed thermionic-tube equipment capable of counting electrical impulses at the rate of 100,000 a second. The advanced arithmetical machines of the future will be electrical in nature, and they will perform at 100 times present speeds, or more.

Moreover, they will be far more versatile than present commercial machines, so that they may readily be adapted for a wide variety of operations. They will be controlled by a control card or film, they will select their own data and manipulate it in accordance with the instructions thus inserted, they will perform complex arithmetical computations at exceedingly high speeds, and they will record results in such form as to be readily available for distribution or for later further manipulation. Such machines will have enormous appetites. One of them will take instructions and data from a whole roomful of girls armed with simple keyboard punches, and will deliver sheets of computed results every few minutes. There will always be plenty of things to compute in the detailed affairs of millions of people doing complicated things.

IV

The repetitive processes of thought are not confined, however, to matters of arithmetic and statistics. In fact, every time one combines and records facts in accordance with established logical processes, the creative aspect of thinking is concerned only with the selection of the data and the process to be employed, and the manipulation thereafter is repetitive in nature and hence a fit matter to be relegated to the machines. Not so much has been done along these lines, beyond the bounds of arithmetic, as might be done, primarily because of the economics of the situation. The needs of business, and the extensive market obviously waiting, assured the advent of mass-produced arithmetical machines just as soon as production methods were sufficiently advanced.

With machines for advanced analysis no such situation existed; for there was and is no extensive market; the users of advanced methods of manipulating data are a very small part of the population. There are, however, machines for solving differential equations--and functional and integral equations, for that matter. There are many special machines, such as the harmonic synthesizer which predicts the tides. There will be many more, appearing certainly first in the hands of the scientist and in small numbers.

If scientific reasoning were limited to the logical processes of arithmetic, we should not get far in our understanding of the physical world. One might as well attempt to grasp the game of poker entirely by the use of the mathematics of probability. The abacus, with its beads strung on parallel wires, led the Arabs to positional numeration and the concept of zero many centuries before the rest of the world; and it was a useful tool--so useful that it still exists.

It is a far cry from the abacus to the modern keyboard accounting machine. It will be an equal step to the arithmetical machine of the future. But even this new machine will not take the scientist where he needs to go. Relief must be secured from laborious detailed manipulation of higher mathematics as well, if the users of it are to free their brains for something more than repetitive detailed transformations in accordance with established rules. A mathematician is not a man who can readily manipulate figures; often he cannot. He is not even a man who can readily perform the transformations of equations by the use of calculus. He is primarily an individual who is skilled in the use of symbolic logic on a high plane, and especially he is a man of intuitive judgment in the choice of the manipulative processes he employs.

All else he should be able to turn over to his mechanism, just as confidently as he turns over the propelling of his car to the intricate mechanism under the hood. Only then will mathematics be practically effective in bringing the growing knowledge of atomistics to the useful solution of the advanced problems of chemistry, metallurgy, and biology. For this reason there will come more machines to handle advanced mathematics for the scientist. Some of them will be sufficiently bizarre to suit the most fastidious connoisseur of the present artifacts of civilization.

V

The scientist, however, is not the only person who manipulates data and examines the world about him by the use of logical processes, although he sometimes preserves this appearance by adopting into the fold anyone who becomes logical, much in the manner in which a British labor leader is elevated to knighthood. Whenever logical processes of thought are employed--that is, whenever thought for a time runs along an accepted groove--there is an opportunity for the machine. Formal logic used to be a keen instrument in the hands of the teacher in his trying of students' souls. It is readily possible to construct a machine which will manipulate premises in accordance with formal logic, simply by the clever use of relay circuits. Put a set of premises into such a device and turn the crank, and it will readily pass out conclusion after conclusion, all in accordance with logical law, and with no more slips than would be expected of a keyboard adding machine.

Logic can become enormously difficult, and it would undoubtedly be well to produce more assurance in its use. The machines for higher analysis have usually been equation solvers. Ideas are beginning to appear for equation transformers, which will rearrange the relationship expressed by an equation in accordance with strict and rather advanced logic. Progress is inhibited by the exceedingly crude way in which mathematicians express their relationships. They employ a symbolism which grew like Topsy and has little consistency; a strange fact in that most logical field.

A new symbolism, probably positional, must apparently precede the reduction of mathematical transformations to machine processes. Then, on beyond the strict logic of the mathematician, lies the application of logic in everyday affairs. We may some day click off arguments on a machine with the same assurance that we now enter sales on a cash register. But the machine of logic will not look like a cash register, even of the streamlined model.

So much for the manipulation of ideas and their insertion into the record. Thus far we seem to be worse off than before--for we can enormously extend the record; yet even in its present bulk we can hardly consult it. This is a much larger matter than merely the extraction of data for the purposes of scientific research; it involves the entire process by which man profits by his inheritance of acquired knowledge. The prime action of use is selection, and here we are halting indeed. There may be millions of fine thoughts, and the account of the experience on which they are based, all encased within stone walls of acceptable architectural form; but if the scholar can get at only one a week by diligent search, his syntheses are not likely to keep up with the current scene.

Selection, in this broad sense, is a stone adze in the hands of a cabinetmaker. Yet, in a narrow sense and in other areas, something has already been done mechanically on selection. The personnel officer of a factory drops a stack of a few thousand employee cards into a selecting machine, sets a code in accordance with an established convention, and produces in a short time a list of all employees who live in Trenton and know Spanish. Even such devices are much too slow when it comes, for example, to matching a set of fingerprints with one of five million on file. Selection devices of this sort will soon be speeded up from their present rate of reviewing data at a few hundred a minute. By the use of photocells and microfilm they will survey items at the rate of a thousand a second, and will print out duplicates of those selected.

This process, however, is simple selection: it proceeds by examining in turn every one of a large set of items, and by picking out those which have certain specified characteristics. There is another form of selection best illustrated by the automatic telephone exchange. You dial a number and the machine selects and connects just one of a million possible stations. It does not run over them all. It pays attention only to a class given by a first digit, then only to a subclass of this given by the second digit, and so on; and thus proceeds rapidly and almost unerringly to the selected station. It requires a few seconds to make the selection, although the process could be speeded up if increased speed were economically warranted. If necessary, it could be made extremely fast by substituting thermionic-tube switching for mechanical switching, so that the full selection could be made in one one-hundredth of a second. No one would wish to spend the money necessary to make this change in the telephone system, but the general idea is applicable elsewhere.

Take the prosaic problem of the great department store. Every time a charge sale is made, there are a number of things to be done. The inventory needs to be revised, the salesman needs to be given credit for the sale, the general accounts need an entry, and, most important, the customer needs to be charged. A central records device has been developed in which much of this work is done conveniently. The salesman places on a stand the customer's identification card, his own card, and the card taken from the article sold--all punched cards. When he pulls a lever, contacts are made through the holes, machinery at a central point makes the necessary computations and entries, and the proper receipt is printed for the salesman to pass to the customer.

But there may be ten thousand charge customers doing business with the store, and before the full operation can be completed someone has to select the right card and insert it at the central office. Now rapid selection can slide just the proper card into position in an instant or two, and return it afterward. Another difficulty occurs, however. Someone must read a total on the card, so that the machine can add its computed item to it. Conceivably the cards might be of the dry photography type I have described. Existing totals could then be read by photocell, and the new total entered by an electron beam.

The cards may be in miniature, so that they occupy little space. They must move quickly. They need not be transferred far, but merely into position so that the photocell and recorder can operate on them. Positional dots can enter the data. At the end of the month a machine can readily be made to read these and to print an ordinary bill. With tube selection, in which no mechanical parts are involved in the switches, little time need be occupied in bringing the correct card into use--a second should suffice for the entire operation. The whole record on the card may be made by magnetic dots on a steel sheet if desired, instead of dots to be observed optically, following the scheme by which Poulsen long ago put speech on a magnetic wire. This method has the advantage of simplicity and ease of erasure. By using photography, however, one can arrange to project the record in enlarged form, and at a distance by using the process common in television equipment.

One can consider rapid selection of this form, and distant projection for other purposes. To be able to key one sheet of a million before an operator in a second or two, with the possibility of then adding notes thereto, is suggestive in many ways. It might even be of use in libraries, but that is another story. At any rate, there are now some interesting combinations possible. One might, for example, speak to a microphone, in the manner described in connection with the speech-controlled typewriter, and thus make his selections. It would certainly beat the usual file clerk.

VI

The real heart of the matter of selection, however, goes deeper than a lag in the adoption of mechanisms by libraries, or a lack of development of devices for their use. Our ineptitude in getting at the record is largely caused by the artificiality of systems of indexing. When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used; one has to have rules as to which path will locate it, and the rules are cumbersome. Having found one item, moreover, one has to emerge from the system and re-enter on a new path.

The human mind does not work that way. It operates by association. With one item in its grasp, it snaps instantly to the next that is suggested by the association of thoughts, in accordance with some intricate web of trails carried by the cells of the brain. It has other characteristics, of course; trails that are not frequently followed are prone to fade, items are not fully permanent, memory is transitory. Yet the speed of action, the intricacy of trails, the detail of mental pictures, is awe-inspiring beyond all else in nature.

Man cannot hope fully to duplicate this mental process artificially, but he certainly ought to be able to learn from it. In minor ways he may even improve, for his records have relative permanency. The first idea, however, to be drawn from the analogy concerns selection. Selection by association, rather than by indexing, may yet be mechanized. One cannot hope thus to equal the speed and flexibility with which the mind follows an associative trail, but it should be possible to beat the mind decisively in regard to the permanence and clarity of the items resurrected from storage.

Consider a future device for individual use, which is a sort of mechanized private file and library. It needs a name, and, to coin one at random, "memex" will do. A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.

It consists of a desk, and while it can presumably be operated from a distance, it is primarily the piece of furniture at which he works. On the top are slanting translucent screens, on which material can be projected for convenient reading. There is a keyboard, and sets of buttons and levers. Otherwise it looks like an ordinary desk.

In one end is the stored material. The matter of bulk is well taken care of by improved microfilm. Only a small part of the interior of the memex is devoted to storage, the rest to mechanism. Yet if the user inserted 5000 pages of material a day it would take him hundreds of years to fill the repository, so he can be profligate and enter material freely.

Most of the memex contents are purchased on microfilm ready for insertion. Books of all sorts, pictures, current periodicals, newspapers, are thus obtained and dropped into place. Business correspondence takes the same path. And there is provision for direct entry. On the top of the memex is a transparent platen. On this are placed longhand notes, photographs, memoranda, all sorts of things. When one is in place, the depression of a lever causes it to be photographed onto the next blank space in a section ~_ the memex film, dry photography being employed

There is, of course, provision for consultation of the record by the usual scheme of indexing. If the user wishes to consult a certain book, he taps its code on the keyboard, and the title page of the book promptly appears before him, projected onto one of his viewing positions. Frequently-used codes are mnemonic, so that he seldom consults his code book; but when he does, a single tap of a key projects it for his use. Moreover, he has supplemental levers. On deflecting one of these levers to the right he runs through the book before him, each page in turn being projected at a speed which just allows a recognizing glance at each. If he deflects it further to the right, he steps through the book 10 pages at a time; still further at 100 pages at a time. Deflection to the left gives him the same control backwards.

A special button transfers him immediately to the first page of the index. Any given book of his library can thus be called up and consulted with far greater facility than if it were taken from a shelf. As he has several projection positions, he can leave one item in position while he calls up another. He can add marginal notes and comments, taking advantage of one possible type of dry photography, and it could even be arranged so that he can do this by a stylus scheme, such as is now employed in the telautograph seen in railroad waiting rooms, just as though he had the physical page before him.

VII

All this is conventional, except for the projection forward of present-day mechanisms and gadgetry. It affords an immediate step, however, to associative indexing, the basic idea of which is a provision whereby any item may be caused at will to select immediately and automatically another. This is the essential feature of the memex. The process of tying two items together is the important thing.

When the user is building a trail, he names it, inserts the name in his code book, and taps it ~out on his keyboard. Before him are the two items to be joined, projected onto adjacent viewing positions. At the bottom of each there are a number of blank code spaces, and a pointer is set to indicate one of these on each item. The user taps a single key, and the items are permanently joined. In each code space appears the code word. Out of view, but also in the code space, is inserted a set of dots for photocell viewing; and on each item these dots by their positions designate the index number of the other item.

Thereafter, at any time, when one of these items is in view, the other can be instantly recalled merely by tapping a button below the corresponding code space. Moreover, when numerous items have been thus joined together to form a trail, they can be reviewed in turn, rapidly or slowly, by deflecting a lever like that used for turning the pages of a book. It is exactly as though the physical items had been gathered together from widely separated sources and bound together to form a new book. It is more than this, for any item can be joined into numerous trails.

The owner of the memex, let us say, is interested in the origin and properties of the bow and arrow. Specifically he is studying why the short Turkish bow was apparently superior to the English long bow in the skirmishes of the Crusades. He has dozens of possibly pertinent books and articles in his memex. First he runs through an encyclopedia, finds an interesting but sketchy article, leaves it projected. Next, in a history, he finds another pertinent item, and ties the two together. Thus he goes, building a trail of many items. Occasionally he inserts a comment of his own, either linking it into the main trail or joining it by a side trail to a particular item. When it becomes evident that the elastic properties of available materials had a great deal to do with the bow, he branches off on a side trail which takes him through textbooks on elasticity and tables of physical constants. He inserts a page of longhand analysis of his own. Thus he builds a trail of his interest through the maze of materials available to him.

And his trails do not fade. Several years later, his talk with a friend turns to the queer ways in which a people resist innovations, even of vital interest. He has an example, in the fact that the outraged Europeans still failed to adopt the Turkish bow. In fact he has a trail on it. A touch brings up the code book. Tapping a few keys projects the head of the trail. A lever runs through it at will, stopping at interesting items, going off on side excursions. It is an interesting trail, pertinent to the discussion. So he sets a reproducer in action, photographs the whole trail out, and passes it to his friend for insertion in his own memex, there to be linked into the more general trail.

VIII

Wholly new forms of encyclopedias will appear, ready-made with a mesh of associative trails running through them, ready to be dropped into the memex and there amplified. The lawyer has at his touch the associated opinions and decisions of his whole experience, and of the experience of friends and authorities. The patent attorney has on call the millions of issued patents, with familiar trails to every point of his client's interest. The physician, puzzled by a patient's reactions, strikes the trail established in studying an earlier similar case, and runs rapidly through analogous case histories, with side references to the classics for the pertinent anatomy and histology. The chemist, struggling with the synthesis of an organic compound, has all the chemical literature before him in his laboratory, with trails following the analogies of compounds, and side trails to their physical and chemical behavior.

The historian, with a vast chronological account of a people, parallels it with a skip trail which stops only on the salient items, and can follow at any time contemporary trails which lead him all over civilization at a particular epoch. There is a new profession of trail blazers, those who find delight in the task of establishing useful trails through the enormous mass of the common record. The inheritance from the master becomes, not only his additions to the world's record, but for his disciples the entire scaffolding by which they were erected.

Thus science may implement the ways in which man produces, stores, and consults the record of the race. It might be striking to outline the instrumentalities of the future more spectacularly, rather than to stick closely to methods and elements now known and undergoing rapid development, as has been done here. Technical difficulties of all sorts have been ignored, certainly, but also ignored are means as yet unknown which may come any day to accelerate technical progress as violently as did the advent of the thermionic tube. In order that the picture may not be too commonplace, by reason of sticking to present-day patterns, it may be well to mention one such possibility, not to prophesy but merely to suggest, for prophecy based on extension of the known has substance, while prophecy founded on the unknown is only a doubly involved guess.

All our steps in creating or absorbing material of the record proceed through one of the senses--the tactile when we touch keys, the oral when we speak or listen, the visual when we read. Is it not possible that some day the path may be established more directly?

We know that when the eye sees, all the consequent information is transmitted to the brain by means of electrical vibrations in the channel of the optic nerve. This is an exact analogy with the electrical vibrations which occur in the cable of a television set: they convey the picture from the photocells which see it to the radio transmitter from which it is broadcast. We know further that if we can approach that cable with the proper instruments, we do not need to touch it; we can pick up those vibrations by electrical induction and thus discover and reproduce the scene which is being transmitted, just as a telephone wire may be tapped for its message.

The impulses which flow in the arm nerves of a typist convey to her fingers the translated information which reaches her eye or ear, in order that the fingers may be caused to strike the proper keys. Might not these currents be intercepted, either in the original form in which information is conveyed to the brain, or in the marvelously metamorphosed form in which they then proceed to the hand?

By bone conduction we already introduce sounds into the nerve channels of the deaf in order that they may hear. Is it not possible that we may learn to introduce them without the present cumbersomeness of first transforming electrical vibrations to mechanical ones, which the human mechanism promptly transforms back to the electrical form? With a couple of electrodes on the skull the encephalograph now produces pen-and-ink traces which bear some relation to the electrical phenomena going on in the brain itself. True, the record is unintelligible, except as it points out certain gross misfunctioning of the cerebral mechanism; but who would now place bounds on where such a thing may lead?

In the outside world, all forms of intelligence, whether of sound or sight, have been reduced to the form of varying currents in an electric circuit in order that they may be transmitted. Inside the human frame exactly the same sort of process occurs.

Must we always transform to mechanical movements in order to proceed from one electrical phenomenon to another? It is a suggestive thought, but it hardly warrants prediction without losing touch with reality and immediateness.

Presumably man's spirit should be elevated if he can better review his shady past and analyze more completely and objectively his present problems. He has built a civilization so complex that he needs to mechanize his records more fully if he is to push his experiment to its logical conclusion and not merely become bogged down part way there by overtaxing his limited memory. His excursions may be more enjoyable if he can reacquire the privilege of forgetting the manifold things he does not need to have immediately at hand, with some assurance that he can find them again if they prove important.

The applications of science have built man a well-supplied house, and are teaching him to live healthily therein. They have enabled him to throw masses of people against one another with cruel weapons. They may yet allow him truly to encompass the great record and to grow in the wisdom of race experience. He may perish in conflict before he learns to wield that record for his true good. Yet, in the application of science to the needs and desires of man, it would seem to be a singularly unfortunate stage at which to terminate the process, or to lose hope as to the outcome.

Huh - must be good code. See GC pause note below, though - sometimes Rust is good. Rust written with a more imperative model would probably cook this alive.

That doesn't make OCaml any less wonderful of a language, though!

I assume this is talking about a dress code

This makes clear that all Code of Ethics apply to any form of technology as well. It is helpful to have a clear statement about technology, given the world that we live in. I wonder though, what about AI? Are there any areas in social work agencies that apply the use of AI in any form? If so, would the ethics change around that? In terms of how to use social media as it relates to my field placement is the use of apps to reach students with information and services about the center in which I intern.

This statement potentially raises a flag for me in terms of power structures and structural inequality. I feel it needs to be more clearly defined in the anticipation of a myriad of dilemmas and possible issues that arise. Does the agency automatically have a position of power? Also, I would think that if an ethical obligation conflicts with relevant laws and regulations could there be a possibility for legal liability ? As someone who is new to the social work field, I am curious to hear about cases where this Code of Ethics was applied and what the outcomes were.

Currently in my field placement we are going through a series of didactics on multicultural supervision. These informative sessions include all staff in a college Wellness Center. Because the college is located in a cultural diverse and historically black area, it is crucial to seek to promote knowledge about ethnic diversity, oppression, and cultural diversity. I am grateful that this is explicitly stated in the Code of Ethics as it is also a research and personal passion of mine to increase awareness, knowledge, action and allyship as it relates to cultural diversity in the clients and communities we serve as social workers.

This area of the code of ethics raises questions for me: when my own moral values conflict with agency policies. I has a client who came through short term residential treatment multiple times in the past two years. It had me thinking about what was missing from his plan to help him sustain long term sobriety from alcohol. This client was a mexican immigrant who did not have a social security number. He couldn't access health insurance or emplyment without it. I spent a lot of hours researching how I could help him get medicaid. The amount of time I spent with one client was beyond the required amount that a counselor would typically spend with a client, but my own morals were telling me that if I did not help him do something differently, he would be back in a month. my task supervisor said this was not my job or part of work study requirement but my morals complelled me to go ahead and see how I could help. I had an internal conflict.

Seeking appropriate consultation, when faced with an ethical dilemma can shed light on the situation and open up new perspectives. Sometimes when we are deep in our own though process there are aspects that we can miss. Sometimes, I feel there isn't going to be a clear answer of what is right and wrong in complex cases, but I feel remaining true to ourselves as professional social workers, we need to be able to lay down our heads at night and say "I did my best."

I do this, ha! I grew up in the south and a lot of words or certain sounds i say have a southern drawl attached to them, subconsiously. When I'm nervous or extremely comfortable with someone, I will slip into a drawl, but when I want to sound more proffesional at work or at my job, I will try to dampen it as much as possible, so I speak clearly and understandibly. I had no idea that it was called code-switching and that other people did it as well.

Violent acts are now being prformed

This section also offers an oppourtunity to explore challenges faced by the profession currently regarding getting bills passed that would benefit oppressed groups. Campaign financing laws have so many loopholes that allow large corporations to control the voice. Code of Ethics could point to more macro work regarding injustices currently happening throughout policy and legislation and discuss the power dynamics when it comes to large corporations controlling campaign financing and turning our representatives into corporate puppets that do their bidding instead of bidding for the people.

Author Response

The following is the authors’ response to the original reviews.

We would like to thank you for considering the above manuscript for publication in eLife and for sending it for review. We would like to thank the editors and reviewers for taking the time to read our manuscript and for their expert comments. These comments have been helpful and have improved our manuscript. We would like to address the following comments:

eLife assessment

This valuable study advances our knowledge of the effects of anxiety/depression treatment on metacognition, demonstrating that treatment increases metacognitive confidence alongside improving symptoms. The authors provide convincing evidence for the state-dependency of metacognitive confidence, based on a large longitudinal treatment dataset. However, it is unclear to what extent this effect is truly specific to treatment, as there was some improvement in metacognitive confidence in the control group.

Thank you for this assessment of the paper. As the change in confidence was not significant among the control group, the last sentence is not factually correct – could we suggest that it be amended to the following: “However, it is unclear to what extent this effect is truly specific to treatment, as changes in metacognitive bias in the iCBT group were not statistically different from those in the control group.”

Reviewer #1 (Public Review)

1) It has been shown previously that there are relationships between a transdiagnostic construct of anxious-depression (AD), and average confidence rating in a perceptual decision task. This study sought to investigate these results, which have been replicated several times but only in cross-sectional studies. This work applies a perceptual decision-making task with confidence ratings and a transdiagnostic psychometric questionnaire battery to participants before and after an iCBT course. The iCBT course reduced AD scores in participants, and their mean confidence ratings increased without a change in performance. Participants with larger AD changes had larger confidence changes. These results were also shown in a separate smaller group receiving antidepressant medication. A similar sized control group with no intervention did not show changes.

The major strength of the study is the elegant and well-powered data set. Longitudinal data on this scale is very difficult to collect, especially with patient cohorts, so this approach represents an exciting breakthrough. Analysis is straightforward and clearly presented. However, no multiple comparison correction is applied despite many different tests. While in general I am not convinced of the argument in the citation provided to justify this, I think in this case the key results are not borderline (p<0.001) and many of the key effects are replications, so there are not so many novel/exploratory hypothesis and in my opinion the results are convincing and robust as they are. The supplemental material is a comprehensive description of the data set, which is a useful resource.

The authors achieved their aims, and the results clearly support the conclusion that the AD and mean confidence in a perceptual task covary longitudinally. I think this study provides an important impact to the project of computational psychiatry.Sspecifically, it shows that the relationship between transdiagnostic symptom dimensions and behaviour is meaningful within as well as across individuals.

We thank the reviewer for their appraisal of our paper and positive feedback on the main manuscript and supplementary information. We agree with the reviewer that the lack of multiple comparison corrections can also justified by key findings being replications and not borderline significance. We have added this additional justification to the manuscript (Methods, Statistical Analyses, page 15, line 568: “Adjustments for multiple comparisons were not conducted for analyses of replicated effects”)

Reviewer #2 (Public Review)

The authors of this study investigated the relationship between (under)confidence and the anxious-depressive symptom dimension in a longitudinal intervention design. The aim was to determine whether confidence bias improves in a state-like manner when symptoms improve. The primary focus was on patients receiving internet-based CBT (iCBT; n=649), while secondary aims compared these changes to patients receiving antidepressants (n=82) and a control group (n=88).

The results support the authors' conclusions, and the authors convincingly demonstrated a weak link between changes in confidence bias and anxious-depressive symptoms (not specific to the intervention arm)

The major strength and contribution of this study is the use of a longitudinal intervention design, allowing the investigation of how the well-established link between underconfidence and anxious-depressive symptoms changes after treatment. Furthermore, the large sample size of the iCBT group is commendable. The authors employed well-established measures of metacognition and clinical symptoms, used appropriate analyses, and thoroughly examined the specificity of the observed effects.

However, due to the small effect sizes, the antidepressant and control groups were underpowered, reducing comparability between interventions and the generalizability of the results. The lack of interaction effect with treatment makes it harder to interpret the observed differences in confidence, and practice effects could conceivably account for part of the difference. Finally, it was not completely clear to me why, in the exploratory analyses, the authors looked at the interaction of time and symptom change (and group), since time is already included in the symptom change index.

We thank the author for their succinct summary of the main results and strengths of our study. We apologise for the confusion in how we described that analysis. We examine state-dependence., i.e. the relationship between symptom change and metacognition change, in two ways in the paper – perhaps somewhat redundantly. (1) By correlating change indices for both measures (e.g. as plotted in Figure 3D) and (2) by doing a very similar regression-based repeated-measures analysis, i.e. mean confidence ~ time * anxious-depression score change. Where mean confidence is entered with two datapoints – one for pre- and one for post-treatment (i.e. within-person) and anxious-depression change is a single value per person (between-person change score). This allowed us to test if those with the biggest change in depression had a larger effect of time on confidence. This has been added to the paper for clarification (Methods, Statistical Analysis, page 14, line 553-559: “To determine the association between change in confidence and change in anxious-depression, we used (1) Pearson correlation analysis to correlate change indices for both measures and, (2) regression-based repeated-measures analysis: mean confidence ~ time * anxious-depression score change, where mean confidence is entered with two datapoints (one for pre- and one for post-treatment i.e., within-person) and anxious-depression change is a single value per person (between-person change score)”).

The analyses have also been reported as regression in the Results for consistency (Treatment Findings: iCBT, page 5, line 197-204: ‘To test if changes in confidence from baseline to follow-up scaled with changes in anxious-depression, we ran a repeated measure regression analyses with per-person changes in anxious-depression as an additional independent variable. We found this was the case, evidenced by a significant interaction effect of time and change in anxious-depression on confidence (=-0.12, SE=0.04, p=0.002)… This was similarly evident in a simple correlation between change in confidence and change in anxious-depression (r(647)=-0.12, p=0.002)”).

2) This longitudinal study informs the field of metacognition in mental health about the changeability of biases in confidence. It advances our understanding of the link between anxiety-depression and underconfidence consistently found in cross-sectional studies. The small effects, however, call the clinical relevance of the findings into question. I would have found it useful to read more in the discussion about the implications of the findings (e.g., why is it important to know that the confidence bias is state-dependent; given the effect size of the association between changes in confidence and symptoms, is the state-trait dichotomy the right framework for interpreting these results; suggestions for follow-up studies to better understand the association).

Thank you for this comment. We have elaborated on the implications of our findings in the Discussion, including the relevance of the state-trait dichotomy to future research and how more intensive, repeated testing may inform our understanding of the state-like nature of metacognition (Discussion, Limitations and Future Directions, page 10, line 378-380: “More intensive, repeating testing in future studies may also reveal the temporal window at which metacognition has the propensity to change, which could be more momentary in nature.”).

Reviewer #3 (Public Review):

1) This study reports data collected across time and treatment modalities (internet CBT (iCBT), pharmacological intervention, and control), with a particularly large sample in the iCBT group. This study addresses the question of whether metacognitive confidence is related to mental health symptoms in a trait-like manner, or whether it shows state-dependency. The authors report an increase in metacognitive confidence as anxious-depression symptoms improve with iCBT (and the extent to which confidence increases is related to the magnitude of symptom improvement), a finding that is largely mirrored in those who receive antidepressants (without the correlation between symptom change and confidence change). I think these findings are exciting because they directly relate to one of the big assumptions when relating cognition to mental health - are we measuring something that changes with treatment (is malleable), so might be mechanistically relevant, or even useful as a biomarker?

This work is also useful in that it replicates a finding of heightened confidence in those with compulsivity, and lowered confidence in those with elevated anxious-depression.

One caveat to the interest of this work is that it doesn't allow any causal conclusions to be drawn, and only measures two timepoints, so it's hard to tell if changes in confidence might drive treatment effects (but this would be another study). The authors do mention this in the limitations section of the paper.

Another caveat is the small sample in the antidepressant group.

Some thoughts I had whilst reading this paper: to what extent should we be confident that the changes are not purely due to practice? I appreciate there is a relationship between improvement in symptoms and confidence in the iCBT group, but this doesn't completely rule out a practice effect (for instance, you can imagine a scenario in which those whose symptoms have improved are more likely to benefit from previously having practiced the task).

We thank the reviewer for commenting on the implications of our findings and we agree with the caveats listed. We thank the reviewer for raising this point about practice effects. A key thing to note is that this task does not have a learning element with respect to the core perceptual judgement (i.e., accuracy), which is the target of the confidence judgment itself. While there is a possibility of increased familiarity with the task instructions and procedures with repeated testing, the task is designed to adjust the difficulty to account of any improvements, so accuracy is stable. We see that we may not have made this clear in some of our language around accuracy vs. perceptual difficulty and have edited the Results to make this distinction clearer (Treatment Findings: iCBT, pages 4-5, lines 184-189: “Although overall accuracy remained stable due to the staircasing procedure, participants’ ability to detect differences between the visual stimuli improved. This was reflected as the overall increase in task difficulty to maintain the accuracy rates from baseline (dot difference: M=41.82, SD=11.61) to follow-up (dot difference: M=39.80, SD=12.62), (=-2.02, SE=0.44, p<0.001, r2=0.01)”.)

However, it is true that there can be a ‘practice’ effect in the sense that one may feel more confident (despite the same accuracy level) due to familiarity with a task. One reason we do not subscribe to the proposed explanation for the link between anxious-depression change and confidence change is that the other major aspect of behaviour that improved with practice did so in a manner unrelated to clinical change. As noted above in the quoted text, participants’ discrimination improved from baseline to follow-up, reflected in the need for higher difficulty level to maintain accuracy around 70%. Crucially, this was not associated with symptom change. This speaks against a general mechanism where symptom improvement leads to increased practice effects in general. Only changes in confidence specifically are associated with improved symptoms. We have provided more detail on this in the Discussion (page 9, lines 324-326: “This association with clinical improvements was specific to metacognitive changes, and not changes in task performance, suggesting that changes in confidence do not merely reflect greater task familiarity at follow-up.”).

2) Relatedly, to what extent is there a role for general task engagement in these findings? The paper might be strengthened by some kind of control analysis, perhaps using (as a proxy for engagement) the data collected about those who missed catch questions in the questionnaires.

Thank you for your comment. We included the details of data quality checks in the Supplement. Given the small number of participants that failed more than one attention checks (1% of the iCBT arm) and that all those participants passed the task exclusion criteria, we made the decision to retain these individuals for analyses. We have since examined if excluding these small number of individuals impacts our findings. Excluding those that failed more than one catch item did not affect the significance of results, which has now been added to the Supplementary Information (Data Quality Checks: Task and Clinical Scales, page 5, lines 181-185: “Additionally, excluding those that failed more than one catch item in the iCBT arm did not affect the significance of results, including the change in confidence (=0.16, SE=0.02, p<0.001), change in anxious-depression (=-0.32, SE=0.03, p<0.001), and the association between change in confidence and change in anxious-depression (r(638)=-0.10, p=0.011)”).

3) I was also unclear what the findings about task difficulty might mean. Are confidence changes purely secondary to improvements in task performance generally - so confidence might not actually be 'interesting' as a construct in itself? The authors could have commented more on this issue in the discussion.

Thank you for this comment and sorry it was not clear in the original paper. As we discussed in a prior reply, accuracy – i.e. proportion of correct selections (the target of confidence judgements) are different from the difficulty of the dot discrimination task that each person receives on a given trial. We had provided more details on task difficulty in the Supplement. Accuracy was tightly controlled in this task using a ‘two-down one-up’ staircase procedure, in which equally sized changes in dot difference occurred after each incorrect response and after two consecutive correct responses. The task is more difficult when the dot difference between stimuli is lower, and less difficult when the dot difference between stimuli is greater. Therefore, task difficulty refers to the average dot difference between stimuli across trials. Crucially, task accuracy did not change from baseline to follow-up, only task difficulty. Moreover, changes in task difficulty were not associated with changes in anxious-depression, while changes in confidence were, indicating confidence is the clinically relevance construct for change in symptoms.

We appreciate that this may not have been clear from the description in the main manuscript, and have added more detail on task difficulty to the Methods (Metacognition Task, page 14, lines 540-542: “Task difficulty was measured as the mean dot difference across trials, where more difficult trials had a lower dot difference between stimuli.”) and Results (Treatment Findings: iCBT, pages 4-5, lines 184-186: “Although overall accuracy remained stable due to the staircasing procedure, participants’ ability to detect differences between the visual stimuli improved.”). We have also elaborated more on how improvements in symptoms are associated with change in confidence, not task performance in the Discussion (page 9, lines 324-326: “This association with clinical improvements was specific to metacognitive changes, and not changes in task performance, suggesting that changes in confidence do not merely reflect greater task familiarity at follow-up”).

4) To make code more reproducible, the authors could have produced an R notebook that could be opened in the browser without someone downloading the data, so they could get a sense of the analyses without fully reproducing them.

Thank you for your comment. We appreciate that an R notebook would be even better than how we currently share the data and code. While we will consider using Notebooks in future, we checked and converting our existing R script library into R Notebooks would require a considerable amount of reconfiguration that we cannot devote the time to right now. We hope that nonetheless the commitment to open science is clear in the extensive code base, commenting and data access we are making available to readers.

5) Rather than reporting full study details in another publication I would have found it useful if all relevant information was included in a supplement (though it seems much of it is). This avoids situations where the other publication is inaccessible (due to different access regimes) and minimises barriers for people to fully understand the reported data.

We agree this is good practice – the Precision in Psychiatry study is very large, with many irrelevant components with respect to the present study (Lee et al., BMC Psychiatry, 2023). For this reason, we tried to provide all that was necessary and only refer to the Precision in Psychiatry study methods for fine-grained detail. Upon review, the only thing we think we omitted that is relevant is information on ethical approval in the manuscript, which we have now added (Methods, Participants, page 11, lines 412-417: “Further details of the PIP study procedures that are not specific to this study can be found in a prior publication (21). Ethical approval for the PIP study was obtained from the Research Ethics Committee of School of Psychology, Trinity College Dublin and the Northwest-Greater Manchester West Research Ethics Committee of the National Health Service, Health Research Authority and Health and Care Research Wales”). If any further information is lacking, we are happy to include it here also.

Reviewer #1 (Recommendations For The Authors):

Minor comments

The first line of the abstract refers to "metacognitive impairments", but the key result is a difference in the mean confidence rating - i.e. could be how participants are using the scale. It's not clear to me that lower mean confidence is necessarily an "impairment" (what's the "right" level of confidence 1-6 for a performance of 70% accuracy). The first line of discussion uses "metacognitive biases" which seems a more accurate description.

We agree that the term bias is more appropriate to use in the Abstract, given that there is not set level to indicate any level of ‘impairment’ associated with under- or over-confidence. This has been changed to ‘biases’ as per the reviewer’s request (Abstract, page 2, line 49). Thank you for this suggestion.

Reviewer #2 (Recommendations For The Authors):

I would suggest being more cautious in the wording relating to the simple effect tests on changes across different treatment arms in the abstract - since no interaction was found it may suggest a difference between arms that is not found significantly. Also since comparison between arms was the secondary aim, first describe interaction effects before simple effects in results.

Thank you for this suggestion, we agree that the lack of significant interaction effect of time and group on confidence is a key finding, which has now been included in the Abstract (page 2, lines 67-71). Additionally, we have rearranged the order of results so the interaction effects precede the simple effects (Results, Comparing iCBT, Antidepressant and Control Groups, page 7, lines 246 – 292:

"When comparing the three groups directly, ANOVA analysis predicting anxious-depression scores with group and time as independent variables revealed a main effect of time (F(1, 1632)=62.99, p<0.001), a main effect of group (F(2, 1632)=249.74, p<0.001), and an interaction effect of group and time (F(2, 1632)=9.23, p<0.001). Examining simple effects in the antidepressant arm, there was a significant reduction in anxious-depression from baseline to follow-up (=-0.61, SE=0.09, p<0.001). Among controls, levels of anxious-depression did not significantly change (=0.10, SE=0.06, p=0.096). Further details of transdiagnostic clinical changes for the antidepressant and controls groups are presented in Figure 4A and Table S4.

Predicting confidence scores using ANOVA analysis with group and time as independent variables revealed a main effect of time (F(1, 1632)=16.26, p<0.001), and no significant main effect of group (F(2, 1632)=2.35, p=0.096). The interaction effect of group and time on mean confidence was not significant (F(2, 1632)=0.60, p=0.550), suggesting that change in confidence did not differ across the three groups. Tests of simple effects revealed that mean confidence significantly increased from baseline (M=3.77, SD=0.88) to follow-up (M=4.07, SD=0.79) in the antidepressant arm (=0.31, SE=0.08, p<0.001) (Figure 4B). Among controls, there was no significant change in confidence from baseline (M=3.68, SD=0.86) to follow-up (M=3.79, SD=0.92) (=0.11, SE=0.07, p=0.103) (Figure 4B).

With respect to task performance, there was a significant main effect of time (F(1, 1632)=15.17, p=0.001) and group (F(2, 1632)=4.56, p=0.011) on mean dot difference when the three groups were included in the model. The interaction effect of time and group on mean dot difference was not significant (F(2, 1632)=1.91, p=0.148), suggesting no differences across the groups in task difficulty changes. In the antidepressant arm, mean dot difference decreased from baseline (M=41.2, SD=13.3) to follow-up (M=35.3, SD=13.1) (=-5.91, SE=1.25, p<0.001), indicating increased task difficulty. There was no significant change in task difficulty among controls from baseline (M=43.0, SD=11.8) to follow-up (M=41.4, SD=13.6) (=-1.64, SE=1.30, p=0.210) (Figure 4C).

While our sample was underpowered to examine individual differences, we conducted an exploratory analysis examining the connection between changes in anxious-depression symptoms and changes in confidence in the antidepressant and controls groups. When examining the effects of time, group and anxious-depression change on mean confidence, there was a significant interaction effect of time and anxious-depression change on mean confidence (F(1, 1626)=4.04, p=0.045), suggesting change in confidence is associated with change in anxious-depression. There was no significant three-way interaction of anxious-depression change, time and group on mean confidence when comparing the three groups (F(2, 1626)=0.08, p=0.928), indicating that the significant association between confidence change and anxious-depression change was not specific to any group. Although not significant, the association between change in confidence and change in anxious-depression was in the expected negative direction in the antidepressant arm (r(80)=-0.10, p=0.381), and among controls (r(86)=-0.17, p=0.111) (Figure 4D)."

Reviewer #3 (Recommendations For The Authors):

Some minor points:

Intro

1) Awkward wording on page 3: 'but little research on how it might impact on metacognition'

We have amended this sentence to make it more clear that relatively less research has been conducted on metacognitive changes following iCBT. We have also provided more detail on a prior study that examined changes in metacognitive beliefs with iCBT, and how this differs from the current study (Introduction, page 3, lines 137-141: “Additionally, iCBT has demonstrated clinical effectiveness in terms of symptom improvement (22–24). While one study found that iCBT modified self-reported metacognitive beliefs (25), it remains unknown if metacognitive confidence in decision-making improves following successful iCBT”).

2) On page 3 the authors note 'but studies typically lacked power to detect effects of antidepressants on cognitive abilities (30-33)' - however, surely this is a problem with this study too, and its relatively small sample of those taking antidepressants?

Thank you for highlighting this. The power comment was in the reference to the larger iCBT arm in this study, but we can appreciate that its placement means that it could be interpreted as being in relation to our smaller antidepressant arm (which we acknowledge is also potentially underpowered). We have reworded this sentence to make it clearer that prior antidepressant studies have not examined the impact of changes in metacognition specifically (Introduction, page 4, lines 147-149: “However, studies examining the impact of antidepressants on cognition have typically focused on cognitive capacities other than metacognition (30–33)”).

Results

3) Fig 2 - please clarify what the error bars indicate.

The error bars represent the standard error around the standardised beta coefficients, which I have added to the description of Figure 2 (page 4, lines 171-172: “The error bars represent the standard error around the standardised beta coefficient”).

4) Awkward wording: 'though it went in the same direction (Figure 4B)'.

This part of the sentence was removed to reduce confusion.

5) This description of the results is somewhat overstated: 'suggesting change in confidence was dependent on change in anxious-depression' (page 7) - this could also be the other way around, or related to a third factor.

We have changed this from ‘dependent’ to ‘is associated with’, which accounts for the unknown directionality and true dependency of confidence changes on changes in anxious-depression (Results, page 7, line 285: “…suggesting change in confidence is associated with change in anxious-depression”).

Methods

6) Please also show how the WSAS in a supplement.

Although this comment is unclear, we have provided additional information on how each item of the WSAS was scored and the overall score range (Supplemental methods, page 2, lines 53-55: “Each WSAS item was scored from 0 ‘not at all’ to 8 ‘very severely’, with overall scores ranging from 0 to 40. Higher WSAS scores indicating higher levels of functional impairment (11)”.

Alan cooper is a Microsoft designer that created visual basic. He created a visual programming language; the code name was "Ruby". This allowed people that user Microsoft/windows to build on what design people made.

Might it be helpful to consider here the data provenance standards, such as those we discussed in context of the Data & Trust Alliance?

I assume this is subject to discussion, and will ultimately be edited to match what is said under "data and dataset"?

This strikes me as overly alarmist. I think it's enough to say "the ability to share and improve the software source code, the preferred form of modifying the work" and leave it at that.

These should come after point 4 right?

Improve phrasing please. This is not very external facing @vinita

```svelte

```

```js / React Hook - sample code to capture mouse coordinates/

import React, { useState, useCallback, useEffect, useRef } from 'react';

function useEventListener(eventName, handler, element = window) { // Create a ref that stores handler const savedHandler = useRef();

// Update ref.current value if handler changes. // This allows our effect below to always get latest handler ... // ... without us needing to pass it in effect deps array ... // ... and potentially cause effect to re-run every render. useEffect(() => { savedHandler.current = handler; }, [handler]);

useEffect(() => { // Make sure element supports addEventListener const isSupported = element && element.addEventListener; if (!isSupported) return;

  // Create event listener that calls handler function stored in ref
  const eventListener = event => savedHandler.current(event);

  // Add event listener
  element.addEventListener(eventName, eventListener);

  // Remove event listener on cleanup
  return () => {
    element.removeEventListener(eventName, eventListener);
  };
},
[eventName, element] // Re-run if eventName or element changes

); }

export default function App() { // State for storing mouse coordinates const [coords, setCoords] = useState({ x: 0, y: 0 });

// Event handler utilizing useCallback ... // ... so that reference never changes. const handler = useCallback( ({ clientX, clientY }) => { // Update coordinates setCoords({ x: clientX, y: clientY }); }, [setCoords] );

// Add event listener using our hook useEventListener('mousemove', handler);

return (

The mouse position is ({coords.x}, {coords.y})

The video we watched for T464 connects to this well and how the children would code switch to complete certain material. (journal vs business letter)

unit 731

nowadays we increasingly only look towards freedom and prosperity in things that rulers promote, not towards wisdom or unity

To remember public static void main(String[] args) repeat it while you sleep. The tranquility of the code will make you fall asleep instantly.

You can copy&paste this code into the next 2 bug fix exercises as well.

This Is Important btw

Integrity is something that is heavily promoted in this class whether that be following the class rules or even the student contract. With this being said it is a privilege to be in the class, so taking the easy way out by using AI or finding some other source to rid of work only takes away from our potential as students. Nothing is learned from someone who cheats, or as a wise man once said, "Give a man a fish, he eats for a day. Teach a man to fish, he eats for a lifetime." Take learning as being taught how to fish and this course will be nothing short of a blast.

Response parameter should cover the payment entity. not QR code entity. please check and change

Fetch a QR Code With id

Fetch a QR Code With id

Close a QR Code

Note: This rebuttal was posted by the corresponding author to Review Commons. Content has not been altered except for formatting.

Learn more at Review Commons

Reply to the reviewers

We have addressed all the queries and suggestions put forth by the reviewers. Major changes include:

1. Expansion of PILOT Functionality and Analysis: We have substantially extended the functionality and analysis capabilities of PILOT, particularly in relation to sample clustering. This enhancement now encompasses the incorporation of statistical tests aimed at identifying cell types and genes associated with distinct patient groups. We applied this expanded feature in an exploratory analysis of sub-clusters within pancreas ductal adenocarcinoma data (PDAC).
2. Clarification of Benchmarking Methods: We have provided clear elucidations of the methods employed for benchmarking PILOT alongside competing methodologies. Our benchmarking approach is notably comprehensive, encompassing twelve different datasets and evaluating four to five competing methods through statistical assessment across three problem domains: clustering, distance measurement, and trajectory estimation. The outcomes of these evaluations consistently demonstrate the superior performance of PILOT's Wasserstein metric across all three problem domains. It is noteworthy that previous studies have often limited their analyses to exploratory evaluations on individual datasets, lacking the level of comprehensive benchmarking undertaken in this study.
3. Examination of Experimental Factors: We have conducted a thorough investigation into the impacts of batch correction, cluster/cell type resolution, and parameter choices used within the PILOT framework.
4. Enhancement of Text Description: We have enhanced the textual descriptions to provide a high-level overview of the PILOT methodology, along with justifications for the methodological decisions made.
5. Improvement of Code and GitHub Repository: To enhance accessibility and promote reproducibility, we have made improvements to the codebase and the associated GitHub repository.

In summary, PILOT stands as a distinctive and all-encompassing framework. It holds the unique distinction of being the sole method offering comprehensive tools for both clustering and trajectory analysis of samples within multiscale single-cell and pathomics data. Moreover, it incorporates statistical methodologies for the interpretation of results. The effectiveness of these tools has been thoroughly validated through the most extensive benchmarking study performed to date on sample-level analysis. The versatility of PILOT is demonstrated through its successful application in exploratory analyses of three distinct datasets: elucidating trajectories in myocardial infarction single-cell RNA-seq data, uncovering trajectories within pathomics data from kidney IgNA patients, and facilitating the clustering of pancreas adenocarcinoma samples. We firmly believe that these contributions hold significant value for the fields of bioinformatics, single-cell genomics, and pathology.

Reviewer #1 (Evidence, reproducibility and clarity):

The paper describes a computational method, PILOT, that uses optimal transport to compute the Wasserstein distance between two individual single-cell samples. It uses PILOT to detect sample (patient) level trajectories and clusters associated with diseases. The method was applied separately to single-cell genomics data and to digital pathology data. The method was applied to several datasets and compared against other tools.

Major comments:

The paper is not easy to follow and should be improved considerably to make it readable and reproducible. Consequently, I was not convinced that the PILOT method is much better than other methods.

We extend our appreciation to the reviewer for their valuable suggestion. We have further refined the manuscript by incorporating a comprehensive and high-level description of our method. This expansion encompasses methodological justifications and clarifications to enhance the overall clarity. Additionally, we wish to emphasize that, to the best of our knowledge, our benchmarking analysis stands as the most comprehensive within the current literature. The results of this analysis unequivocally demonstrate that PILOT surpasses all competing methods in at least one of the various computational analysis tasks, namely clustering, trajectory estimation, and distance evaluation.

Furthermore, we have undertaken significant enhancements in the codebase of PILOT, coupled with a reorganization of the associated GitHub repository. This effort includes the development of in-depth and improved tutorials that faithfully replicate the analyses conducted on datasets related to myocardial infarction, pancreas adenocarcinoma, and pancreas pathomics (https://pilot.readthedocs.io/en/latest/). This changes guarantee the reproducibility of the PILOT framework.

See below for specific changes and additional clarifications.

At first read of the title and abstract, I got the impression that the method analyzes single cell and pathomics data concurrently rather than separately. This should be fixed.

We have changed the text of the abstract and introduction to make clear that PILOT is either applied to single cell or pathomics data independently.

The usage of Wasserstein distance to compute distance between single-cell samples is elegant and is the main strength of this study. Given that PILOT is the main achievement, it should be described more carefully and in a detailed manner.

For example, in the first Results paragraph, "The test indicates for features explaining the predicted pseudotime by fitting either linear or quadratic models" - I could not understand this sentence. Also, which test do the authors refer to? A few sentences down, there is a reference to a Wald test, is that it?

PILOT has three major parts: (1) a method for measuring distance of samples with optimal transport; (2) an patient level unsupervised analysis part (clustering or trajectory analysis) and (3) a part for explaining predicted trajectories/clustering. The sentence mentioned before, refers to the interpretation approach after trajectory analysis. Here, we fit linear, quadratic or linear quadratic models to find association of predicted sample pseudo-time with data features (gene expression values in scRNA or morphological features in pathomics data). This fit can be done for all cells in the data or only for cells from a specific type. In the case of a cell specific fit, we use a Wald test to check if the cell type fit differs from all other cell types in the data, i.e. the gene is associated with the trajectory and the expression changes are specific to the cluster at hand.

While these details were found in the method section, we agree with the referee that they can be better introduced in the main manuscript. We have therefore improved the first subsection of the results and Figure 1 to reflect this.

One of the key aspects of the Wasserstein distance is the cost metric. The determination of the cost metric should be detailed as part of the Results. Have the authors considered and estimated other ways to define the distance?

This is an interesting question. Currently, PILOT uses the Cosine metric. In our revision, we evaluate other metrics (Euclidean, Manhattan, and Chebyshev). This benchmarking indicates that the Cosine and Manhattan performed best regarding the clustering problem (ARI), while Cosine was better than all metrics for the Silhouette statistic; and Cosine and Euclidean performed best regarding AUPR. Therefore, we adopt the Cosine metric in the paper. We include these results in the revised manuscript and in Sup. Fig. 5F-H.

Figure 1 provides a schematic view of PILOT. However, there is no explanation of the notation, which makes it confusing rather than helpful. Also, what is the relationship between J and j, if any?

We understand that the figure 1 was problematic, as it did not introduce the formulation. We have now improved the first sub-section of the results page and figure 1 to improve this.

The motivation and usage of adjusted Rand index (ARI) and Friedman-Nemenyi tests should be provided. Currently, they are not clear, including why those tests are suitable in the cases shown.

The adjusted Rand index is a well known metric to evaluate clustering results when labels are known. Among others this metric has many interesting features as it does not require an association of clusters with class labels. Moreover, it has a correction for random clustering solutions, therefore values lower than zero indicate poor solutions and values of 1 a perfect solution.

The Friedman-Nemenyi test allows us to compare the performance of several algorithms whenever evaluated in the same data sets. Here, the null hypothesis is that all algorithms have the same performance (same ARI statistic). The test is nonparametric and is based on the rank of the algorithm at each data set. This is important, as ARI values (or any other evaluation statistic) are data set specific, e.g. some clustering problems are more difficult than others. By evaluating the rank, the test indicates which methods perform relatively better than others. Moreover, it follows a rigorous statistical framework including correction for multiple testing. This test has an increasing adoption in the machine learning community (Demsar et al., JMLR, https://jmlr.org/papers/v7/demsar06a.html).

We have added phrases with these justifications in the main text (subsection Evaluation of patient-level clustering and trajectory analysis) and included a new section in the materials and methods with more information in the experimental design of the benchmarking analysis.

Fig. 2 the use of method colors should be constant across panels.

We have changed the colors of panels in figure 2A-C (and equivalent panels everywhere else) to avoid confusions.

The proportions method works at least as well as PILOT in 2B and 2C (silhouette and AUPR). Explain why PILOT is better.

The benchmarking analysis shows that PILOT has the highest ARI value (clustering performance) at absolute and ranking levels (Fig. 2A). Moreover the Friedman-Neymeni test indicates this PILOT has significantly higher ranking than all evaluated methods. Regarding Silhouette (distance evaluation) and AUPR (trajectory evaluation) both proportion and PILOT have similar absolute values (Fig. 2B and 2C; panel left), while PILOT has a superior ranking in both cases (Fig. 2B and 2C panel right). Friedman-Neymeni test indicates higher ranking of PILOT than PhEMD for Silhouette and higher ranking of PILOT than PhEMD and pseudo-Bulk regarding trajectory evaluation. The difference in the results on absolute and ranking values can be understood by looking at the statistics in table Table S1. PILOT has highest AUPR in 8 out 12 data sets; proportion has highest values in 5 (including 4 ties with PILOT); proportion-PHATE had 3 best results (including 3 ties with both PILOT and proportions), while PhEMD is best in one data set and Pseudo-bulk in 3 (including 1 tie with PILOT). Altogether, PILOT obtained a higher or equal AUCPR in 9 out of the 12 data sets. We have also changed Fig.2A, 2B and 2C to include all data points and to show the mean, as this provides a better visualization of the previously reported results.

Altogether, these results indicate that PILOT outperforms all competing methods in at least one of the evaluated problems (clustering, trajectory and distance estimation) and ranks favorably in all evaluated scenarios. We have changed the manuscript text to reflect these results.

Likewise, Figure 2C,D and Figures S1 and S2 don't show a clear and consistent advantage for PILOT over other methods. Explain what advantage of PILOT do the fraction panels highlight in Fig. 2E and Fig. S3. Fig. 2C is not mentioned in the text.

Figure 2D, 2E, and now figures S2 and S3 represent visualizations of the results, which were statistically evaluated in panels of Fig.2A-2C. As discussed in the previous point, PILOT does perform better than all methods for the clustering problem and performs better or as good as the proportion test on 9 of the 12 evaluated data sets in the trajectory problem. We also have improved the text to include references to all figures in the main text.

I assume Kidney IgAN (text) and Kidney IgA (fig. 2) are the same.

The correct name is IgAN and this has been corrected in Figure 2.

Fig. 3B fix the p-value notation (what is p=1.05E?) and R2 (R square?). Nrte tha both this problem also occurs in other figs. Fig. 3B shows the major cellular changes.

We now adopt the term “R-squared” in the figures. Also, the previous version did not display p-values properly. We apologize for this. This has been fixed now.

Are these changes consistent with known ones? Explain and provide references. Are there cell types that were expected to show a change and did not? Same questions for Fig. 3C wrt genes. Is this an exploratory analysis highlighting interesting candidate genes? If so, it should be described as such.

Cardiac remodeling after myocardial infarction is characterized by loss of cardiomyocytes, infiltration by immune cells (myeloid and lymphocytes) and increase in myofibroblast populations (doi.org/10.1038/s41392-022-00925-z;doi.org/10.3389/fcvm.2019.00026). PILOT indicates these populations, with the exception of lymphocytes, are most relevant at both clustering levels (see Sup. Fig, 6). Particularly important are results from the low granularity analysis, as this indicates particular macrophage/fibroblast sub-populations (SPP1+ Mac. and Myofibroblast) with increase in disease. PILOT could not detect changes in lymphocyte cells, but this is explained by the poor coverage of these cells in the data set (>3%). We have updated the main manuscript to reflect this.

We also explicitly mention that the analysis of genes and cells are exploratory analysis.

The point of Fig. S6 and its major findings should be mentioned in the text (or it can be removed).

We now make the reference to the gene ontology analysis presented in the new Figure S7 more explicit in the text.

Fig. 4B legend - eGFR not GFR. What do the high-low values of Fig. 2B refer to?

We have fixed these points.Hhigh and low values of panel 4B refer to the eGFR.

Fig. S12 is out of order in supp file.

This has been fixed.

AUCPR - explain.

The AUCPR stands for area under the curve of the precision recall (AUCPR) curve. We have now improved the explanation of the evaluation metric in the main text and methods section.

The github looks like work in progress with many internal comments (eg, add ,edit, etc). I could not find the tutorials.

We have removed all the comments, improved the repository organization and code. The tutorials are explicitly mentioned in the main github page (https://github.com/CostaLab/PILOT/) and in readthedocs webpage (https://pilot.readthedocs.io/). It include tutorials replicating analysis with trajectory inference and clustering problems, which are discussed in the manuscript.

In the process of code review, we have noticed that while we could replicate all the analysis, the procedure for selection of healthy cardiomyocyte genes was distinct (gene were ranked by regression model fit p-value) than the analysis of the myofibroblast genes (genes were ranked by the Wald test p-value). As explained before, the Wald test, which compares the expression of the regression model fits across samples, is a more appropriate criteria, as it finds cluster and trajectory specific genes. We have changed the analysis of the cardiomyocyte to make the gene selection to be based on the Wald-test p-value. New results recover other sarcomere related genes (MYBPC3 and MYOM1) as being dysregulated during disease progression. These findings are in accordance with observations made in the original study presenting the data (Kuppe et al. 2022). We have updated Fig.3 and respective genes accordingly.

Minor comments:

Introduction: "Alternatively, trajectory analysis can be performed to uncover disease progression allowing the characterization of early disease events." Citations should be added (some appear later in the text).

We included a reference to PhEMD.

"Currently, there are no analytical methods to compare two single cell experiments from the same tissue from two distinct individuals." There have been several comparisons among data from patients, (e.g. Cain et al, 2023), so the authors should be more careful/accurate in their statements.

We assume that the referee mentions https://www.nature.com/articles/s41593-023-01356-x. Indeed, we were not aware of this recently published study. The manuscript focuses on comparing cell proportion changes (estimated by deconvolution) between distinct phenotypes and does not provide any approach for sample level analysis of single cell data. This is in our view a different problem than the one addressed by PILOT or PhEMD. We refer to it in our manuscript, as its cell community based analysis is an interesting approach for the interpretation of PILOT results.

"Except for PhEMD, all related methods9, 11, 12 require labels of patients for their analysis and cannot be used in the unsupervised analysis " - this sentence comes immediately after describing ref 13, which can be used in unsupervised analysis and accordingly is not cited in this sentence. The authors did well in describing ref 13 (a bioRxiv paper), and its description should come after this sentence.

We changed the text to reflect this.

"These can be clusters", clustered?

Done.

" acquire an injury cell states" remove an.

Done.

"As for scRNA-seq, there is no analytical method which is able to compare two or more histological slides based on morphometric properties of their structures." The sentence seems to refer to pathomics, not to sc data as suggested in "As for scRNA-seq"

This has been rephrased.

"Thus PILOT represents the first approach to detect unknown patient trajectories and clusters" patient clusters were also observed by others (eg ref 13, Cain et al).

This has been rephrased.

Equation 7 - Cosine(Mi,Mi) should be Cosine(Mi,Mj)

Done.

In the beginning of the Results, PILOT is not referred to as a package but as a researcher ("PILOT explores").

This has been rephrased.

Reviewer #1 (Significance):

In general, the paper is a Methods paper. Hence, likely audience includes computational biologists interested in methodologies, not to biologists interested in the actual findings.

Although I am among the likely audience, I was not convinced by the merits of the method, potentially due to the way the paper was written.

I do not have sufficient expertise to check the math.

In this revision, we have significantly enhanced the text to incorporate high-level descriptions of methods tailored for non-computational experts. Additionally, we have refined the description of the benchmarking process, which, as far as our knowledge extends, stands as the most comprehensive in the literature. This comprehensive analysis strongly underscores the statistical superiority of PILOT when compared to other methods. Lastly, PILOT presents an unique framework, encompassing methods for trajectory analysis, clustering, and interpretation of sample-level analyses within the realm of multiscale single-cell genomics and pathomics data.

Reviewer #2 (Evidence, reproducibility and clarity):

Joodaki et al. propose PILOT, a computational method for analysing single-cell genomics and pathomics data. PILOT is a method that enable clustering, trajectory analysis, and feature detection at a patient level using scRNA-seq data. This is an important task and represent the growing application of scRNA-seq to understand diseases and other perturbations to other biological systems. In particular, PILOT enables unsupervised analysis which alleviate the need of patient labels required by many alternative methods. We have the following comments for the authors' consideration.

1. A key consideration in dealing with scRNA-seq data at a patient level is the batch effect in the data. Typically, each patient sample may be treated as a "batch" especially when they are processed separately to obtain a scRNA-seq dataset that are subsequently combined with scRNA-seq datasets from other patients to form a single dataset. Analysing these data without batch correction may lead to the identification of "cell types" and "states" that are driven by batch effect. In Figure 1. PILOT takes a clustered and integrated scRNA-seq data as input for analysis. I wonder how PILOT would behave if there is a strong batch effect in the data and how would the authors propose to handle them?

This is an interesting question. Currently, PILOT is using the batch procedure used in the paper proposing the original data. We evaluate now the impact of batch correction methods implemented in scanpy (Harmony, bknn and Scanorama). We focus here on single cell data, which we have access to the original count matrix (Lupus, COVID, and Diabetes). We observe no impact of the batch correction algorithm in these data sets (see Sup. Fig. 5C-E). These results are now included in the manuscript.

We have noticed however that strong batch effects in the lung cell atlas or the kidney cell atlas.For the lung cell atlas, we observed that single cell data measured from distinct techniques (Seq-well, Drop-seq, 10x 5’ and 10x 3’) or distinct tissue sampling approaches confounded results for all evaluated approaches. Therefore, we restricted the analysis to the technology with more samples (10x genomics 3’) and to lung tissues. This sample selection was previously described in the material and methods. Of note, the use of samples from distinct 10x genomic version kits (v1, v2 or v3) did not impact results. For the kidney cell atlas, we also observed a strong batch between single nuclei and single cell protocols. Here, we opted to focus on the largest cohort of single cell RNA experiments (see Review Fig. 1). Altogether, PILOT and other evaluated methods do require samples to be analyzed with an uniform technique and sampling approaches. We now include a brief discussion about this open point in the “Discussion” section. This is an important topic of future research.

Review Fig. 1. - Data of the Kidney Precision Medicine Project was measured using either single cell or single nucleus protocols. All evaluated methods were affected by the differences in these technologies and could not separate disease status in this data.

1. It appears that the Wasserstein distance (W) matrix of the samples was used for patient clustering and also trajectory analysis. However, most of the figures presented in the manuscript are for trajectory analysis. Since the patient clustering were performed prior to trajectory analysis, could the authors visualize the patient data based on the W before performing disease trajectory estimation?

Indeed, despite the clustering-based analysis (ARI statistics; Fig. 2A) the current manuscript focuses on results of the trajectory analysis. We now include additional features for clustering analysis. This includes heatmap visualizations of the OT distance matrices together with Leiden clustering (Sup. Fig. 1). See points 4 and 5 below for further changes regarding clustering analysis.

1. In trajectory analysis in figure 2D and E, why not use Multi-scale PHATE which appears to be specifically designed for trajectory analysis? The authors also mentioned SCANCell. While these methods require labels of patients for their analysis, it would be interesting to know how well they perform in comparison to PILOT if such information is available.

This is an interesting point. Multiscale-PHATE is based on doing a multi-resolution clustering of the cells. It then applies PHATE (instead of diffusion map) to find a non-linear embedding on the cell proportions across samples and resolutions. While this analysis is presented at Multiscale-PHATE manuscript (Fig. 5), we could not find any code or functionality in their github to replicate this (https://github.com/KrishnaswamyLab/Multiscale_PHATE). Moreover, we were not able to find a function to find the cluster/resolution associations of cells to reimplement the above mentioned analysis following the descriptions of the manuscript. We also contacted authors, but obtained no reply. It is also important to state that Multi-PHATE used a supervised filter to select cell types for further analysis.

Alternatively, we now include an evaluation of the use of cell proportions followed by a PHATE embedding in the trajectory based evaluation, which is close to the method proposed in Multiscale-PHATE. Our benchmarking indicates that Multiscale-PHATE is the third best ranked method being overpassed by proportion and PILOT. Regarding SCANCell, it focuses on the interpretation of cell communities and it uses embedding/distances by exploring PhEMD. Therefore, its performance in the trajectory or clustering performance problem is the same as PhEMD. We refer to these points in the text now.

1. The current design of PILOT appears to assume that there is always a "smooth" trajectory in the data. Is this going to be the case in reality? What if we have a well separated and distinct groupings of the patients and controls data? In the latter case, imposing a trajectory seems artificial. I am also not sure how meaningful the trajectory analysis would be if, biologically, such a smooth transition is not present in the data.

The EMD based distance can be used both for clustering or trajectory analysis. Also, PILOT performed quite well in the clustering problem benchmarking (Fig. 2A). The choice of application lies on the problem at hand. In our view, both the kidney pathomics and the myocardial infarction data (explored in Fig. 3 and Fig. 4) represent medical data with potential disease trajectories. We now expand the PILOT framework to include new visualizations and statistical methods to improve the interpretation of the clusterings (see point 2; Fig. S1; and point 5 and new Fig. 5).

1. The feature analysis is also built on trajectory analysis using regression models. Again, how would this work out if there isn't a smooth trajectory/transition in the data (e.g. the data are obtained from a discrete case-control study)?

We expanded the PILOT framework to also include statistical tests for accessing changes in cell populations and markers for the clustering problem. First, we use a Welch’s t-test to evaluate cell proportion changes associated with detected clusters. Next, we use a differential analysis test from limma to find genes within a cluster, whose gene expression is changing between the two groups of samples for a given cluster of cells. While these are standard approaches in the literature, this further improves the functionality of PILOT as a general framework for patient level analysis. This is now described in PILOT manuscript (Results subsection “Patient level distance with Optimal Transport” and methods).

We also include in the manuscript an explorative analysis of a sub-cluster found in the PDAC data. This analysis could find a population of PDAC patients displaying higher levels of malignant cells and marked by both increase in hypoxia and fibrosis pathways. This example highlights how PILOT can be used to find potentially interesting groups of samples. These are implemented in the main manuscript (Fig. 5). We also include a new tutorial of PILOT with this analysis (see https://pilot.readthedocs.io/).

1. It is not clear from the formulation of PILOT (and also Figure 1) if the cell type labels is required/used or the cluster id of a clustering algorithm was used instead. The author also mentioned that the clustering output does not have much impact on the downstream analysis. I wonder why and if so can we group the data in any way we want for downstream analysis? This can be useful when one would like to focus on certain grouping of cells.

Clustering at the cells (or structure level) is required. For the benchmarking analysis, we have used the cell annotation reported in the original paper, which were derived via clustering analysis. The use of annotated clusters is crucial for interpretation. We also included in the original manuscript an analysis on the impact of the clustering resolution of the Leiden algorithm. This indicates that the change in resolution did not have a high impact in the clustering (ARI) of the samples (Sup. Fig. 5A-B). However, this analysis could not consider any interpretation of results, as cluster labels were not present.

We believe, however, that the granularity of the clustering will impact the interpretation of the sample analysis. To investigate this, we evaluate how using higher level annotation/clustering of the heart myocardial infarction (also reported in Kuppe et al. 2022) impacts our ability to find cell specific changes. We observe similar changes whenever using low resolution clustering (decrease of cardiomyocytes, increase in fibroblasts and myeloid cells). However, this analysis loses a lot of important nuances found in the high resolution clustering (see Sup. Fig. 6). For example, it does not recover the fact that damaged cardiomyocyte populations have a slower decay than healthy myocytes. Or the fact that myofibroblasts has an increase in the latter disease trajectory stage, while progenitor fibroblast cells (Fibro_Scara5) have an increase previous to myofibroblasts. These results show how low resolution clustering can lead to loss of interesting information contained in cellular sub-states or cell sub-populations. This is now discussed in the results subsection ‘PILOT trajectories detect events associated with cardiac remodeling in myocardial infarction’.

Reviewer #2 (Significance):

PILOT is designed for analyzing scRNA-seq data at a patient level. There is a growing application of scRNA-seq to diseases and the development of computational tools for analyzing such data at phenotype level is critical. The key aspect of PILOT compared to other currently available tools is that it enables unsupervised analysis which alleviate the need of patient labels required by many alternative methods.

Thanks for this very positive feedback and constructive comments.

maximum characters stored in a QR

Response code samples are missing

Bring new knowledge graphs to life with ease, and generate lightbulb moments for your teams, using your own data.

Data Graphs is the essential no-code Knowledge Graph Workbench that helps you get started quickly, and avoid the common graph database problems that slow people down.

It is another good definition of DH.

When mind mapping (like a spider web) you branch out your ideas on the main topic. When branching out you can use colors to color code different big ideas or can include pictures/drawings.

Free QR code sites

WORLD Italian cheesemakers microchip parmesan in bid to fight copycats BY EMMET LYONS

AUGUST 18, 2023 / 11:45 AM / CBS NEWS

One of the world's most famous cheeses is also one of the most counterfeited dairy products on the planet.

Competing manufacturers have for years produced cheaper imitations of Italy's parmigiano reggiano, a favored topping for pasta and salad.

FILE PHOTO: A storage area for Parmesan cheese wheels is pictured at storehouse shelf at 4 Madonne Caseificio dell'Emilia dairy cooperative in Modena A storage area for Parmesan cheese wheels is pictured at storehouse shelf at 4 Madonne Caseificio dell'Emilia dairy cooperative in Modena, Italy, February 16 2016. ALESSANDRO BIANCHI / REUTERS Now, the producers of parmigiano reggiano say they have innovated a way to prevent any pretenders from ripping off their product.

Parmigiano Reggiano Consortium (PRC), the association that oversees production of the cheese, says it has been putting microchips in its product as part of a technological trial which would allow consumers to trace parmesan cheese back to its place of origin.

Under special protections designated by the European Union, parmigiano reggiano is the only kind of cheese which can be called parmesan within Europe and it must be made in a small part of northern Italy, including in the provinces of Parma and Reggio Emilia.

PRC are hopeful that their new technology will help to enforce that protected status. Their microchips are about the size of a grain of salt and have been inserted into the labels found on the rind of about 120,000 wheels of parmigiano reggiano. The chips work as scannable food tags with a QR code label (link)

QR code popularity

Why do they hate Greek letters so much in America? Greek letters are concise way of naming values you otherwise should name with some long symbol strings. In addition, Greek argument and value names provide the connection between code and theory making the user more prepared to understand corresponding papers if she had to.

Note: This rebuttal was posted by the corresponding author to Review Commons. Content has not been altered except for formatting.

Learn more at Review Commons

Reply to the reviewers

Reviewer #1 (Evidence, reproducibility and clarity):

In this manuscript the author is presenting a deep-learning model used to predict the development stage of zebrafish embryo. A robust method that can accurately classify a zebrafish into different development stages is highly relevant for many researchers working with zebrafish and hence the importance in developing methods like this is high.

The manuscript is overall ok. However, more data is needed to convince the reader that the method is robust enough to work with other samples in other labs. This would greatly improve the impact of the publication.

We agree with the reviewer and have included in our revised manuscripts additional test data that was acquired at a different laboratory to the training data (Figures 5 - 7).

Page 6.<br /> - How is the data acquired?

Images used to do initial model training are the same as those used in a previous study - the details of image acquisition are contained in the relevant publication (doi: 10.12688/wellcomeopenres.18313.1). However, we have now added “Zebrafish Husbandry” and “Live Imaging” for newly-acquired images. We have added a table (Table 1) listing details of all image data used in the study.

Page 8.<br /> "This indicates that whileKimmelNet can be used successfully with noisier test data than that on which it was trained,there is an upper limit to how noisy the data can be."<br /> - This is an obvious statement there will always be an upper limit on noise.

We agree with the reviewer that this statement is not terribly informative. This section (“KimmelNet’s prediction accuracy is not significantly impacted by moderate levels of additive noise”) has been removed from the revised manuscript in favour of incorporating a section detailing testing of the model on newly-acquired images (“KimmelNet can generalise to previously unseen data”).

Page 9.<br /> - Are only wildtype embryos used? How would this work on different mutants. To evaluate the robustness of the method this it would be valuable to test on some mutant line with known developmental difference from the wild type.

We agree with the reviewer that testing on a mutant line would lend more weight to our findings. For example, the p53-/- zebrafish has a reported, published developmental delay, but we did not have access to that line. However, the developmental delay reported for the p53-/- mutant is virtually indistinguishable from that effected by a temperature change. We therefore focussed our efforts on developmental delay affected by altering incubation temperature only.

Image data.<br /> - I would strongly suggest that the author should include a description of the data in the manuscript. A description of how the data is acquired, microscope, different batches, type of embryos used.

The image data used in the first draft of the manuscript is the same as that used in a previous publication (Jones et al. 2022), which contains all the relevant details the reviewer seeks. However, we have now added the relevant information for the newly-acquired image data.

"Random160translation in the y-direction was avoided due to the aspect ratio of the images (width>161height) - any artifacts introduced by translation in the x-direction would be removed by the162centre crop layer, but this would not be the case for translation in the y-direction."<br /> - Could this be solved by using some border method reflection, repetition or fixed value?

The reviewer is correct that some form of image reflection or repetition could be utilised. However, given the nature of our images, if an embryo is located close to the image boundary, reflection/repetition can result in some odd artefacts, so we minimised the use of such fill methods (also used by the random zoom augmentation layer). We could instead use an arbitrary fixed value, as the reviewer suggested, but finding a value suitable for all images is difficult.

Page 10.<br /> Addition of Noise to Image Data<br /> - This should be added in the training phase. This would probably improve the robustness of the network and also improve the results on the test data.

We agree with the reviewer and have now added a random Gaussian noise layer for data augmentation purposes during model training (see Figure 1).

• Supplementary 3 images with high noise. It is worrying that the network is not able to handle the noise in this figure. Looks like the features that is used to distinguish the developmental stage of the embryo is still clearly seen with this high noise level? Retrain the model with noise as an augmentation to improve this.

As the reviewer suggested, addition of random noise is now incorporated into model training. The new version of the manuscript does not include the same supplemental figures, but instead includes additional testing on newly-acquired data instead.

Reviewer #1 (Significance):

The development of methods like this is highly relevant in the zebrafish community. Staging and evaluating the developmental stage for zebrafish is common and is of interest to the broad community. A lot of this work today is done manually, limiting the throughput, and adding human bias.

The limit of this study is the dataset used for training and evaluation. Firstly, it is not clear about the structure of the data and how it is acquired, different types of fish or imaging setup etc. For a method to be useful to the community it needs to be robust enough to handle different types of fish (transgenic lines). The manuscript would be greatly improved by adding this to the training and evaluation.

We have now added additional datasets for the purposes of evaluating the model.

My expertise is image analysis and machine learning for quantification of biological samples, with focus on zebrafish screening.

Reviewer #2 (Evidence, reproducibility and clarity):

Summary<br /> The paper "Automated staging of zebrafish embryos with KimmelNet" by Barry et al., presents a method to automatically stage developmental timepoints of zebrafish embryos based on convolutional neural networks (CNN). The authors show that a CNN trained on ~20k images can determine time post fertilization on test-image sets with an accuracy on the range of a few hours. This technique undoubtedly has the potential to become very useful for any zebrafish researchers interested in developmental timing as it eases analysis and removes potential subjective bias.

Major comments<br /> In its current form the paper lacks sufficient graph annotations and method descriptions. This makes it hard in places to judge the validity of the claims. I do believe that the presented method can be useful and is likely valid but to be convincing, the authors need to spend more time expanding the methods, justifying their choices of analysis and clarifying figure annotations.

We believe that we have addressed the reviewer’s concerns in this revised manuscript, as detailed in response to the specific points below.

Specific points:<br /> 1) The annotation of the training data is not described and specifically it is unclear how valid the staging of the training data itself is. The authors state in the introduction "the hours post fertilization (hpf) [...] provides only and approximation of the actual developmental stage". It is therefore critical to know how this was accounted for in the annotation of the training data. Since the quality of the training data will ultimately limit the best-case quality of Kimmel Net. The authors need to go into some detail here even though the training data appears to be from another published dataset.

The reviewer raises a valid point – two individual zebrafish embryos that are x hours post-fertilisation are not necessarily at the same developmental stage. However, we believe it is reasonable to assume that two populations of embryos x hours post-fertilisation are, on average, at the same developmental stage and it is this assumption that forms the basis for our approach. Given the inherent variability the reviewer refers to, we are not suggesting that our model would be particularly accurate for staging individual embryos. However, we are very confident (and we believe the data in the manuscript supports this) that given a population of embryos, our model will provide an accurate rate of development. We have added a paragraph (lines 131-141) to address this point.

2) Why were "test predictions fit to a straight line through the origin". On the one hand this makes sense (since the slope would indicate the correspondence) but it should be clarified why an intercept was omitted in the fit. After all it is unclear if Kimmel net correctly identifies 0Hpf embryos.

The reviewer makes a valid point – we do not know what predictions KimmelNet would produce for images of embryos closer to 0 hpf. However, we felt an equation of the form y=mx was a reasonable choice for two reasons. First of all, it matches the form of the Kimmel equation, which, despite its flaws, we are using as a benchmark of sorts – the absence of a y intercept makes comparisons with the Kimmel equation straightforward. Secondly, a “perfect” model would produce a straight line fit with y=x, so the lack of a y intercept seemed a reasonable constraint to impose. We have added some brief text (lines 103-105) to clarify this choice.

3) The methods do not list how the mean of the absolute error was calculated from 3B/C. I think this should be the mean of the absolute error (not the mean of the error) but in that case the numbers listed in the text appear rather small given the histograms in 3 B/C. A clear statement in the methods would clarify this issue.

We have now added a “Statistical Analysis” section under Materials & Methods to detail exactly what was used to calculate the values given for error analysis. We have calculated the mean of the error, not the mean of the absolute error, as we wish to illustrate that the mean is close to zero. We have used the standard deviation of the errors to illustrate that there is a significant spread in the error values, as depicted in Figure 3C and D.

Minor comments<br /> 1) The Y-axis in Figure 2B is simply labeled "Loss" - what is the unit of this loss? HPF? Or HPF^2? This is important for judging the quality of the fit

We thank the reviewer for drawing our attention to this omission. The loss is hpf2 (mean squared error) and we have updated the plot to reflect this.

2) Figure 3 B. I would suggest changing the labels of the confidence intervals in the legend. "Inner and outer" is already clear from the figure itself, so labels that state that these are derived from n=100 vs. n=20 test image sized samples would be more useful to the reader

We thank the reviewer for this suggestion – we have updated the figure legend accordingly.

Referees cross-commenting

I concur with comments issued by the other reviewers. I think it will be especially important to address the comments related to testing the method on mutants (Reviewer #1) and training the model in the presence of noise to increase its robustness (Reviewers #1 and #3) as well as addressing the overall annotation/generation of the training data (Reviewers #1 and #2).

We believe we have now addressed all of these concerns. The model has been retrained with additional data augmentation incorporating random noise, tested on newly-acquired data and we have added tables summarising the details of all image data used in this study.

I think these points will be critical to make the paper useful by increasing transparency and ensuring reproducibility in other labs with different imaging conditions, strains, mutants, etc.

Reviewer #2 (Significance):

Developmental delay is a common occurrence that can be caused by genetic and environmental background effects. It is therefore highly desirable to properly quantify this variable. The work presented here makes an important step in this direction, by allowing to quantify developmental timepoints independent of subjective staging. This speeds up analysis, increases reproducibility and enhances rigor. However, as my comments above indicate, the significance also depends on the ability of potential users to judge the quality of the work. Once those issues have been addressed, I think the work will be of broad interest to the developmental biology community, first and foremost labs utilizing the zebrafish model. However, as the authors state, the presented model architecture could be trained with the data from other species as well.

Expertise: Zebrafish, quantitative analysis, behavior, neuroscience

We thank the reviewer for their positive feedback.

Reviewer #3 (Evidence, reproducibility and clarity):

Summary:

Properly staging embryos of zebrafish embryos is important, yet provides challenging since it can depend on many factors, such as temperature, water quality, fish population density, etc. Here, the authors provide a deep-learning-based model called KimmelNet that allows the prediction of the age of zebrafish embryos, using 2D brightfield images. The technique is robust to weak measurement noise and can also be used to identify developmental delays from a very small number of experimental data.

The code is accessible to the reader, open-source and should be useable by experimentalists without huge effort.

Major comments:

I suggest retraining the model and application of the model to additional data for the following reasons:<br /> • Why did the authors not train for (high) measurement noise and heterogeneous background illumination? Would that not make the model more robust? In principle, creating training should not be considerably harder than before, since the manipulation of the images has been already shown in the manuscript and the authors just need to run it again on the HPC cluster. If there are no technical or administrative constraints (access to the cluster, computational effort, high costs, etc.), the authors should retrain their model.

We thank the reviewer for this suggestion. As detailed in Figure 1, with a view to making the model more robust, we have now added several more layers of data augmentation, including the addition of random noise, and retrained our model.

• For Fig. S2 and S3 it is not clear if there is such a strong deviation from the Kimmel equation due to measurement noise or due to the background illumination. The saliency maps appear as if they are mainly affected by the illumination, and maybe less by the noise. Would it be possible to apply the model to a case without artificial noise, but with heterogeneous background illumination to identify what has a bigger impact?

We thank the reviewer for this suggestion. We have now replaced the “artificial” examples used in the previous version of the manuscript with newly-acquired data (Figure 5), which exhibits different characteristics to that used for training.

Additionally, the authors need to clarify what exactly they are comparing in this manuscript and rework their interpretation of their findings:<br /> • When comparing the predictions between KimmelNet and the Kimmel equation, the authors use an equation of the form y=mx. Could the authors please elaborate on why they introduce the constraint of y(0)=0? It might be naturally given by the so-called Kimmel equation, but by looking at Fig 3a, it seems like an equation of the form y=mx+a would be more appropriate and it appears like KimmelNet introduces an offset of around a=2h for 25 Celsius. The authors need to discuss this.

The main rationale for using an equation of the form y=mx is to be consistent with the Kimmel equation (see lines 103-105). The reviewer is correct that an equation of the form y=mx+c may well produce a better fit to the data, but omitting a y intercept makes comparison with the Kimmel Equation trivial.

• In lines 5-8 the authors say that developmental stage progression does not only depend on temperature, but also on population density, water quality etc. and they explain that usually not only hpf, but also staging guides based on morphological criteria are used! If that is true, how good is their training data set that only uses hpf and not the other important guides? How did the authors test that these factors have no impact on their training data?

We have now added a paragraph (lines 131-141) to address this point.

Since this tool has the potential to have a big impact on zebrafish research, it would be nice to provide some examples of how exactly this could be achieved:<br /> • Could the authors discuss how exactly their tool is useful to experimentalists? Is it the idea that if an experimentalist wants to investigate an embryo of a particular stage, they apply KimmelNet to images of embryos to identify the stage of the embryo and only then undertake their planned experiment? Is that a realistic undertaking?

As is evidenced by the errors presented in Figure 3C & D, testing KimmelNet on individual images of embryos may well result in a large error in the predicted hpf. As such, it is not appropriate to use the tool in such a manner. However, to modify the example provided by the reviewer, should an experimentalist have a population of embryos they wished to stage, then yes, KimmelNet would certainly be an appropriate tool for doing so.

• Would it be possible to provide a tutorial (or even video tutorial if such skills are available in the group of authors) that provides real examples of how to apply the technique? This would make it easier for people without advanced Python/Deep-Learning skills to use the tool, hence improving the impact of KimmelNet.

A lack of user-friendly interfaces for applying deep learning methods in biology is well-documented – basic knowledge of python and tools like jupyter notebooks are often necessary (https://doi.org/10.1038/s41592-023-01900-4). However, we have endeavoured to make the running of KimmelNet as easy as possible for new users. A jupyter notebook instance can be run on Binder with absolutely no set-up required. To run KimmelNet on their own data, biologists just need to download the Git repo and replace the test images with their own data. We have updated the landing page on the GitHub repo to provide more specific step-by-step instructions for each of these tasks. We will also endeavour to upload our model to the BioImage Model Zoo (https://bioimage.io/#/) to further increase accessibility.

I am very critical towards equation 1. Please note that I don't think this has any impact on the quality of the technique provided in this manuscript and the significant flaws can already be found in Kimmel 1995 (which is not under review here). That is why I suggest rewriting of this manuscript to not support an over-interpretation of this equation.<br /> • I do not think that the Kimmel equation is an established term. At least a Google Scholar Search for "Kimmel equation" only gives one result: the preprint of this manuscript.<br /> • The equation has no mathematical meaning regarding its units (subtracting temperature and a unitless value). I also very rarely see equations with Degrees Celsius and not Kelvin.<br /> • Additionally, the equation provides a linear relationship between the development time and temperature h(T) and in Kimmel et al, it is shown that this is only true for 25-33 Celsius. Such a linearisation is not very surprising for a small temperature range, but I am not sure how true it is for higher temperature differences. Hence, I feel that it is very bold to give a specific name to such an equation, giving it an importance that it does not deserve.

We appreciate the reviewer’s concerns and have removed explicit references to “The Kimmel Equation”, without substantively changing the content of the manuscript.

Minor comments:

• For the measurement noise cases it would be nice to have some example images of fish with the specific noise levels in Fig S1 and Fig S2.

We have now removed the “synthetic” additive noise test data, previously depicted in Figures S1-3, in favour of newly-acquired images in Figures 5-7.

• Could the authors hypothesize why they predict a slower dynamic for 25 Celsius than predicted by the Kimmel equation?

Referring to Figure 2 in Kimmel et al (1995), it is apparent that the straight lines are by no means perfect fits to the datapoints. In Fig 2A in particular, some datapoints for the 25C data fall well below the line fit. While the published equation suggests a rate of development 80.5% of the rate at 28.5C, according to Fig 2A, an alternative line fit could give a developmental rate as low as 70-75%, which would be in agreement with our data.

Reviewer #3 (Significance):

Strengths of the study:

An easy-to-use method to automatically stage zebrafish embryos and identify differences in the developmental stage is very important for the zebrafish community and the technique in this manuscript definitely novel. The tool is can be used without large effort and the authors suggest that it can also find applications beyond zebrafish embryos. Hence, it is not only interesting to the zebrafish community, but to a broader developmental biology audience.

Weakness of the study:<br /> The main weakness of the manuscript is in the training data used for the deep-learning model and the apparent large impact of heterogeneous background illumination. If that is not solved, it is unclear if this technique will find an application in the zebrafish community.

We believe this weakness has now been addressed by incorporating additional data augmentation measures in the training process and testing the model on newly-acquired data.

Field of expertise of the reviewer: Image Analysis, Mathematical Modelling, Biological Physics. While I have limited experience in deep learning, I cannot evaluate the specific details of the network architecture. I also have no experience in zebrafish research.

Note: This preprint has been reviewed by subject experts for Review Commons. Content has not been altered except for formatting.

Learn more at Review Commons

Referee #3

Evidence, reproducibility and clarity

Summary:

Properly staging embryos of zebrafish embryos is important, yet provides challenging since it can depend on many factors, such as temperature, water quality, fish population density, etc. Here, the authors provide a deep-learning-based model called KimmelNet that allows the prediction of the age of zebrafish embryos, using 2D brightfield images. The technique is robust to weak measurement noise and can also be used to identify developmental delays from a very small number of experimental data.

The code is accessible to the reader, open-source and should be useable by experimentalists without huge effort.

Major comments:

I suggest retraining the model and application of the model to additional data for the following reasons:<br /> - Why did the authors not train for (high) measurement noise and heterogeneous background illumination? Would that not make the model more robust? In principle, creating training should not be considerably harder than before, since the manipulation of the images has been already shown in the manuscript and the authors just need to run it again on the HPC cluster. If there are no technical or administrative constraints (access to the cluster, computational effort, high costs, etc.), the authors should retrain their model.<br /> - For Fig. S2 and S3 it is not clear if there is such a strong deviation from the Kimmel equation due to measurement noise or due to the background illumination. The saliency maps appear as if they are mainly affected by the illumination, and maybe less by the noise. Would it be possible to apply the model to a case without artificial noise, but with heterogeneous background illumination to identify what has a bigger impact?

Additionally, the authors need to clarify what exactly they are comparing in this manuscript and rework their interpretation of their findings:<br /> - When comparing the predictions between KimmelNet and the Kimmel equation, the authors use an equation of the form y=mx. Could the authors please elaborate on why they introduce the constraint of y(0)=0? It might be naturally given by the so-called Kimmel equation, but by looking at Fig 3a, it seems like an equation of the form y=mx+a would be more appropriate and it appears like KimmelNet introduces an offset of around a=2h for 25 Celsius. The authors need to discuss this.<br /> - In lines 5-8 the authors say that developmental stage progression does not only depend on temperature, but also on population density, water quality etc. and they explain that usually not only hpf, but also staging guides based on morphological criteria are used! If that is true, how good is their training data set that only uses hpf and not the other important guides? How did the authors test that these factors have no impact on their training data?

Since this tool has the potential to have a big impact on zebrafish research, it would be nice to provide some examples of how exactly this could be achieved:<br /> - Could the authors discuss how exactly their tool is useful to experimentalists? Is it the idea that if an experimentalist wants to investigate an embryo of a particular stage, they apply KimmelNet to images of embryos to identify the stage of the embryo and only then undertake their planned experiment? Is that a realistic undertaking?<br /> - Would it be possible to provide a tutorial (or even video tutorial if such skills are available in the group of authors) that provides real examples of how to apply the technique? This would make it easier for people without advanced Python/Deep-Learning skills to use the tool, hence improving the impact of KimmelNet.

I am very critical towards equation 1. Please note that I don't think this has any impact on the quality of the technique provided in this manuscript and the significant flaws can already be found in Kimmel 1995 (which is not under review here). That is why I suggest rewriting of this manuscript to not support an over-interpretation of this equation.<br /> - I do not think that the Kimmel equation is an established term. At least a Google Scholar Search for "Kimmel equation" only gives one result: the preprint of this manuscript.<br /> - The equation has no mathematical meaning regarding its units (subtracting temperature and a unitless value). I also very rarely see equations with Degrees Celsius and not Kelvin.<br /> - Additionally, the equation provides a linear relationship between the development time and temperature h(T) and in Kimmel et al, it is shown that this is only true for 25-33 Celsius. Such a linearisation is not very surprising for a small temperature range, but I am not sure how true it is for higher temperature differences. Hence, I feel that it is very bold to give a specific name to such an equation, giving it an importance that it does not deserve.

Minor comments:

• For the measurement noise cases it would be nice to have some example images of fish with the specific noise levels in Fig S1 and Fig S2.
• Could the authors hypothesize why they predict a slower dynamic for 25 Celsius than predicted by the Kimmel equation?

Significance

Strengths of the study:

An easy-to-use method to automatically stage zebrafish embryos and identify differences in the developmental stage is very important for the zebrafish community and the technique in this manuscript definitely novel. The tool is can be used without large effort and the authors suggest that it can also find applications beyond zebrafish embryos. Hence, it is not only interesting to the zebrafish community, but to a broader developmental biology audience.

Weakness of the study:

The main weakness of the manuscript is in the training data used for the deep-learning model and the apparent large impact of heterogeneous background illumination. If that is not solved, it is unclear if this technique will find an application in the zebrafish community.

Field of expertise of the reviewer:

Image Analysis, Mathematical Modelling, Biological Physics. While I have limited experience in deep learning, I cannot evaluate the specific details of the network architecture. I also have no experience in zebrafish research.

Example of sematic error

https://www.ebay.com/itm/304964453291

Vintage Wood Library Card File Table Top Index Dovetail Gaylord Bros 13x7x4

Listed in Spring 2023 for $24.74 + $56.80 shipping from Kingwood, TX to Los Angeles (emailed about being high on 2023-09-10)

Coupon code for 10%: WINDSOFAUTUMN

Visual Studio Code est en anglais, comment le passer à une version française, svp.

turn around the title of the color code by 180 degrees. All axis/legend titles should "look" into the same direction. Good scientific convention to avoid headaches ;-)

TODO - Update

After code. add a line to link Action description and action parameter values below

Action.code, Action.data, Action.error, and action paramater values should be under same heading for Action. Should be Linked above after linkNewAccount code.

x

It's a syntactical inversion of the imperative statement oriented javascript code.

Author Response

Evaluation Summary:

The manuscript shows that retinal ganglion cell light responses in awake mice differ substantially from those under two forms for anesthesia and previously attained ex vivo recordings. This difference is central to our understanding of how ganglion cell responses relate to behavior. There are a few technical issues and issues about how the work is presented that could be strengthened.

We thank the reviewers for their constructive comments. We have addressed all the issues, and added substantially more data and analysis results in the revised manuscript, further supporting our findings that awake responses are larger, faster, and more linearly decodable in the mouse retina than those responses under anesthesia or ex vivo.

Reviewer #1 (Public Review):

This paper compares output signals from the mouse retina in three conditions: awake mice, anaesthetized mice, and isolated retinas. The paper reports substantial differences, particularly between awake and either of the other conditions. Retinal signaling has been well studied using ex vivo preparations, with an assumption that the findings from those studies can be carried over to how the retina operates in vivo. The results from this paper at a minimum indicate a need to be cautious about that assumption. There are several technical issues that need testing or further explanation, and several issues about the presentation that could be clarified.

Spike sorting

The paper does not describe any control analyses that test for contamination in spike sorting. These are needed to evaluate the work.

We have reported the details of our spike sorting procedure in the revised manuscript (Data Analysis section in Methods and Figure 1). In short, single-units were identified by clustering in principal component space, followed by manual inspection of spike waveform (triphasic as expected from axonal signals; e.g., revised Figure 1F-H; Barry, 2015) as well as auto- and cross-correlograms (minimal inter-spike interval above 1 ms for a refractory period; e.g., revised Figure 1I-K). A small fraction of visually responsive cells (20/282, awake; 21/325, isoflurane; 1/103, FMM) had a small fraction of interspike intervals below 2 ms; but, whether or not including them in the analysis did not affect our main conclusions.

Light levels

The paper argues that differences in light level cannot account for the results. According to the methods, light levels were about two-fold higher at the retina in array recordings as compared to the front of the eye for in vivo recordings. The main text indicates that they differ less, it's not clear why the numbers in the main text and methods are different. Aside from this issue, this comparison does not consider the loss of light between the front of the eye and the retina. It is crucial that the paper provide a more detailed description of light levels. This should include converting those light levels to units that include the spectral output of the light source used (e.g. to isomerizations per rod or cone per second).

The maximum light intensity of our in vivo setup was 31.3 mW/m2 (with 15.9 mW for UV LED and 15.4 mW/m2 for blue LED). Following the suggestion by the reviewer, we calculated the photon flux on the mouse retina in vivo by taking into account the loss of light by the eye optics. In short, assuming 50% and 68% transmittance at 365 nm and 454 nm, respectively (Jacobs & Williams 2007), the pupil size of 1 mm and the retinal diameter of 4 mm with the stimulus covering 73° in azimuth and 44° in elevation, we obtained the photon flux on the mouse retina in vivo as 3.81×103 and 6.64×103 photons/s/μm2 for UV and blue light, respectively. Assuming a total photon collecting area of 0.2 μm² for cones and 0.5 μm² for rods (Nikonov et al. 2006), and a relative sensitivity of rods, S- and M-cones to be [UV, Blue]=[25, 60], [90, 0], [25, 60]%, respectively (Jacobs & Williams 2007), we then estimated the photoisomerization (R) rate as: 2.5×103 R/rod/s, 0.7×103 R/S-cone/s, and 1.0×103 R/M-cone/s.

In contrast, the maximum light intensity of the in vitro set up was 36 mW/m2 as reported in Vlasiuk and Asari (2021). The photon flux on the isolated retina was then estimated to be around 9×104 photons/s/μm2 (under the assumption that the white light from a CRT monitor is centered around 500 nm). Assuming the sensitivity of rods, S- and M-cones to be 40, 2 and 40%, respectively, we then obtained 4×104 R/rod/s, 2×103 R/S-cone/s, and 4×104 R*/Scone/s.

Thus, the light intensity level was about ten times larger for the in vitro recordings than for the in vivo recordings. The amount of light reaching the retina in the awake condition should also be somewhat smaller than that under anesthesia due to pupillary reflexes. Past studies suggest that the darker the stimulus is, the slower the kinetics is and the smaller the response is for RGCs in an isolated retina (Wang et al 2011). Thus, the light intensity difference cannot simply account for the higher firing and faster kinetics in the awake condition than ex vivo or in the anesthestized condition.

We have revised the manuscript accordingly.

Comparison with other work

The authors accurately point out that there is not much prior work on retinal outputs in awake animals. The paper, however, minimally describes the work that does exist. The Hong et al. (2018) paper, in particular, should be discussed. There are several differences between the results of that paper and the present paper. These include the fraction of recorded cells that are DS cells, and the maintained firing rates (though this does not appear to be studied systematically in Hong et al.).

In the discussion section of the revised manuscript, we clarified connections to the existing studies on the retinal activity in vivo. To our knowledge, none of the past studies provided descriptive statistics on the awake RGC response properties (Hong et al., 2018; Schroeder et al., 2020; Sibille et al., 2022). Nevertheless, consistent with our study, we can see high baseline activity in the reported examples from C57BL6 mice (Figure 3C, Schroeder et al. 2020; Figure S7h, Sibille et al. 2022).

Hong et al (2018), in contrast, reported somewhat different as pointed out by the reviewer. Firstly, they found a relatively low baseline activity in RGCs of albino CD1 mice. We think that this is likely due to general impairments of the vision/retina associated with albinism. While equipped with normal electroretinogram signals, CD1 mice showed no optomotor response and a reduced number of rods (Abdeljalil et al 2005; Brown et al 2007). This suggests a certain level of retinal dysfunction in these mice. Secondly, Hong et al (2018) reported a higher fraction of direction-selective RGCs in their recordings (>50% at a DS index threshold of 0.3). This is even higher than one would expect from anatomical and physiological studies ex vivo on BL6 mice (about a third; Sanes and Masland, 2015; Baden et al., 2016; Jouty et al 2013). Besides the effect of albinism, we think that this overrepresentation of DS cells in Hong et al (2018) arose as a consequence of the low baseline activity. As discussed above, the higher the baseline activity, the lower the DS/OS index by definition (Eq.(3) in Methods). Indeed we found much more cells with high DS/OS index values in our anesthetized data than in awake ones (42-54% vs 17% at an index value threshold of 0.15; Figure 2), even though these recordings were done in the same experimental set up.

A related issue is that there are a few comparisons of ex vivo RGC responses with behavioral sensitivity. Smeds et al. (2019) is one example. More generally, the long-standing observation that dark-adapted sensitivity approaches limits set by Poisson fluctuations in photon absorption, and that prior RGC measurements are consistent with this result, is hard to explain if the RGCs are firing at high spontaneous rates under these conditions. RGC responses will certainly change with light level, but this merits discussion in the paper.

As the reviewer pointed out, the retina may employ different coding principles under different light levels. In a scotopic condition, ex vivo studies reported a high tonic firing rate for OFF RGC types (~50 Hz, OFF sustained alpha cells in mice; Smeds et al 2019; ~20 Hz, OFF parasol cells in primates; Ala-Laurila and Rieke, 2014), while a low tonic firing for ON cell types (<1Hz for both ON sustained alpha in mice and ON parasol in primates). These ON cells were shown to be responsible for light detection by firing in the silent background, hence compatible with the sparse feature detection strategy. In contrast, our recordings were done in a high mesopic / low photopic range where both rods and cones are supposedly active. Unlike the scotopic condition with rod vision, we then found high firing in awake recordings in general, indicating that no visual feature can be readily detectable as brief firing events in the silent background. To explore the implications of such firing patterns on visual coding, we took a modelling approach in the revised manuscript. We found that a latency-based temporal code was not preferable in the awake condition (Figure 7); and that a linear decoder worked significantly better with the population responses in the awake condition to capture the presented random fluctuation of the light intensity (Figure 8). While we have not tested any behavioural relevance in our study besides correlation to locomotion/pupil size, it is then possible that the retina may work in different modes under different light intensity regimes (Tikidji-Hamburyan et al 2015).

We clarified these points in the revised discussion section.

Sampling bias

The paper argues that sampling bias is not likely to contribute substantially to the results because of the wide variety of cell types recorded (line 431). This does not seem like a particularly strong argument, especially given the large degree of overlap in the distributions of most quantities across preparations. The argument about many cell types could be made more strongly if the distributions were completely separated, but that is not the case.

We cannot deny the presence of a sampling bias in our datasets, and as the reviewer pointed out, we made comparisons only at a population level, but not at the level of individual cells or cell-types. However, the anesthetized and awake recordings were done with the same recording setup and techniques, and thus subject to the same sampling bias. Hence, the difference in the RGC response properties between these conditions cannot be explained by the sampling bias per se.

Sensitivity

The firing rates in response to 10% contrast sinusoids are quite low, as are the maximal firing rates for high contrast sinusoids. Relatedly, the modulation produced by the noise stimuli, particularly for the array recordings, is weak. This raises concerns about the health of some of the preparations.

To our knowledge, in vivo contrast responses reported here were comparable to ex vivo data in previous reports (mouse, Jouty et al 2018, Pearson and Kerschensteiner 2015; rat, Jensen 2017, 2019). Likewise, the static nonlinearity and its upper bound for ex vivo responses were comparable between this study and previous reports (Santina et al. 2013; Kerschensteiner et al 2008; Cantrell et al 2010; Trapani et al 2022).

We also examined batch effects in the response to the noise stimuli. We found certain variabilities across preparations in each recording condition, but not to the extent to discard any particular data as an obvious outlier (Figure 6 – figure supplement 1). While it is difficult to tell the health status of preparations retrospectively, we thus believe that the effects were negligible.

Efficient coding

Sparse firing is not a universal property of retinal ganglion cell responses. Primate midget RGCs, for example, have pretty high maintained firing rates as shown in many past studies. Mouse RGCs have also been reported to operate in a mode similar to the high firing rate On cells reported here (Ke et al. 2014). A more balanced discussion of this past work is needed.

As the reviewer pointed out, some retinal ganglion cells show high firing under certain conditions. In a scotopic condition, for example, OFF cells have high firing rates, while ON cells fire virtually nothing unless a light stimulus is presented (Ke et al 2014; Smeds et al 2019). At the behavoural level, a single-photon detection above chance level nevertheless relies on the information from the ON but not the OFF pathway (Smeds et al 2019). Thus, the sparse coding framework still works as a valid strategy here, if not universal.

This is, however, very different from what we report here. In a high-mesopic/low-photopic light level, we found a general increase of firing across all cell categories in the awake condition, compared to the anesthetized or ex vivo recordings (Figures 3 and 6). While this lowers information transfer rate (bits/spike; Figure 7), we found that the awake responses were more linearly decodable than the responses in the other conditions (Figure 8). We also ran a simulation and showed that a latency-based temporal code is not preferable for the awake responses (Figure 7 – figure supplement 1). These results suggest that the retina in awake condition is in favor of a rate code, though we have not tested all light levels or any behavioural relevance here.

We clarified these points in the revised manuscript.

Role of eye movements

Could eye movements be at least partially responsible for the differences in response properties? Specifically, small fixational eye movements might produce a constantly varying input that could modulate firing.

As described above (Essential Review item #2), eye movements were rarely observed during the head-fixed awake recordings. Eliminating those events from the analysis did not change our overall conclusions, and thus their contributions should be minimal in this study. It should also be noted that we mainly used full-field stimulation, and thus microsaccades should not substantially affect the amount of light impinging on the retina. We clarified these points in the revised manuscript.

Reviewer #2 (Public Review):

The technical achievements presented in the manuscript represent a tour de force, as optical tract recordings in awake mice have only rarely been done before. The substantial number of neurons recorded in both awake and anaesthetized conditions form a precious and worldwide unique dataset. However, since the recordings represent a non-standard approach, it would be, in my view, highly beneficial to show more details about the success of the method. How did the authors post-hoc identify electrode contacts located in the optical tract, how did the spike waveforms look like, what were the metrics of spike sorting quality, etc.

We added more details about our recording and analysis methods in the revised manuscript. Below are answers to the reviewer’s specific questions:

• The probe was coated with a fluorescent dye (DiI stain) and its location was verified histologically after the recordings (Figure 1E).

• Spike waveforms typically had a triphasic shape (e.g., Figure 1F-H) as expected from axonal signals (Barry, 2015).

• Single-units were identified by clustering in principal component space, followed by manual inspection of spike shape as well as auto- and cross-correlograms. Most units had a minimum interspike interval above 2 ms (93%, awake; 94%, isoflurane; 99%, FMM); and no units had the interspike intervals below 1 ms for a refractory period (e.g., Figure 1I-K), except for 1 (out of 103) for FMM-anesthetized recordings.

We then selected visually responsive cells (SNR>0.15; see Eq.(1) in Methods) for the analyses.

The authors go a long way in characterising the functional response properties of the recorded neurons and relating them to previous ex-vivo recordings. Based on the responses they find, the authors claim that they identified "... a new response type [which] likely emerged due to high baseline firing in awake mice". Regarding this claim, how do the authors rule out that it corresponds to any of the previously described cell types? For instance, the very sharp transient or brief modulations by the contrast part of the stimulus might have been missed in previous classifications based on calcium responses (e.g. Baden et al. 2016), where a number of cell types seem to respond equally strong to grey and white and have an elevated response throughout the sinusoidal modulation of contrast. I acknowledge that the authors touch upon the possibility that the newly described OFFsuppressive ON cells correspond to a known cell type in the discussion, but I would recommend changing the phrasing of the results to avoid potential misunderstandings.

We agreed with the reviewer and revised the manuscript accordingly. Here we have two possibilities. Firstly, as the reviewer pointed out, this kind of response dynamics could be overlooked previously because of a difference in the recording modality (Ca imaging; Baden et al 2016) or clustering methods (Jouty et al 2019). Secondly, these cells may belong to one of the cell-types described in the past ex vivo studies, but exhibited distinct response dynamics in vivo as an emerging property of the awake condition. This is an interesting topic to pursue in future studies.

The manuscript makes the interesting suggestion that "the retinal output characteristics [...] observed in vivo, [...] provide a completely different view on the retinal code". Given that this conclusion would change the way we should think about and do retinal neuroscience, in my view, the authors should take a few more steps to quantitatively demonstrate the implications of their findings on retinal coding, e.g. how much lower is the information transmitted per spike, how much does a temporal code based on spike timing suffer with the latencies observed in vivo. If the authors could quantify through computational modelling approaches the consequences of the observed differences, they might also be able to revise their title / main message, i.e. that "Awake responses SUGGEST inefficient dense coding in the mouse retina".

To explore functional implications of our findings, we performed three more analyses as suggested by the reviewer. Specifically,

1) We showed that the information transmitted per spike was significantly lower in awake condition, while the total information rate was comparable (Figure 7).

2) We tested the performance of a linear decoder applied on the firing rate in response to full-field noise, and showed that it worked significantly better for the awake population responses (Figure 8).

3) We simulated RGC responses to a full-field contrast change at different intensities in different conditions, and showed that a latency coding did not work well with awake responses, compared to ex vivo or anesthetized responses (Figure 7 – figure supplement 1).

These results strengthened our conclusion that awake response dynamics were different from anesthetized or ex vivo responses, all arguing against the sparse efficient coding principles at least at a light level we examined. We nevertheless kept the title as is because we have not explored the retinal coding properties per se. Our main claim stays on the visual response characteristics of retinal outputs in awake mice.

Reviewer #3 (Public Review):

The manuscript by Boissonnet, Tripodi, and Asari compares retinal ganglion cell (RGC) light responses in awake mice (recorded in the optic nerve) with those under two forms for anaesthesia and previously attained ex vivo recordings. This is a well motivated study looking at a question that is really critical to the field.

The presentation is generally clear and compelling. My suggestions are relatively minor and aimed at improving an already very strong article.

1) More cells in the awake condition would help strenghten the conclusions. Only 51 cells are reported, and mouse RGCs comprise more than 40 different types. The authors are well aware of the possible confound of sampling bias, and the best way to mitigate this issue in this experimental paradigm is simply to record more cells. The anesthsia conditions each have about 100 cells, which is better.

We made substantially more recordings in the awake condition, reaching 282 cells (in 15 animals) in total in the revised manuscript. This does not yet allow for a full cell-type classification as in the past ex vivo studies. Nevertheless, we did our best to broadly classify visual responses, and showed that the overall conclusions remained the same: awake RGCs had higher baseline firing and faster response kinetics in general. For details, see above our response to the Essential Revision item #1.

2) It took me longer than it should have (had to look up the previous paper cited) to figure out that the ex vivo comparison data were recorded at 37{degree sign}C. This is an important detail since most ex vivo recordings are at 32{degree sign}C. The authors should make this clear in the text and perhaps say something in the Discussion about comparisons to the larger body of literature of ex vivo studies at 32{degree sign}.

We are aware that most ex vivo studies on the retina were performed at 32 °C, which is lower than physiological body temperature (37 °C). However, the temperature of the ocular surface is around 37 °C (Vogel et al 2016), suggesting that the retina should operate at 37 °C in vivo. This is why we decided to perform ex vivo experiments at 37 °C in our previous study (Vlasiuk and Asari, 2021), allowing us to make a fair comparison between the ex vivo and in vivo recordings.

We clarified the point in the revised manuscript.

3) Direction and orientation selectivity should be separated in Fig. 2 and not combined into the confusing term "motion sensitive." Motion sensitivity has another meaning in the literature for RGCs that respond preferentially to moving over static stimuli without direction or orientation preference (Kuo et al., 2016; Manookin et al., 2018)

We agree with the reviewer. In the revised manuscript, we separated the direction and orientation selective cells (Figure 2), and avoided the term “motion sensitive.”

4) While I am certainly sympathetic to the argument that the RGC spike code is "inefficient" in the sense that it does not conform to efficient coding theory (ETC), I think it's oversimplified to claim that the present data is a key argument against ETC. Plenty of ex vivo data has already shown ETC to be incomplete at best, and misguided at worst, since it includes the implicit assumption that image reconstruction is the retina's objective function (or even that the experimenter has any idea what that objective function is). For example, OFF sustained alpha (OFF delta in guinea pig) RGCs are not quite sparse feature detectors even ex vivo, and they seem to be optimized to transmit contrast with high SNR (Homann and Freed, 2017). In general, the enormous coverage factor of the RGC population seems to make ETC untenable to begin with, as discussed in (Schwartz, 2021) and elsewhere. I realize that there are still people attached to simplistic forms of ETC as a key principle of retinal computatiion, so I am not asking for the authors to completely remove this angle. Rather, a more nuanced treatment of the issue both in the introduction and the discussion is warranted.

We totally agree that we are not the first to argue against the efficient coding principles in the retina (Schwartz, 2021). The main argument in this study is that certain aspects of the RGC activity are distinct in an awake condition, such as the baseline firing and response kinetics, and thus we cannot simply translate our knowledge obtained from ex vivo studies into awake animals. To explore the implications on retinal computations, we showed in the revised manuscript that 1) awake responses have a comparable total information transfer rate (in bits per second; Figure 7A) but are less efficient (i.e., lower bits per spikes; Figure 7B); 2) awake responses are not in favor of a latency-based temporal code (Figure 7 – figure supplement 1); and 3) a linear decoder worked significantly better with awake responses (Figure 8), even though an image reconstruction is not necessarily the objective function of the retina. These results point out a need to rethink about retinal function in vivo, including the efficient coding theory.

We thank the reviewer for the suggestion, and revised the manuscript accordingly.

References

Homann, J., and Freed, M.A. (2017). A mammalian retinal ganglion cell implements a neuronal computation that maximizes the SNR of its postsynaptic currents. Journal of Neuroscience 37, 1468-1478.

Kuo, S.P., Schwartz, G.W., and Rieke, F. (2016). Nonlinear Spatiotemporal Integration by Electrical and Chemical Synapses in the Retina. Neuron 90, 320-332.

Manookin, M.B., Patterson, S.S., and Linehan, C.M. (2018). Neural Mechanisms Mediating Motion Sensitivity in Parasol Ganglion Cells of the Primate Retina. Neuron 97, 13271340.e4. Schwartz, G.W. (2021). Retinal Computation (Academic Press).

Author Response

The following is the authors’ response to the original reviews.

Reviewer #1 (Public Review):

Summary of the major findings -

1) The authors used saturation mutagenesis and directed evolution to mutate the highly conserved fusion loop (98 DRGWGNGCGLFGK 110) of the Envelope (E) glycoprotein of Dengue virus (DENV). They created 2 libraries with parallel mutations at amino acids 101, 103, 105-107, and 101-105 respectively. The in vitro transcribed RNA from the two plasmid libraries was electroporated separately into Vero and C6/36 cells and passaged thrice in each of these cells. They successfully recovered a variant N103S/G106L from Library 1 in C6/36 cells, which represented 95% of the sequence population and contained another mutation in E outside the fusion loop (T171A). Library 2 was unsuccessful in either cell type.

2) The fusion loop mutant virus called D2-FL (N103S/G106L) was created through reverse genetics. Another variant called D2-FLM was also created, which in addition to the fusion loop mutations, also contains a previously published, evolved, and optimized prM-furin cleavage sequence that results in a mature version of the virus (with lower prM content). Both D2-FL and D2-FLM viruses grew comparably to wild type virus in mosquito (C6/36) cells but their infectious titers were 2-2.5 log lower than wild type virus when grown in mammalian (Vero) cells. These viruses were not compromised in thermostability, and the mechanism for attenuation in Vero cells remains unknown.

4) Next, the authors probed the neutralization of these viruses using a panel of monoclonal antibodies (mAbs) against fusion loop and domain I, II and III of E protein, and against prM protein. As intended, neutralization by fusion loop mAbs was reduced or impaired for both D2-FL and D2-FLM, compared to wild type DENV2. D2-FLM virus was equivalent to wild type with respect to neutralization by domain I, II, and III antibodies tested (except domain II-C10 mAb) suggesting an intact global antigenic landscape of the mutant virion. As expected, D2-FLM was also resistant to neutralization by prM mAbs (D2-FL was not tested in this batch of experiments).

5) Finally, the authors evaluated neutralization in the context of polyclonal serum from convalescent humans (n=6) and experimentally infected non-human primates (n=9) at different time points (27 total samples). Homotypic sera (DENV2) neutralized D2-FL, D2-FLM, and wild type DENV similarly, suggesting that the contribution of fusion loop and prM epitopes is insignificant in a serotype-specific neutralization response. However, heterotypic sera (DENV4) neutralized D2-FL and D2-FLM less potently than wild type DENV2, especially at later time points, demonstrating the contribution of fusion loop- and prM-specific antibodies to heterotypic neutralization.

Impact of the study-

1) The engineered D2-FL and D2-FLM viruses are valuable reagents to probe antibodies targeting the fusion loop and prM in the overall polyclonal response to DENV.

2) Though more work is needed, these viruses can facilitate the design of a new generation of DENV vaccine that does not elicit fusion loop- and prM-specific antibodies, which are often poorly neutralizing and lead to antibody-dependent enhancement effect (ADE).

3) This work can be extended to other members of the flavivirus family.

4) A broader impact of their work is a reminder that conserved amino acids may not always be critical for function and therefore should not be immediately dismissed in substitution/mutagenesis/protein design efforts.

Evaluating this study in the context of prior literature -

The authors write "Although the extreme conservation and critical role in entry have led to it being traditionally considered impossible to change the fusion loop, we successfully tested the hypothesis that massively parallel directed evolution could produce viable DENV fusion-loop mutants that were still capable of fusion and entry, while altering the antigenic footprint."

".....Previously, a single study on WNV successfully generated a viable virus with a single mutation at the fusion loop, although it severely attenuated neurovirulence. Otherwise, it has not been generated in DENV or other mosquito-borne flaviviruses"

The above claims are a bit overstated. In the context of other flaviviruses:

• A previous study applied a similar saturation mutagenesis approach to the full length E protein of Zika virus and found that while the conserved fusion loop was mutationally constrained, some mutations, including at amino acid residue 106 were tolerated (PMID 31511387).

• The Japanese encephalitis virus (JEV) SA14-14-2 live vaccine strain contains a L107F mutation in the fusion loop (in addition to other changes elsewhere in the genome) relative to the parental JEV SA14 strain (PMID: 25855730).

• For tickborne encephalitis virus (TBEV-DENV4 chimera), H104G/L107F double mutant has been described (PMID: 8331735)

There have also been previous examples of functionally tolerated mutations within the DENV fusion loop:

• Goncalvez et al., isolated an escape variant of DENV 2 using chimpanzee Fab 1A5, with a mutation in the fusion loop G106V (PMID: 15542644). G106 is also mutated in D2-FL clone (N103S/G106L) described in the current study.

• In the context of single-round infectious DENV, mutation at site 102 within the fusion loop has been shown to retain infectivity (PMID 31820734).

We thank the reviewer for these comments. We have adjusted the text above to better reflect and credit the prior literature. Text is modified as follows in the discussion session.

“Previous reported mutations in the fusion loop are mainly derived from experimental evolution using FL-Ab to select for escape mutant or by deep mutational scanning (DMS) of the Env protein for Ab epitope mapping. Mutations in the FL epitope were observed in a DENV2-NGC-V2 (G106V)39, attenuated JEV vaccine strain SA14-14-2 (L107F)40, attenuated WNV-NY99 (L107F)41. While most of the mutations, including the double mutations reported here lead to attenuation of the virus. A recent DMS study showed that Zika-G106A has no observable impact on viral fitness42. Interestingly, we also recovered a mutation G106L, suggesting position 106 and 107 might be the most tolerable position for mutation in mosquito borne flavivirus FL. On the other hand, tick borne flavivirus as well as vector only flavivirus show a more diverse FL composition. The inflexibility of mosquito borne flavivirus might be due to the evolution constraint of the virus to switch between mosquito and vertebrate hosts.”

Appraisal of the results -

The data largely support the conclusions, but some improvements and extensions can benefit the work.

1) Line 92-93: "This major variant comprised ~95% of the population, while the next most populous variant comprised only 0.25% (Figure 1C)".

What is the sequence of the next most abundant variant?

The sequence of the next most abundant variant has been added to the text.

2) Lines 94-95: "Residues W101, C105, and L107 were preserved in our final sequence, supporting the structural importance of these residues." L107F is viable in other flaviviruses.

We acknowledge that the L107F mutation has been described in other flaviviruses, including the tick-borne flaviviruses DTV and POWV. This mutation in JEV is associated with viral attenuation. This sentence is referring to the fact that, in our libraries, we did not recover variants with mutations at these positions, in contrast to D2-FL with variants at N103 and G106, indicating less mutational tolerance. However, we want to re-direct the focus of this manuscript to engineer a viable DENV that is antigenically different in the FL epitope, but not which residue is more tolerance for mutation.

3) Figure 2c: The FLM sample in the western blot shows hardly any E protein, making E/prM quantitation unreliable.

The samples used in Figure 2C derive from the growth curve endpoint (Figure 2A), in which there is a 1-log difference in viral titer between D2 and D2-FLM. Equivalent volumes of viral supernatant were loaded in the gel, explaining the reduced intensity of the E band in D2-FLM. The higher exposure on the right shows the E band more clearly for D2-FLM. The Western blot assay comparing prM/E ratio as a measure of maturation state was described and validated in our previous study (Tse et al. 2022, mbio). The methods and figure legend have been updated to include greater detail. The polyclonal E antibody was specifically chosen for this study as our previously used monoclonal antibody targeted the fusion loop. The polyclonal antibody was raised against a fragment of E (AA 1-495) and should have minimal effect by the fusion loop mutations.

4) Lines 149 -151: "Importantly, D2-FL and D2-FLM were resistant to antibodies targeting the fusion loop. While neutralization by 1M7 is reduced by ~2-logs, no neutralization was observed for 1N5, 1L6, and 4G2 for either variant (Figure 3 A)".

a) Partial neutralization was observed for 1N5, for D2-FL.

The text has been updated to more accurately describe the 1N5 neutralization data.

b) Do these mAbs cover the full spectrum of fusion loop antibodies identified thus far in the field?

We did not test every known fusion loop antibody that has been described, instead focusing on 1M7, 1N5, 1L6, and 4G2, which were previously described by Smith et al and Crill et al. We also modified the text in discussion to reflect the possibility of other FL-Ab that are not affected by out mutations.

“We have tested a panel of FL-Ab; however, we cannot exclude the possibility that other FL-Abs may not be affected by N103S and G106L. However, we have shown that saturation mutagenesis could generate mutants with multiple amino acid changes, and we are currently using D2-FLM as backbone to iteratively evolve additional mutations in FL to further deviate the FL antigenic epitope.”

c) Are the epitopes known for these mAbs? It would be useful to discuss how the epitope of 1M7 differs from the other mAbs? What are the critical residues?

Critical residues for these antibodies have been described. They are as follows: 1M7: W101R, W101C, G111R; 1N5: W101R, L107P, L107R, G111R; 1L6: G100A, W101A, F108A; 4G2: G104H, G106Q, L107K. The critical residues for 1M7 are slightly different than the others, perhaps explaining the residual binding to D2-FL. Note that the critical residue identified previously for 1M7 and 1N5 do not overlap with D2-FLM mutations, suggesting the FL mutations has extending effect on the antigenic FL epitope.

d) Maybe the D2-FL mutant can be further evolved with selection pressure with fusion loop mAbs 1M7 +/-1N5 and/or other fusion loop mAbs.

We agree that it may be possible to further evolve D2-FL using antibody selection, although we have not yet performed these experiments, we are currently performing iterative saturation mutagenesis and directed evolution to further evolve away from the natural FL.

5) It would have been useful to include D2-M for comparison (with evolved furin cleavage sequence but no fusion loop mutations).

Neutralization data for some of the mAbs against D2-M can be found in our previous study (Tse et al. 2022 mBio), in which no difference in neutralization was observed compared to DV2 wildtype. Given the limited resources of the anti-DENV NHP and human serum, we did not add D2-M for comparison. Although some insight can be deduced from the D2-FL vs D2-FLM comparison, we agree future studies that are designed to delineate CR-Ab population between prM, FL and other CR-epitopes should include D2-M for comparison.

6) Data for polyclonal serum can be better discussed. Table 1 is not discussed much in the text. For the R1160-90dpi-DENV4 sample, D2-FL and D2-FLM are neutralized better than wild type DENV2? The authors' interpretation in lines 181-182 is inconsistent with the data presented in Figure 3C, which suggests that over time, there is INCREASED (not waning) dependence on FL- and prM-specific antibodies for heterotypic neutralization.

We remade Table 1 to show dilution factors instead of dilution factor-1 of FRNT50.

In general, our human convalescent sera from heterotypic infection (DENV1, 3 and 4) showed none to low neutralization against our DENV2. FRNT50s were between 1: 40 – 1:200. Given the weak potency of the antiserum, it is difficult to compare the FRNT50s between DV2-WT and D2-FLM.

Similarly, in a different NHP cohort (2nd NHP cohort shown in Table 1), only one DENV4 infected NHP (R1160) showed a low heterotypic titer against DENV2. The detectable FRNT50s were between 1: 50 – 1:90. The value was extrapolated based on a single data point (1:40) which has above 50% neutralization. Given the Hill slope of all the neutralization curves were below 0.5, the FRNT50 values is should not be

In conclusion, we do not think serum from Table 1 is potent enough to shows difference between the viruses. The intension to show the negative data in Table 1 is to highlight the difference in serum heterogeneity in DENV infected patients and experimental infected NHPs.

As the reviewer pointed out, the dependence of FL-Ab in later time points increased (the difference between DV2 and D2-FL at 20dpi vs 60dpi vs 90dpi), suggesting non-FL CR-Ab is waning but not prM- and FL-Abs. We rewrote the sentence as follow:

“These data suggest that after a single infection, many of the CR Ab responses target prM and the FL and the reliance on these Abs for heterotypic neutralization increase overtime (Figure 3C).”

Suggestions for further experiments-

1) It would be interesting to see the phenotype of single mutants N103S and G106L, relative to double mutant N103S/G106L (D2-FL).

2) The fusion capability of these viruses can be gauged using liposome fusion assay under different pH conditions and different lipids.

3) Correlative antibody binding vs neutralization data would be useful.

We thank the reviewer for the suggestions; we agree these would be of interest and, indeed, these studies are currently underway. In regard to single mutants, these were present in the initial plasmid library but did not enrich after viral production and passage. Two possible explanations can be drawn, 1) The stochastic of directed evolution prevents a single mutant with similar fitness to enriched. 2) The two mutations are compensatory to each other to make a functional mutant. The 2nd hypothesis highlights the difference between saturation mutagenesis (this study) and DMS (in previous studies).

Fusion capability is indeed very interesting, however, the mechanistic difference or not between wildtype FL and the mutated FL in supporting fusion is not the focus of this study. Instead, we are currently working on adapting the D2-FLM in mammalian cells. If successful, the difference in fusion mechanism between the Vero adapted and D2-FLM in different lipid, insect vs mammalian would be of interest.

We are currently developing whole virus ELISA; we avoid using rE monomer for the study as it might neglect the conformation Ab.

Reviewer #2 (Public Review):

Antibody-dependent enhancement (ADE) of Dengue is largely driven by cross-reactive antibodies that target the DENV fusion loop or pre-membrane protein. Screening polyclonal sera for antibodies that bind to these cross-reactive epitopes could increase the successful implementation of a safe DENV vaccine that does not lead to ADE. However, there are few reliable tools to rapidly assess the polyclonal sera for epitope targets and ADE potential. Here the authors develop a live viral tool to rapidly screen polyclonal sera for binding to fusion loop and pre-membrane epitopes. The authors performed a deep mutational scan for viable viruses with mutations in the fusion loop (FL). The authors identified two mutations functionally tolerable in insect C6/36 cells, but lead to defective replication in mammalian Vero cells. These mutant viruses, D2-FL and D2-FLM, were tested for epitope presentation with a panel of monoclonal antibodies and polyclonal sera. The D2-FL and D2-FLM viruses were not neutralized by FL-specific monoclonal antibodies demonstrating that the FL epitope has been ablated. However, neutralization data with polyclonal sera is contradictory to the claim that cross-reactive antibody responses targeting the pre-membrane and the FL epitopes wane over time.

Overall, the central conclusion that the engineered viruses can predict epitopes targeted by antibodies is supported by the data and the D2-FL and D2-FLM viruses represent a valuable tool to the DENV research community.

Reviewer #1 (Recommendations For The Authors):

1) Line 51-52: "Currently, there is a single approved DENV vaccine, Dengvaxia." Line 56-57: "Other DENV vaccines have been tested or are currently undergoing clinical trial, but thus far none have been approved for use."

It should be specified for the global audience that this applies to the United States. Takeda's DENV vaccine, QDENGA is approved in Indonesia, European Union, and Brazil.

The text has been modified to include this information.

2) Line 62-63: - "The core fusion loop-motif DRGWGNGCGLFGK is highly conserved..." Lines 78-80: - We generated two different saturation mutagenesis libraries, each with 5 randomized amino acids: DRGXGXGXXXFGK (Library 1) and 79 DRGXXXXXGLFGK (Library 2).

It may be useful for the readers if the amino acid numbers are stated. The core fusion loop motif DRGWGNGCGLFGK (Eaa98-110) is highly conserved. We generated two different saturation mutagenesis libraries, each with 5 randomized amino acids: DRGXGXGXXXFGK (Library 1; Xaa 101,103, 105-7) and DRGXXXXXGLFGK (Library 2; Xaa 101-105).

This information has been added to the text.

3) Line 91-92: "Bulk Sanger sequencing revealed an additional Env-91 T171A mutation outside of the fusion-loop region."

It looks like the mutation T171A is in domain I of the E protein and does not seem to interface with the fusion loop. Is that why it wasn't pursued further?

The E171A mutation was included in the infectious clone for D2-FL and D2-FLM. The text has been modified to clarify this inclusion.

4) Lines 82-85: "Saturation mutagenesis plasmid libraries were used to produce viral libraries in either C6/36 (Aedes albopictus mosquito) or Vero 81 (African green monkey) cells and passaged three times in their respective cell types."

a) What was the size of the libraries? How does one make sure that the experimental library actually has all the amino acid combinations that were intended?

Each library has 5 randomized amino acids, so there are 205 = 3.2 million combinations. In these experiments, sequencing of the plasmid libraries revealed about 2 million unique amino acid sequences, or approximately 62.5% library coverage. The actual plasmid diversity is expected to be higher than 2 million as our deep sequencing has limited coverage.

b) The wild type sequence was excluded from the libraries, correct?

The wild-type sequence was not specifically excluded from the libraries, as there is no easy method to do so. Wild-type sequence was detected in the plasmid libraries but was not selected in the C6/36 library. However, in the Vero library, we recovered WT virus.

5) Table 1: - Please include in the table description, what the colors indicate.

We remade Table 1 to show dilution factors instead of dilution factor-1 of FRNT50 and removed the unnecessary color code. We also added all relevant information in the table legend.

6) Lines 246-248: "Previously, a single study on WNV successfully generated a viable virus with a single mutation at the fusion loop, although it severely attenuated neurovirulence."

It may be worthwhile to mention the WNV mutation (L107F) as some readers may be curious about where this mutation is relative to the ones described in this study.

This information has been added to the text. We also included the previously described FL mutations in flaviviruses in the text.

Reviewer #2 (Recommendations For The Authors):

Major Critique:

• There is a disconnect between Fig 2A and 2C. FL and FLM viruses have much lower levels of prM-E expression in the viral supernatants based on the western blot in 2C. Why isn't E being detected in the Western? Is the particle-to-pfu ratio skewed in the mutant viruses? Is it possible that the polyclonal is targeting the cross-reactive prM and FL epitopes, and if so would using a monoclonal antibody targeting a known DIII-epitope (2D22) yield a different western result? Also, the legend and methods for Fig 2C are not clear. What is actually being tested in the Western blot? Were equivalent volumes of the different viral preps used?

The samples used in Figure 2C derive from the growth curve endpoint (Figure 2A), in which there is a 1-log difference in viral titer between D2 and D2-FLM. Equivalent volumes of viral supernatant were loaded in the gel, explaining the reduced intensity of the E band in D2-FLM. The higher exposure on the right shows the E band more clearly for D2-FLM. The Western blot assay comparing prM/E ratio as a measure of maturation state was described and validated in our previous study (Tse et al. 2022, mBio) and the methods have been updated to include greater detail. The polyclonal E antibody was specifically chosen for this study as our previously used monoclonal antibody targeted the fusion loop. The polyclonal antibody was raised against a fragment of E (AA 1-495) and should not be affected by the fusion loop mutations. 2D22 is a conformational antibody and does not work in western blot.

• Table 1: The data within Table 1 is ignored in the text, and some of this data contradicts the central conclusions of the manuscript.

o A.) Some of the convalescent data contradicts the hypothesis. DS0275 had an equivalent neut between DV2 and D2-FLM, DS1660, and R1160 (90) had better neut against the D2-FLM than DV2. Discussion of these samples is warranted.

o C.) The description in the legend does not adequately describe the table. What do the colors represent? What are the numerical values being displayed? What is in parentheses, (I assume the challenge strain)? The limit of detection is reported as 1:40; 0.25. 1:40 is 0.025 which matches most of the data? There is inadequate description of these experiments in the materials and methods.

We remade Table 1 to show dilution factors instead of dilution factor-1 of FRNT50 and removed the unnecessary color code. We also added discussion for Table 1 and clarify the difference between the three cohorts of serum in the text with the corresponding references.

In general, our human convalescent sera from heterotypic infection (DENV1, 3 and 4) showed none to low neutralization against our DENV2. FRNT50s were between 1: 40 – 1:200. Given the weak potency of the antiserum, it is difficult to compare the FRNT50s between DV2-WT and D2-FLM.

Similarly, in a different NHP cohort (2nd NHP cohort shown in Table 1), only one DENV4 infected NHP (R1160) showed a low heterotypic titer against DENV2. The detectable FRNT50s were between 1: 50 – 1:90. The value was extrapolated based on a single data point (1:40) which was above 50% neutralization. Given the Hill slope of all the neutralization curves were below 0.5, the FRNT50 values are not reliable.

In conclusion, we do not think sera from Table 1 is potent enough to show difference between the viruses. The intension to show the negative data in Table 1 is to highlight the difference in serum heterogeneity in DENV infected patients and experimental infected NHPs.

Minor critique:

Figure 1C: Legend is not clear for this panel. What is on the x-axis of the bubble plots? Are these mutations across the entire viral genome or is this just the prM-E sequence?

The X-axis is a scatter of all of the sequences contained in the library, similar to graphs used for plotting CRISPR screen results. These represent individual sequences from the saturation mutagenesis libraries in the fusion loop of E as described in Figure 1B.

The wording in Lines 92-94 is not clear. It looks like the T171A mutation was present in 95% of the sequences (Line 92). Yet this sequence was not incorporated into the variant virus. What is the rationale for omitting this mutation in downstream variant virus generation?

The 95% in Line 92 refers to the variant containing N103S/G106L mutations as seen in Figure 1C. The high-throughput sequencing approach did not include residue 171, so the presence of the T171A mutation in combination with fusion loop mutations cannot be determined. However, the E171A mutation was included in the infectious clone for D2-FL and D2-FLM. The text has been modified to clarify this inclusion.

The authors discuss the potential of the D2-FL or D2-FLM virus as a potential vaccine platform in the abstract, introduction, and conclusion. This is a good idea, but the authors provide no evidence of feasibility in this manuscript.

The ultimate goal to engineer a viable DENV with distinct FL antigenic epitope is for it use as live attenuated vaccine. As this is the rationale for the study, we introduce the concept throughout the manuscript. The current study demonstrated the possibility to mutate a novel fusion loop motif in DENV and provided evidence to show the favorable antigenic properties of D2-FLM. We agree with the reviewer that definitive work in animal to show vaccine efficacy need to be done and are currently undergoing. To avoid misleading our audience, we tone down the emphasis of vaccine use in the text.

Line 150-153: Figure 3A demonstrates that the FL-specific antibodies broadly do not neutralize the mutant viruses. However, the conclusions are overstated in the text. 1N5 neutralizes the D2-FL variant.

The text has been updated to more accurately describe the 1N5 neutralization data.

Lines 175-182: The authors make a lot of assumptions about the target of the polyclonal target without any evidence.

These lines reference studies that showed greater enhancement by antibodies targeting the fusion loop and prM as compared to other cross-reacting antibodies. The assumption that both our manuscript and others have drawn was that Abs that are cross-reactive and weakly neutralizing are more prone for ADE. As discussed, other groups have attempted to mutate the FL from recombinant E protein to achieve similar goal to remove the fusion loop epitope to reduce ADE. We have re-written the sentence in the followings:

“As FL and prM targeting Abs are the major species demonstrated to cause ADE in vitro, we and others hypothesized these Abs are responsible for ADE-driven negative outcomes after primary infection and vaccination,10–12,32 we propose that genetic ablation of the FL and prM epitopes in vaccine strains will minimize the production of these subclasses of Abs responsible for undesirable vaccine responses. Indeed, covalently locked E-dimers and E-dimers with FL mutations have been engineered as potential subunit vaccines that reduce the availability of the FL, thereby reducing the production of FL Abs.33–36”

Note: This preprint has been reviewed by subject experts for Review Commons. Content has not been altered except for formatting.

Learn more at Review Commons

Referee #3

Evidence, reproducibility and clarity

Summary:

The present manuscript presents a thorough description of the relative localization (in space and time) of a number of proteins of the early secretory pathway. To that aim, the authors used by their custom-made 3D live cell super-resolution microscope (SCLIM) and the yeast S. cerevisiae as a model system. The main claim of these data is that the early secretory pathway in S. cerevisiae is organized by maturation from a newly proposed yeast ERGIC compartment all the way to the trans-Golgi network (TGN).

Major comments:

I have two major comments regarding this manuscript:

1. It is not clear to me how the presented data shows the existence of an ERGIC in the yeast S. cerevisiae. I understand, and appreciate from this text too, that a clear definition of ERGIC, even in a mammalian system, is unclear. For this reason, I would first suggest that the authors provide a clear definition of what ERGIC means to them. Next, the experiments herein presented are all based on a very careful, thorough and nicely organized spatio-temporal mapping of a large number of early secretory pathway proteins, including the ERGIC53 yeast "counterpart" Emp46 (it would help to add, even as a supplementary figure, an alignment/sequence comparison between the human ERGIC53 and the S. cerevisiae emp46). However, the data presented here does not clearly indicate to me that there is a bona fide ERGIC in yeast. Couldn't it just be that what the authors call ERGIC is a cis-Golgi cisterna? I understand that the BFA experiments show a different behavior for some proteins, which fits with what the authors previously names GECCO in plants, so why not calling this GECCO? Again, it will be important to provide definitions of these compartments for the audience. Next, my main concern here is that this is all based on SCLIM, which is a very nice technique, but the resolution is limited in both space and time (by the way, it would be nice to explicitly measure of quantify the spatial resolution in x-y-z). Hence, it is not possible to discern whether an "independent" ERGIC is formed as compared to cis-Golgi cisterna. Electron microscopy (possibly CLEM) could help somehow resolve that and massively increase the strength of the claims, but I do understand this might be difficult for this group and very time consuming, so it might be important to clearly state the limitation of the herein presented data. A possible alternative to test if protein that are seen segregated are within the same membrane (as claimed here) would be to do trapping experiments where a reagent induces dimerization between the two proteins (when tagged with specific tags, such as FKBP/FRB).
2. I could not find any details (maybe I have missed them) about how many times experiments were replicated and the statistical significance of the findings herein reported. In most figures, examples of microscopy images/videos are shown, and selected lines profiles are presented. However, it is not clear how robust these experiments are. Some ideas:

2.1.) The major source of quantification is the peak-to-peak time distance between two proteins. In Table S1 some stdev is presented, but not clear how it is find (it is the sted of all n number of puncta? or of the mean duration per cell? or of the mean duration per experiment? I would suggest that the authors provide the results shown in Table S1 plotted as a histogram or superplot (see e.g. https://rupress.org/jcb/article/219/6/e202001064/151717/SuperPlots-Communicating-reproducibility-and) and clearly explain how statistics is performed.

2.2) Also, the time-lapse movies are acquired with a 5s gap between time points. How is this included in the incertainty of the peak-to-peak duration in Table S1?

2.3) In pg. 7 the authors write "Although experimental variation was high, the two zones appeared to be spatially segregated". Can the authors provide quantitative and statistical support of this claim?

2.4) It is not clear to me how the puncta for analysis are selected. For example, in Fig. 1C, the punctum shown already shows some initial co-localization (it could be e.g. that a peak value was prior or after the duration of the time lapse movie, thereby biassing the computation of the peak-to-peak duration). So, if one would consider those spots e.g., positive for Emp46 that do not contain Mnn9 signal, how often do you see conversion (that is, appearance of Mnn9 signal)? Along the same lines, in pg. 8 the authors write "... signal appeared first and then mnn9-mCherry came up". Details on how this quantification is done and statistical analysis would be needed, to my opinion, to support the claim.

Minor comments:

1. The color code for the 3 color microscopy images is nice, however, the use of green and red for the 2 color images is a bit unfortunate for some people (like myself) who suffer from color blindness. I'd suggest to use green and magenta instead.
2. Pg. 8: have the authors tested Rer1 vs Emp46?
3. Pg. 8: I was of the impression that GRASP65 (GORASP1) is considered to be a cis-Golgi protein (see e.g., Tie et al eLife 2018). Then, what the authors call "ERGIC" couldn't it simply be a cis-Golgi cisterna?
4. pg. 13: "propose to define Grh1, Rer1, and Sed5 as yeast ERGIC/GECCO...". What about Emp46?
5. The first part of the manuscript (up to mid page 13) is clearly focused on defining ERGIC in yeast, then the paper appears as a set of experiments aimed at adding more components in their spatio-temporal mapping. This is ok, but is should be clearly motivated and explained in the Title, abstract and intro.
6. The visualization of colocalization according to the opacity (as said in the methods) is somehow confusing to me. Are the 3D images projections or 3D renderings (no axes are seen)? In e.g. Fig. 6G or 8L, regions where green and magenta (or green and red) are colocalized do not appear white (or yellow), which visually suggests to the inattentive reader that there is no colocalization, when there is.
7. I have not understood what this sentence in pg. 18 means: Similar segregation patterns are also observed during the Golgi-TGN maturation process (Tojima et al., 2019). "We propose that the ERGIC, Golgi, and TGN can coexist as structurally and functionally distinct zones within a single, maturing cisterna." Are they referring to ERGIC, Golgi, and TGN steady state components (proteins) or the structures themselves?
8. The introduction of new data (mammalian data) in the discussion is odd. It might be ok, but I would frame it within a results section and use it later in the discussion.
9. Fig.9: the arrows should go from protein to protein (some seem to go from in between proteins, such as the bottom-most arrow with 87.8 s time duration. Also in panel B, bottom part, some proteins are missing (Erd2 ad Chs5).
10. Fig. 1 and many other: in the line profiles the distance in the x axis has no units or labels. Please add this and the direction of the line profile (an arrowhead would suffice).

Significance

General assessment:

The experiments herein presented are based on a very careful, thorough and nicely organized spatio-temporal mapping of a large number of early secretory pathway proteins, including the ERGIC53 yeast "counterpart" Emp46. However, the data presented here does not convincingly show that there is a bona fide ERGIC in yeast. A major limitation is that the experiments are all based on a state-of-the-art, but still with a limited resolution, fluorescence microscopy technique. Ultrastructural data (e.g., CLEM) would massively help support or revisit the claims presented in this manuscript regarding the existence of an ERGIC compartment in yeast. Also, adding the information about the number of biological replicates and proper statistical analyses on the presented results would be needed to further support the claims.

Advance:

This manuscript builds on the authors' custom build 4D super-resolution microscope (SCLIM) and on previous results (e.g., Tojima et al., J. Cell Sci. 2019). The main novelty is in the study of a number of new early secretory pathway proteins and in the proposal of the existence of a non-stable, maturing ERGIC compartment in S. cerevisiae.

Audience:

This paper might be attractive for a broad audience of cell biologists, especially those interested in membrane biology, cell compartmentalization, and intracellular trafficking and secretion.

Describe your expertise:

I am an expert in membrane trafficking.

Good case -- importance of adding tests to validate response codes and times, ensuring optimal server performance and response.

This section provides a step-by-step guide on setting up PyCharm for automated testing using the 'Pytest Web Framework'.

The efficiency of a front-end framework can significantly impact project timelines. Choosing a framework that aligns with the project's deadline and the team's proficiency can ensure timely delivery.

After converting the Postman test into a JMeter format, users can easily adjust the concurrency settings by configuring the number of threads in the JMeter UI, offering flexibility in load testing scenarios.

This code snippet in Postman ensures that the number of displayed results on a single page is always less than the total count of results, ensuring pagination is functioning correctly.

response code is missing

needs to be in code format

Clavier codé connecté

I think at the bare minimum code and raw data files should be uploaded somewhere that is publicly accessible, given the probably high interest in this piece of work. I can only speak for microbial data since that is my background, but 16S amplicons should be uploaded to the NCBI SRA, and the commands or code used for 16S analysis can easiliy be put on github.

echidnapped #diablo #icbmechatro

I've "asked the machine that sometimes ... acthually makes things for me to produce a series of tools related to ticonderoga/this ..

here specifically like lego-blox a set of 23 molecules mirroring the 23 BCAA "nucleotides" could mo,st likely very easily be used to create a useful to life storage paradigm that will be significantly more robust than simply DNA.

RNA radio recception inside the endoplasmic reticulum has been the subject of my "drakul fantasies" since deciding all on my own that chemotherapy (like astra zenica and bayer) could easily be the salvation of our .. "augmented reaities towards ... not staying in reality" if only we cared to implement ... a layer of something between shapeshifting and "democrat-people are the best system money and space-computers have ever had take care of them."