Several new tools have recently been developed to leverage long-reads for taxonomic profiling

Lemur additionally requires a marker gene (MG) database, whereas Magnet requires a (ideally small) set of genomes

Our results indicate that Lemur can efficiently process large datasets within minutes to hours in limited computational resource settings.

can improve precision by detecting and filtering out many false positive calls

Lemur and Magnet have limitations that vary by use case. Reliance on bacterial marker genes necessarily implies it cannot generalize to viral genome classification

reliance on the marker genes makes it less sensitive than alternatives like Kraken 2 or MetaMaps, which use all long reads and complete genomes.

our study focused on taxonomic profiling and binary presence and absence metrics for taxa

The EM algorithm begins by initializing F (t) to the uniform distribution and initializing P (r|t) for each read and taxon pair (r, t).

The goal of Magnet is to detect and remove potential false positives by performing competitive read alignment leveraging all of the reads mapped against the entire reference genome

As input, Magnet requires reads as well as a taxonomic abundance profile (estimated from the input reads e.g. using Lemur)

Lastly, Magnet marks species as present or absent.

investigated the concordance between the species and genus level calls across the tools

we built a new marker gene database with 43 markers for bacteria+archaea and 48 markers for fungi

We then built the database using recent versions of NCBI RefSeq: version 221 for both bacteria (329,194 assemblies) and archaea (1,911) and version 222 for fungi (564)

The final database was 4.1 GB, containing 3,335,783 sequences.

Lemur, a marker-gene-based long-read taxonomic profiler

Magnet, a genome-based validation tool for confirming the presence and absence of microbial genomes present in a sample

we introduce Lemur and Magnet, a pair of tools optimized for lightweight and accurate taxonomic profiling for long-read shotgun metagenomic datasets

Multiple genome alignment, the process of identifying nucleotides across multiple genomes which share a common ancestor

We introduce a partitioning option to Parsnp, which allows the input to be broken up into multiple parallel alignment processes which are then combined into a final alignment

improves our previous method, MHG-Finder, by utilizing a guide tree to significantly improve scalability and provide more informative biological results

Whole-genome alignments play a crucial role in downstream analyses in comparative genomic studies

A maximal homologous group, or MHG, is defined as a maximal set of maximum-length sequences whose evolutionary history is a single tree

MHG is formed by identifying and grouping all homologous sequences

evolutionary events

are encapsulated within the same MHG

processes such as horizontal gene transfer or gene duplication and loss may disrupt this homology by recombining only parts of genes, causing gene fission or fusion

Structural variants (SVs), genomic alterations of 10 base pairs or more, play a pivotal role in driving evolutionary processes and maintaining genomic heterogeneity within bacterial populations

Bacterial genome dynamics

encompassing a single metagenome coassembly graph constructed from all samples in a series

log fold change in graph coverage between subsequent samples is then calculated to call SVs

show rhea to outperform existing methods for SV and horizontal gene transfer (HGT) detection in two simulated mock metagenomes

innovative approach leverages raw read patterns rather than references or MAGs to include all sequencing reads in analysis

studying SVs across diverse and poorly characterized microbial communities

Recent work utilizes coassembly graphs for metagenomes to decompose strain diversity into haplotypes (30), but to the best of our knowledge, this is the first time coassembly graph patterns have been used for automated detection of SVs in a metagenome series.

In isolate genomics, the goal of SV detection is relatively straightforward: detect long genomic differences between a sequence and reference genome that can be classified as an insertion, deletion, inversion, duplication, translocation, or any combination

You can install modules from nf-core/modules in your pipeline using nf-core modules install. A module installed this way will be installed to the ./modules/nf-core/modules directory.

Nextflow automatically creates and activates the Conda environment(s) given the dependencies specified by each process.

The use of Conda recipes specified using the conda directive needs to be enabled explicitly in the pipeline configuration file (i.e. nextflow.config):

conda.enabled = true

Specifying the Conda environments in a separate configuration profile is therefore recommended to allow the execution via a command line option and to enhance the workflow portability

process.conda = 'samtools'

You can also download Conda lock files from Wave build pages. These files list every package and its dependencies, so Conda doesn’t need to resolve the environment. This makes environment setup faster and more reproducible.

conda '/some/path/my-env.yaml'

conda.useMicromamba

Uses the micromamba binary instead of conda to create Conda environments

Please only use Conda as a last resort, i.e., when it’s not possible to run the pipeline with Docker or Singularity.

Any channel in the workflow can be assigned to an output, including process and subworkflow outputs. This approach is intended to replace the publishDir directive.

A named workflow is a workflow that can be called by other workflows:

As a best practice, params should be used only in the entry workflow and passed to workflows and processes as explicit inputs.

assembly

challenging in complex environmental samples consisting of hundreds to thousands of populations

Mapler is a metagenome assembly and evaluation pipeline

Hi-Fi long read

novel metrics assessing the diversity that remains uncaptured by the assembly process

output: path "${greeting}-output.txt" script: """ echo '$greeting' > '$greeting-output.txt'

You can think of view() as a debugging tool, like a print() statement in Python

We prefer to be explicit to aid code clarity, as such the $it syntax is discouraged and will slowly be phased out of the Nextflow language.

We are using an operator closure here - the curly brackets.

splitCsv() reads each line into an array, and each comma-separated value in the line becomes an element in the array

a process will emit value channels if it is invoked with all value channels, including simple values which are implicitly wrapped in a value channel.

In the strict syntax, variables must be declared with def and must not specify a type:

each row is simply a list of columns.

output: path "UPPER-${input_file}" script: """ cat '$input_file' | tr '[a-z]' '[A-Z]' > 'UPPER-${input_file}'

You know how to collect outputs from a batch of process calls and feed them into a joint analysis or summation step.

things could get a little tricky, because we need to be able to handle an arbitrary number of input files. Specifically, we can't write the command up front, so we need to tell Nextflow how to compose it at runtime based on what inputs flow into the process.

we're not adding the operator in the context of a channel factory, but to an output channel.

write another pipeline that calls on one of those processes, you just need to type one short import statement to use the relevant module. This is better than just copy-pasting the code, because if later you decide to improve the module, all your pipelines will inherit the improvements.

They encapsulate applications and dependencies in portable, self-contained packages that can be easily distributed. Containers are also key to enabling predictable and reproducible results.

Nextflow was one of the first workflow technologies to fully embrace containers for data analysis pipelines.

Today, workflows may comprise dozens of distinct container images. Pipeline developers must manage and maintain these containers and ensure that their functionality precisely aligns with the requirements of every pipeline task.

Wave — a container provisioning and augmentation service that is fully integrated with the Nextflow and Nextflow Tower ecosystems.

Wave allows developers to manage containers as part of the pipeline itself

Our platform combines novel hardware with AI-enabled bioinformatics to unlock the personalized medicine potential of the gut microbiome

launch directory

profiles are applied in the order in which they are specified on the command line.

The standard profile is used by default when no profile is specified.

Config scopes are used to group related config options

The only difference when compared with legacy syntax is that the process is not bound with specific input and output channels, as was previously required using the from and into keywords respectively

Another exciting feature of Nextflow DSL 2 is the ability to compose built-in operators, pipeline processes and sub-workflows with the pipe (|) operator

mimics the expressiveness of the Unix pipe model

due to their inherent complexity and the limited availability of decontamination pipelines compared with those for marker gene datasets

multiple profiles can be loaded

later profiles can overwrite earlier profiles.

Please only use Conda as a last resort i.e. when it’s not possible to run the pipeline with Docker, Singularity, Podman, Shifter, Charliecloud, or Apptainer.

Development and prototyping: When you need flexibility to update packages
Custom/proprietary software: When containers aren't available
Resource-constrained environments: Where container overhead is problematic
Legacy systems: Where container runtimes aren't available

Primary: Use Docker/Singularity containers
Development: Use Wave to generate containers from conda specs
Fallback: Use conda only when containers aren't feasible
Future: Leverage conda lock files for maximum reproducibility

Process directives allow the specification of settings for the task execution such as cpus, memory, container, and other resources in the workflow script.

it’s strongly suggested to define the process settings in the workflow configuration file instead of the workflow script

quick alternative to building Conda packages in the local computer

Wave allows the provisioning of containers based on the conda directive used by the processes in your pipeline

Seqera Platform access token is not mandatory, but it is recommended in order to access private container repositories and pull public containers without being affected by service rate limits

FastGA finds alignments between two genome sequences more than an order of magnitude faster

stores millions of alignments in a fraction of the space of a conventional CIGAR-string

using a trace-point encoding

We carefully separate the problems of genome alignment and genome homology

by not conflating these two steps, FastGA can be used for other downstream tasks such as finding recurrent insertions due to transposable elements

The key idea is to reduce the number of k-mers inspected for seed matches by using only those that are minimizers in a window of some small size

it is a way to perform a dry-run

provide a dummy script that mimics the execution

ou can define a command stub, which replaces the actual process command when the -stub-run or -stub command-line option is enabled:

In this Review, we examine the main machine learning concepts, tasks and applications that are relevant for experimental and clinical microbiologists

provide the minimal toolbox

understand, interpret and use machine learning

BugBuster is a fully containerized, modular, and reproducible workflow implemented in Nextflow. The pipeline streamlines analysis at level of reads, contigs, and metagenome-assembled genomes (MAGs), offering dedicated modules for taxonomic profiling and resistome characterization.

Thanks to the use of containers, BugBuster can be deployed with minimal configuration on workstations, high-performance clusters, or cloud platforms

an update to GTDB-Tk that uses a divide-and-conquer approach where user genomes are initially placed into a bacterial reference tree with family-level representatives followed by placement into an appropriate class-level subtree comprising species representatives

Genome Taxonomy Database (GTDB; https://gtdb.ecogenomic.org) provides a phylogenetically consistent and rank normalized genome-based taxonomy for prokaryotic genomes sourced from the NCBI Assembly database.

GTDB uses relative evolutionary divergence (RED) to delineate higher-rank taxa and average nucleotide identity (ANI) to delineate species clusters

Groovy-style type annotations should be used instead:

The take: section is used to declare the inputs of a named workflow:

Statements and script declarations can not be mixed at the same level.

Abstract

unlike the linear number of k-mers in a sequence, the number of subsequences grows exponentially

best practices guidelines enforced by the project further ensure that the pipelines are robust, well-documented, and validated against real-world datasets.

Convert basic Nextflow modules to nf-core compatible modules

potential limitation of shotgun sequencing is the complexity of bioinformatics pipelines required for its analysis

scripts were made available for researchers

KBase platform [37] offers a user-friendly interface that allows for the analysis of data using most of the tools described in this publication

Unexpectedly, k-mer approaches resulted in rather high false positive rates, which may lead to misinterpretations of microbial community composition.

course demonstrates how to implement a simple variant calling pipeline with GATK (Genome Analysis Toolkit)

linear workflow

accessory files

understand what it does and how it should be configured before attempting to run it

ln -s $NXF_HOME/assets pipelines

nf-core project enforces strong guidelines for how pipelines are structured, and how the code is organized, configured and documented.

subworkflows

reuse chunks of code across different pipelines

flexible while minimizing maintenance burden

'utility' or housekeeping subworkflows

accessory functions

Nextflow works best with an active internet connection, as it is able to fetch all pipeline requirements.

It may seem like a lot of work to accomplish the same result as the original pipeline, but you do get all those lovely reports generated automatically

features of nf-core, including input validation and some neat metadata handling capabilities

unstructured data does not have a predefined data model, it is not easily processed and analyzed through conventional data tools and methods. It is best managed in nonrelational or NoSQL databases or in data lakes, which are designed to handle massive amounts of raw data in any format.

NCBI taxonomy dump

highlights source code in red for errors and yellow for warnings

Problems tab. Here, you can search for diagnostics

language server parses scripts and config files according to the Nextflow language specification, which is more strict than the Nextflow CLI

Include declarations in scripts and config files act as links, and ctrl-clicking them opens the corresponding script or config file.

view the definition of a symbol (e.g., a workflow, process, function, or variable),

can format your scripts and config files based on a standard set of formatting rules

right-click the symbol, select Rename Symbol

Format Document command in the command palette

There are three most popular pipelines used for NGSanalyses: QIIME, mothur and MetAMOS

Using first-passage analysis validated by Monte Carlo simulations, we quantitatively characterize nucleotide-specific error rates during RNA polymerase II transcription

New taxa are added to the Taxonomy database as data are deposited for them.

Visualization were created with pgfplots

knowledge gap between microbes that are only studied en masse as communities and those select few species whose molecular, genetic, or physiological diversity is studied in detail.

apply these tools to the myriad species that live in the understudied corners of our world.

Statisticians have taught for decades that the most efficient and robust experimental designs vary multiple factors simultaneously and then deconvolve the effects and interactions with simple statistical models

counted the number of PubMed articles that refer to each species in their title or abstract

few sips can help lower your overall body temperature, mimicking its natural decline before you sleep

PCN was then calculated for each sample as the ratio between the mean coverage of plasmid contigs and the mean coverage of the chromosome

enables us to follow the relative abundances of these strains over time in the human donor

For each microbiome sample, its MNS was derived by searching its sequence against those of all samples produced by past studies

evaluate UMA models on a diverse set of applications

Pairs of short reads with small edit distances, along with their unique molecular identifier tags, have been exploited to correct sequencing errors in both reads and tags.

Sync Google Drive in linux using podman and rclone

let’s automate the command using systemd

First, we create a symbolic link to the unit file in the /etc/systemd/system directory.

It’s easier than teaching kids, and it’s more exciting in some ways,” said an AI trainer

More recently, Lior Pachter has argued that t-SNE and the related UMAP do not serve a meaningful purpose in data analysis and are only useful for producing art.

feature extraction attempts to reduce the dimensionality of a dataset by building a compressed representation of the input features

Methods like t-stochastic neighbor embedding (t-SNE) and uniform manifold approximation and projection (UMAP) faithfully capture and reveal local and non-linear relationships in complex microbiome datasets, but their tuning is finicky

Whether taxonomic or functional profiles provide a better discriminatory power in downstream analysis is subject to debate [23,24,25].

Different channels can have the same package, so conda must handle these channel collisions.

many environments can still not be classified well at the species level11 because of database incompleteness. Our strategy for tackling this problem was to design sylph so that researchers can create customized databases from their novel genomes or MAGs, although this requires the generation of new genomes for researchers working in undercharacterized microbiomes.

make it too easy to build manufacturing sites that could cause more pollution

avoid rigorous environmental review

16S amplicon sequencing exhibited extreme bias toward the most abundant taxon

let AI surface which applicants match your given criteria.

especially interested incandidates with prior wet lab experience and a generalist quantitative mindset.

Minimap2 follows a typical seed-chain-align procedure as is used by most full-genome aligners

Myers-Briggs Type Indicator

We extracted 202 question-answer pairs from the KB and 39 questions generated by GPT-4 for training and testing purposes

David A. Relman,

capability to create mirror life is likely at least a decade away and would require large investments and major technical advances

$50 copay/visit

If you make a 1 byte change and push the file again, you'll use another 500 MB of storage and no bandwidth

KPop, a novel versatile method based on full k-mer spectra and dataset-specific transformations, through which thousands of assembled or unassembled microbial genomes can be quickly compared

simplified signatures (“sketches”) based on some dataset-independent choices

the most relevant methods to classify or compare microbial genomes based on k-mers can be broadly divided into the following categories:

KPop is able to accurately map sequences onto a low-dimensional space