pretraining_tp (int, optional, defaults to 1) — Experimental feature. Tensor parallelism rank used during pretraining. Please refer to this document to understand more about it. This value is necessary to ensure exact reproducibility of the pretraining results. Please refer to this issue.

max_position_embeddings (int, optional, defaults to 2048) — The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048).

On the first GPU, the prompts will be ["a dog", "a cat"], and on the second GPU it will be ["a chicken", "a chicken"]. Make sure to drop the final sample, as it will be a duplicate of the previous one.

what if we then wanted to do something with the results of all the GPUs? (Say gather them all and perform some kind of post processing) You can pass in apply_padding=True to ensure that the lists of prompts are padded to the same length, with extra data being taken from the last sample. This way all GPUs will have the same number of prompts, and you can then gather the results.

This is only needed when trying to perform an action such as gathering the results, where the data on each device needs to be the same length. Basic inference does not require this.

With 🤗 Accelerate, we can simplify this process by using the Accelerator.split_between_processes() context manager (which also exists in PartialState and AcceleratorState). This function will automatically split whatever data you pass to it (be it a prompt, a set of tensors, a dictionary of the prior data, etc.) across all the processes (with a potential to be padded) for you to use right away.

torch.distributed.get_rank()

Once you’ve finished training, make sure to run Accelerator.end_training() so that all the trackers can run their finish functionalities if they have any. Copied accelerator.end_training()

When you are ready to log any data, Accelerator.log() should be used. A step can also be passed in to correlate the data with a particular step in the training loop. Copied accelerator.log({"train_loss": 1.12, "valid_loss": 0.8}, step=1)

At the start of your experiment Accelerator.init_trackers() should be used to setup your project

potentially add any experiment hyperparameters to be logged: Copied hps = {"num_iterations": 5, "learning_rate": 1e-2} accelerator.init_trackers("my_project", config=hps)

accelerator.init_trackers("my_project"

For printing statements you only want executed once per machine, you can just replace the print function by accelerator.print.

The local means per machine: if you are running your training on two servers with several GPUs, the instruction will be executed once on each of those servers. If you need to execute something only once for all processes (and not per machine) for instance, uploading the final model to the 🤗 model hub, wrap it in a test like this: Copied if accelerator.is_main_process:

progress_bar = tqdm(range(args.max_train_steps), disable=not accelerator.is_local_main_process)

Some of your instructions only need to run for one process on a given server: for instance a data download or a log statement.

You should only pass the learning rate scheduler to prepare() when the scheduler needs to be stepped at each optimizer step.

use the option split_batches=True when creating and initializing your Accelerator, in which case the batch size will always stay the same, whether you run your script on 1, 2, 4, or 64 GPUs.

The report includes the number of training steps, number of skipped optimizer updates (likely due to overflows in mixed-precision training), current learning rate, and current momentum.

While DeepSpeed has a pip installable PyPI package, it is highly recommended that it gets installed from source to best match your hardware and also if you need to enable certain features, like 1-bit Adam, which aren’t available in the pypi distribution.

While the fp16 weights are fine for resuming training, if you finished finetuning your model and want to upload it to the models hub or pass it to someone else you most likely will want to get the fp32 weights.

This ideally shouldn’t be done during training since this is a process that requires a lot of memory, and therefore best to be performed offline after the training is complete.

DeepSpeed stores fp32 master weights in its custom checkpoint optimizer files, which are global_step*/*optim_states.pt (this is glob pattern), and are saved under the normal checkpoint.

If you use gradient accumulation with bf16-enabled, you need to be aware that it’ll accumulate gradients in bf16, which may not be what you want due to this format’s low precision, as it may lead to a lossy accumulation. A work is being done to fix that and provide an option to use a higher precision dtype (fp16 or fp32).

bf16 has the same dynamic range as fp32 and thus doesn’t require loss scaling.

With the 🤗 Trainer you can use --tf32 to enable it, or disable it with --tf32 0 or --no_tf32. By default the PyTorch default is used.

If you’re using the Ampere-architecture based GPU, pytorch version 1.7 and higher will automatically switch to using the much more efficient tf32 format for some operations, but the results will still be in fp32.

the only time you will want to not use it is when the model you’re using doesn’t behave well under this training mode. Typically this happens when the model wasn’t pretrained in the fp16 mixed precision (e.g. often this happens with bf16-pretrained models).

You can also take the HF Transformers modeling code and replace torch.utils.checkpoint with the DeepSpeed’s API. The latter is more flexible since it allows you to offload the forward activations to the CPU memory instead of recalculating them.

HF Transformers models don’t know anything about DeepSpeed’s activation checkpointing, so if you try to enable that feature in the DeepSpeed config file, nothing will happen. Therefore you have two ways to take advantage of this very beneficial feature: If you want to use a HF Transformers models you can do model.gradient_checkpointing_enable() or use --gradient_checkpointing in the HF Trainer, which will automatically enable this for you. torch.utils.checkpoint is used there.

Activation checkpointing and gradient checkpointing are two distinct terms that refer to the same methodology. It’s very confusing but this is how it is.

Before beginning to train BLOOM-176B I spent 2 days on this process and was able to increase throughput from 90 to 150 TFLOPs! This effort saved us more than one month of training time.

Here is a full ZeRO-3 all-enabled manually set configuration file. It is here mainly for you to see what the typical values look like, but we highly recommend using the one with multiple auto settings in it.

Here is a full ZeRO-2 all-enabled manually set configuration file. It is here mainly for you to see what the typical values look like, but we highly recommend using the one with multiple auto settings in it.

It’s possible to adjust ZeRO-3 configuration to make it perform closer to ZeRO-2: set stage3_param_persistence_threshold to a very large number - larger than the largest parameter, e.g., 6 * hidden_size * hidden_size. This will keep the parameters on the GPUs. turn off offload_params since ZeRO-2 doesn’t have that option.

modern NVMe transfer speeds in mind (as of this writing one can have ~3.5GB/s read, ~3GB/s write peak speeds)

Make sure that your nvme_path is actually an NVMe, since it will work with the normal hard drive or SSD, but it’ll be much much slower.

sub_group_size controls the granularity in which parameters are updated during optimizer steps. Parameters are grouped into buckets of sub_group_size and each buckets is updated one at a time. When used with NVMe offload in ZeRO-Infinity, sub_group_size therefore controls the granularity in which model states are moved in and out of CPU memory from NVMe during the optimizer step. This prevents running out of CPU memory for extremely large models.

ZeRO-Infinity allows for training incredibly large models by extending GPU and CPU memory with NVMe memory.

When used with NVMe offload in ZeRO-Infinity, sub_group_size therefore controls the granularity in which model states are moved in and out of CPU memory from NVMe during the optimizer step. This prevents running out of CPU memory for extremely large models. You can leave sub_group_size to its default value of 1e9 when not using NVMe offload.

stage3_gather_16bit_weights_on_model_save enables model fp16 weights consolidation when model gets saved. With large models and multiple GPUs this is an expensive operation both in terms of memory and speed. It’s currently required if you plan to resume the training.

The following configuration values depend on the model’s hidden size: reduce_bucket_size: hidden_size*hidden_size stage3_prefetch_bucket_size: 0.9 * hidden_size * hidden_size stage3_param_persistence_threshold: 10 * hidden_size therefore set these values to auto and the Trainer will automatically assign the recommended values. But, of course, feel free to set these explicitly as well.

“reuse distance” is a metric we are using to figure out when will a parameter be used again in the future, and we use the stage3_max_reuse_distance to decide whether to throw away the parameter or to keep it. If a parameter is going to be used again in near future (less than stage3_max_reuse_distance) then we keep it to reduce communication overhead.

stage3_max_live_parameters is the upper limit on how many full parameters you want to keep on the GPU at any given time.

1e9 would consume ~2GB. The memory is shared by stage3_max_live_parameters and stage3_max_reuse_distance, so it’s not additive, it’s just 2GB total.

stage3_max_live_parameters and stage3_max_reuse_distance. They should have minimal impact on performance unless you are doing activation checkpointing.

This feature can improve the throughput at the cost of making less memory available to other processes. Pinned memory is set aside to the specific process that requested it and its typically accessed much faster than normal CPU memory.

The following is an example of configuration for ZeRO stage 3:

gradient accumulation steps (more copying between optimizer steps)

enabling offload_optimizer should reduce GPU RAM usage (it requires "stage": 2)

an example of configuration for ZeRO stage 2:

currently DeepSpeed doesn’t validate parameter names, so if you misspell any, it’ll use the default setting for the parameter that got misspelled.

This section has to be configured exclusively via DeepSpeed configuration - the Trainer provides no equivalent command line arguments.

The zero_optimization section of the configuration file is the most important part (docs), since that is where you define which ZeRO stages you want to enable and how to configure them.

The first one is not quite interesting for scalability purposes

In your own programs, you can also use the following approach if you’d like to modify the DeepSpeed config as a master and configure TrainingArguments based on that. The steps are: Create or load the DeepSpeed configuration to be used as a master configuration Create the TrainingArguments object based on these values Do note that some values, such as scheduler.params.total_num_steps are calculated by Trainer during train, but you can of course do the math yourself.

even though we use the mixed precision training, using full precision checkpoint is the best practice.

还有一个很重要的，混音。我听了很多AI翻唱的，目前出了我和Eternity丨L，好像没有其他UP做混音的，都是干音+伴奏。

model_input_names (List[string], optional) — The list of inputs accepted by the forward pass of the model (like "token_type_ids" or "attention_mask"). Default value is picked from the class attribute of the same name.

Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it’s useful.

Data collators are objects that will form a batch by using a list of dataset elements as input.

⚠️ Shifting the inputs and labels to align them happens inside the model, so the data collator just copies the inputs to create the labels.

DataCollatorForLanguageModeling supports both masked language modeling (MLM) and causal language modeling (CLM). By default it prepares data for MLM, but we can switch to CLM by setting the argument mlm=False:

DataCollatorForLanguageModeling collator, which is designed specifically for language modeling (as the name subtly suggests). Besides stacking and padding batches, it also takes care of creating the language model labels

Use the end-of-sequence token as the padding token and set mlm=False. This will use the inputs as labels shifted to the right by one element:

This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop.

Converts a string in a sequence of tokens, using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Takes care of added tokens.

Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing self.convert_tokens_to_ids(self.tokenize(text)).

convert_ids_to_tokens

解码阶段给定当前生成词在第 ii i 个transformer层的向量表示为 ti∈Rb×1×ht^{i}\in R^{b\times 1\times h}t^{i}\in R^{b\times 1\times h} 。推断计算分两部分：更新KV cache和计算第 iii 个transformer层的输出。

embedding层不需要中间激活

对于 softmax()softmax()softmax() 函数，需要保存函数的输入 QKTQK^TQK^T ，占用显存大小为 2bs2a2bs^2a2bs^2a ，这里的 aaa 表示注意力头数。

词嵌入矩阵的参数量也较多，词向量维度通常等于隐藏层维度 hhh ，词嵌入矩阵的参数量为 VhVhVh

最后的输出层的权重矩阵通常与词嵌入矩阵是参数共享的

MLP块

self-attention块

synced_gpus (bool, optional) — Whether to continue running the while loop until max_length. Unless overridden this flag will be set to True under DeepSpeed ZeRO Stage 3 multiple GPUs environment to avoid hanging if one GPU finished generating before other GPUs. Otherwise it’ll be set to False.

modifying your script to run on GPT-J with FP16 on an 3090, with input_ids.shape[1]=16 and max_new_tokens=256, we get: 14071MB of GPU usage with use_cache=False 13233MB of GPU usage with use_cache=True The difference becomes more visible with large models and large sequence lengths

Our new technologies for optimizing inference cost and latency include: Inference-adapted parallelism allows users to efficiently serve large models by adapting to the best parallelism strategies for multi-GPU inference, accounting for both inference latency and cost.Inference-optimized CUDA kernels boost per-GPU efficiency by fully utilizing the GPU resources through deep fusion and novel kernel scheduling.Effective quantize-aware training allows users to easily quantize models that can efficiently execute with low-precision, such as 8-bit integer (INT8) instead of 32-bit floating point (FP32), leading to both memory savings and latency reduction without hurting accuracy.

I wonder if the current approach of statically-compiled CUDA kernels is sustainable. Perhaps there is value to considering JIT compilation, e.g. with Triton or NVRTC?

In 🤗 Accelerate this conversion happens automatically when calling prepare() and passing in your model.

For best performance, this data collator should be used with a dataset having items that are dictionaries or BatchEncoding, with the "special_tokens_mask" key, as returned by a PreTrainedTokenizer or a PreTrainedTokenizerFast with the argument return_special_tokens_mask=True.

Data collator that will dynamically pad the inputs received, as well as the labels.

batched (bool, defaults to False) — Provide batch of examples to function.

when tokenizing each element into chunks of the specified context size, we create many samples from each document. We just need to make sure to delete the existing columns, since they have a conflicting size. If we wanted to keep them, we could repeat them appropriately and return them within the Dataset.map() call:

Answer grading is difficult in general. This grading logic is designed to be conservative and will sometimes reject correct answers, though it does so less frequently than the normalization logic from MATH. Our logic might sometimes admit incorrect answers, though we've put effort into minimizing this.

// Total time in milliseconds spent on labeling this solution. "total_time": 278270,

Enlarge outputentropyAdd noises ontoparameter

In this way, we do not have to collection data after each upd

Data collection is in the “forloop” of training iteratio

Minus by a baseline 𝑏−𝑏−𝑏−𝑏−𝑏Make 𝐺𝑡′ have positive and negative value

𝐺1′ = 𝑟1 + 𝛾𝑟2 + 𝛾2𝑟3 + ......

Make it take (or don’t take) a specific action ො𝑎 givenspecific observatio

The pop-up menu at the lower left of the interface allows you to choose among the following models:

Predictor values are displayed in the text boxes on the horizontal axis and are marked by vertical dashed blue lines in the plots.

rstool plots a 95% simultaneous confidence band for the fitted response surface as two red curves.

If spec.loader.create_module does not return None, then any pre-existing attributes will not be reset. Also, no AttributeError will be raised if triggered while accessing spec or setting an attribute on the module.

List of gradio.components to use as inputs.

In practice, very low pass rates are difficultor impossible to estimate, so we restrict to problems P and models M such that given some largesample budget, every problem is solved at least once by every model.

the Inverse Scaling Prize [ 44 ] proposedseveral tasks for which model performance decreases as a function of scale. Similarly to a recentresult by Wei et al. [45], we find that GPT-4 reverses this trend, as shown on one of the tasks calledHindsight Neglect [46] in Figure 3.

Predictions on the other five buckets performed almost as well, the main exception beingGPT-4 underperforming our predictions on the easiest bucket.

We chose to look at loss because it tends to be less noisy than other measures acrossdifferent amounts of training compute.

we predicted GPT-4’s final loss on ourinternal codebase (not part of the training set) by fitting a scaling law with an irreducible loss term(as in Henighan et al. [15]): L(C) = aCb + c

A power law fit to the smaller models (excluding GPT-4) is shown as the dottedline; this fit accurately predicts GPT-4’s performance.

快速傅立叶变换的核心思想也是将系数向量迅速变换为点值向量，再迅速的将点值向量还原成系数向量，其中还原的操作我们称之为IDFTIDFTIDFT。

tol, rtolfloat, optionalIteration stops when error between last two iterates is less than tol OR the relative change is less than rtol.

Why do we use the n + 1 order derivative of f to describe the error of a polynomial of degree atmost n? The intuition is that the n + 1 order derivative of a degree n polynomial is identically 0, sothat the difference between the actual function and the polynomial interpolant can be encapsulatedby this derivative.

我便用一段话概括：学长在 CMU Joint program 跟着 Neubig 暑研，然后 18 年入学了 CMU MLT，20 年毕业后去了 Hudson River Trading 做 Algorithm Engineer。现在有了 H1B，人在加州，做的工作和 NLP 没有任何关系。至于 MLT，他入学的时候 MLT 只有 30 人，1 /3 的人能够拿到老板的赞助免除学费和生活费，虽然也是僧多粥少。没有和 CMU 的 joint program，他认为自己去不了 CMU MLT。此外，MLT 毕业时，转本校 PhD 的也不少。至于转本校，基本是 2 + (3+)，很少需要 2 + 5 年…

如果一个模块导入另一个模块，而后者又导入另一个模块，则第一个模块的 sys.path 是解释器搜索第二个导入语句的位置。

sys.path 并不会依赖当前程序的工作路径 - os.getcwd()，仅仅依赖第一个脚本所在的路径：

在解释器环境下，sys.path[0] 就是解释器启动时所在的路径

(str, Parameter) – Tuple containing the name and parameter

torch.nn.init.constant_(tensor, val)[source] Fills the input Tensor with the value val\text{val}val.

torch.nn.init.normal_(tensor, mean=0.0, std=1.0)[source] Fills the input Tensor with values drawn from the normal distribution N(mean,std2)\mathcal{N}(\text{mean}, \text{std}^2)N(mean,std2).

All the functions in this module are intended to be used to initialize neural network parameters, so they all run in torch.no_grad() mode and will not be taken into account by autograd.

为什么训练的时候warm up这么重要？这个问题目前还没有被充分证明，我们只能从直觉上和已有的一些论文[1,2,3]得到推测：有助于减缓模型在初始阶段对mini-batch的提前过拟合现象，保持分布的平稳有助于保持模型深层的稳定性

在大型网络训练初期，我们需要用较小的学习率先学n个step

Open-Source Vizier implements a variety of sophisticated algorithms for tuning ML models, including Bayesian Optimization algorithms.

Summary: Bayesian optimization tools are a compelling option once we’re done exploring for good search spaces and have decided what hyperparameters even should be tuned at all.

However, we should only adopt changes that produce improvements that outweigh any complexity they add.

Usually, we can get away with only recharacterizing the trial variance after major changes to the pipeline

before adopting a candidate change, consider running the best trial N times to characterize the run-to-run trial variance.

It is all well and good to make comparisons of validation error rates estimated on a finite validation set using fastidious statistical tests, but often the trial variance alone can produce statistically significant differences between two different trained models that use the same hyperparameter settings.

the most important sources of variation that might cause such an inconsistent result

different random seeds. For example, different random initializations, training data shuffles, dropout masks, patterns of data augmentation operations, and orderings of parallel arithmetic operations, are all potential sources of trial variance.

Summary: Examining the training curves is an easy way to identify common failure modes and can help us prioritize what actions to take next.

In general, it can be very difficult to know if the search space has been sampled densely enough. 🤖

basic hyperparameter axis plots where we plot the validation objective value versus one of the hyperparameters (e.g. learning rate). Each point on the plot corresponds to a single trial.

If all trials are infeasible for learning rates greater than some threshold value, and if the best performing trials have learning rates at the edge of that region, the model may suffer from stability issues preventing it from accessing higher learning rates.

A search space is suspicious if the best point sampled from it is close to its boundary. We might find an even better point if we expanded the search range in that direction.

For example, do the best trials have training curves consistent with problematic overfitting?

In some cases, a large number of infeasible points can indicate a bug in the training code.

reparameterizing the search space

infeasible (i.e. trials that diverge, get really bad loss values, or fail to run at all because they violate some implicit constraint)

Before analyzing a given set of experiments to make progress toward their original goal, we should ask ourselves the following additional questions

Since running experiments can be expensive, we also want to take the opportunity to extract other useful insights from each group of experiments, even if these insights are not immediately relevant to the current goal

For example, if our goal is to select the best optimizer out of Nesterov momentum and Adam, we could create one study in which optimizer="Nesterov_momentum" and the nuisance hyperparameters are {learning_rate, momentum}, and another study in which optimizer="Adam" and the nuisance hyperparameters are {learning_rate, beta1, beta2, epsilon}.

it ensures that we obtain a relatively uniform sampling of values of the scientific hyperparameters

searches the scientific parameters uniformly

conditional hyperparameters can cause problems since it is hard to specify a search space unless the set of nuisance hyperparameters is the same for all values of the scientific hyperparameters.

include the scientific parameters in the same search space as the nuisance hyperparameters and use a search algorithm to sample values of both the scientific and nuisance hyperparameters in a single study.

We can use any gradient-free optimization algorithm, including methods such as Bayesian optimization or evolutionary algorithms, to optimize over the nuisance hyperparameters

A study specifies a set of hyperparameter configurations to be run for subsequent analysis. Each configuration is called a "trial".

the more nuisance hyperparameters we attempt to tune, the greater the risk we fail to tune them sufficiently well for each setting of the scientific hyperparameters and end up reaching the wrong conclusions from our experiments.

With limitless resources, we would leave all non-scientific hyperparameters as nuisance hyperparameters so that the conclusions we draw from our experiments are free from caveats about fixed hyperparameter values.

When designing a new round of experiments, we first identify the scientific hyperparameters for our experimental goal. At this stage, we consider all other hyperparameters to be nuisance hyperparameters.

The activation function could be a fixed hyperparameter if we have determined in prior experiments that the best choice of activation function is not sensitive to model depth, or if we are willing to limit our conclusions about the number of hidden layers to only cover this specific choice of activation function

The learning rate is a nuisance hyperparameter because we can only fairly compare models with different numbers of hidden layers if the learning rate is tuned separately for each number of layers (the optimal learning rate generally depends on the model architecture).

When we are eventually ready to be greedy, we can focus purely on the validation error even if the experiments aren't maximally informative about the structure of the tuning problem.

Identify which hyperparameters the validation error is most sensitive to, which hyperparameters interact the most and therefore need to be re-tuned together, and which hyperparameters are relatively insensitive to other changes and can therefore be fixed in future experiments.

Although one might think we would spend most of our time trying to maximize performance on the validation set, in practice we spend the majority of our time trying to gain insight into the problem, and comparatively little time greedily focused on the validation error.

We call it a launch when we update our best configuration (which may or may not correspond to an actual launch of a production model).

if an unnecessarily large step budget is chosen initially, it might be hard to change it down the road, e.g. once the learning rate schedule is tuned for that number of steps.

training for fewer steps means that each training run is faster and uses fewer resources, boosting tuning efficiency by reducing the time between cycles and allowing more experiments to be run in parallel

training for more steps can improve performance and makes hyperparameter tuning easier (see Shallue et al. 2018).

at minimum means that the trained model performs much better than random chance on the validation set

For example, start with a constant learning rate before adding fancy decay schedules.

"Simple" means avoiding bells and whistles wherever possible; these can always be added later

Before beginning hyperparameter tuning we must determine the starting point. This includes specifying (1) the model configuration (e.g. number of layers), (2) the optimizer hyperparameters (e.g. learning rate), and (3) the number of training steps.

Batch norm is complicated and, in general, should use a different batch size than the gradient computation to compute statistics.

The hyperparameters that interact most strongly with the batch size, and therefore are most important to tune separately for each batch size, are the optimizer hyperparameters (e.g. learning rate, momentum) and the regularization hyperparameters.

The optimal values of most hyperparameters are sensitive to the batch size.

Choosing the batch size to minimize resource consumption

Choosing the batch size to minimize training time

shouldn't be used to directly tune the validation set performance

This is particularly relevant in the beginning stages of a project when we are trying to find the best values of various other hyperparameters (e.g. architecture hyperparameters) while treating optimizer hyperparameters as nuisance parameters.

If the training throughput increases only up to some maximum batch size, then we should only consider batch sizes up to that maximum batch size, even if a larger batch size is supported by the hardware.

If this is not the case then the training pipeline has a bottleneck such as I/O or synchronization between compute nodes.

When the accelerators aren't yet saturated, if the batch size doubles, the training throughput should also double (or at least nearly double).

larger batch sizes can be more prone to overfitting and may require stronger regularization and/or additional regularization techniques

the difference in validation set performance between two batch sizes typically goes away if the training pipeline is optimized independently for each batch size.

Increasing the batch size may either decrease, increase, or not change the resource consumption.

Summary: When starting a new project, try to reuse a model that already works.

When possible, try to find a paper that tackles something as close as possible to the problem at hand and reproduce that model as a starting point.

There is already a pipeline set up that does training and evaluation, and it is easy to execute training and prediction jobs for various models of interest

anguage modeling (LM) loss to generate captions given images

nless otherwise specified, all results re-ported in this paper as “BLIP” uses ViT-B

shares the same cross-attention layers

image-text matching (ITM) loss to distinguishbetween positive and negative image-text pairs

an image-text contrastive (ITC) loss to align the vision and language representations.

layers except for SA leads to better performance comparedto not sharing

f the SA layers are shared,the model’s performance would degrade due to the conflictbetween the encoding task and the decoding task

During pre-training, the text encoder and decoder share allparameters except for the self-attention layers

e usethe same pre-training dataset as Li et al. (2021a) with 14Mimages in total, including two human-annotated datasets(COCO and Visual Genome (Krishna et al., 2017)), andthree web datasets (Conceptual Captions (Changpinyo et al.,2021), Conceptual 12M (Changpinyo et al., 2021), SBU cap-tions (Ordonez et al., 2011)). We also experimented with anadditional web dataset, LAION (Schuhmann et al., 2021),which contains 115M images with more noisy texts1

ncrease theimage resolution to 384 × 384 during finetuning

ake random image crops ofresolution 224 × 224 during pre-training

AdamW (Loshchilov & Hutter, 2017)optimizer with a weight decay of 0.05. The learning rateis warmed-up to 3e-4 (ViT-B) / 2e-4 (ViT-L) and decayedlinearly with a rate of 0.85.

a [CLS] token is appended to the beginningof the text input to summarize the sentence

as been adopted by themore recent methods (Li et al., 2021a; Kim et al., 2021).

ViTis more computation-friendly

using pre-trained object detectorsfor visual feature extraction (Chen et al., 2020)

divides an input image intopatches and encodes them as a sequence of embeddings,with an additional [CLS] token to represent the global im-age feature

3.1. Model Architecture

Captioning and Filtering (CapFilt): a new dataset boos-trapping method for learning from noisy image-text pairs.

An MED can operate either asa unimodal encoder, or an image-grounded text encoder,or an image-grounded text decode

Multimodal mixture of Encoder-Decoder (MED): a newmodel architecture for effective multi-task pre-training andflexible transfer learning

encoder-decoder models have not beensuccessfully adopted for image-text retrieval tasks.

ncoder-based models are less straightfor-ward to directly transfer to text generation tasks (e.g. imagecaptioning),

Demonstrates strong generalization ability when directly transferred to video-language tasks in a zero-shot manner.

using a stochastic decoding method (nucleus sampling) is better than using beam search for caption generation, due to the higher level of diversity in the synthetic captions.