Although the agents do not yet attainhuman-level performance, we will soon describe scaling experiments which suggest thatthis gap could be closed substantially simply by collecting more data.

The regularisation schemes presented in the last section can improve the generalisationproperties of BC policies to novel inputs, but they cannot train the policy to exert active con-trol in the environment to attain states that are probable in the demonstrator’s distribution.

The mouselook action distribution is in turn also defined autoregressively: the first sampled actionsplits the window bounded by(−1,1)×(−1,1)in width and height into 9 squares. Thesecond action splits the selected square into 9 further squares, and so on. Repeating thisprocess several times allows the agent to express any continuous mouse movement up to athreshold resolution.

effective di-mensionality of a Bayesian neural network is inverselyproportional to the variance of the posterior distribu-tion.

Yet this also implies non i.i.d. samples! Indeed, even if one could directly sample from the state-action distribution (like having its analytical form or an infinite experience replay buffer) and thus draw i.i.d. samples, the dependency will occur across optimization steps: if I draw a sample and use it to update my policy, I also update the distribution from which I will draw my next sample and then my next sample depends on my previous sample (since it conditioned my policy update).

Prefix-tuning prepends a sequence ofcontinuous task-specificvectors to the input, whichwe call aprefix, depicted by red blocks in Figure 1(bottom). For subsequent tokens, the Transformercan attend to the prefix as if it were a sequence of“virtual tokens”, but unlike prompting, the prefixconsists entirely of free parameters which do notcorrespond to real tokens.

I guess a stepping-stone towards this would be to optimize morphological growth processes to generate a body with a particular form in 3D (that would be quite similar to the differentiable CA, except that here the “cells” move in 3D space and have physical interaction that depend on their internal parameters and states)

(and that would be also novel to use a population-based IMGEPs using gradient descent for local optimization towards self-generated goals)

For this reason, we wereunable to collect baselines such as an equivalent amount of high-quality human demonstrations forsupervised baselines. See D for more discussion. We leave this ablation to future work.

it’s unclear how much one can optimizeagainst the reward model until it starts giving useless evaluations.

Previous work on fine-tuning language models from human feedback [73] reported “a mismatchbetween the notion of quality we wanted our model to learn, and what the humans labelers actuallyevaluated”, leading to model-generated summaries that were high-quality according to the labelers,but fairly low-quality according to the researchers.

We rely on detailed procedures toensure high agreement between labelers and us on the task, which we describe in the next section

We conjecture that this gap occurs because the models “cheat” by only optimizing for performance on the benchmark, much like a student who passed an exam by studying only the questions on past years’ exams.

n fact, without visiting any states at all, sincethe queries are synthetic.

he x-axis represents the number of queries to the user, where each queryelicits a label for a single state transition(s, a, s0).

having to visit unsafe states during the training process

As discussed inSection4.3and illustrated in the right-most plot of Figure5, the baselines learn a reward model that incorrectly ex-trapolates that continuing up and to the right past the goalregion is good behavior.

⌧query= maxz0,a0,z1,...,zTJ(⌧)+logp(⌧)

Here, the states2R64⇥64⇥3is anRGB image with a top-down view of the car (Figure3), andthe actiona2R3controls steering, gas, and brake

he idea is to elicit labels for examples that themodel is least certain how to label, and thus reduce modeluncertainty.

To simplify our experiments,we sample trajectories⌧by following random policies thatexplore a wide variety of states. We use the observed trajec-tories to train a likelihood model

(4) maximize novelty of trajecto-ries regardless of predicted rewards, to improve the diversityof the training data.

In complex domains,the user may not be able to anticipate all possible agentbehaviors and specify a reward function that accuratelydescribes user preferences over those behaviors

it is far easier to obtain reliability beyond a certain margin by mechanisms in the end hosts of a network rather than in the intermediary nodes,[nb 4] especially when the latter are beyond the control of, and not accountable to, the former

all causal explanationsare necessarily robust in this extreme case

Goldblum et al.[119]which empirically observes that the large width behavior of ResidualNetworks does not conform to the infinite-width limit.

WhileCNN-VECpossess translation equivariance but not invariance (§3.11), we believe it can effectivelyleverage equivariance to learn invariance from data

This is caused by poor conditioning of pooling networks. Xiao et al.[33](Table 1) show that theconditioning at initialization of aCNN-GAPnetwork is worse than that ofFCNorCNN-VECnetworksby a factor of the number of pixels (1024 for CIFAR-10). This poor conditioning of the kerneleigenspectrum can be seen in Figure 8. For linearized networks, in addition to slowing training by afactor of 1024, this leads to numerical instability when usingfloat32

egularization parameter

We add the superscript “all" to emphasize that gradient-based training of the networks is alwaysperformed on the entire dataset, while NNGP inference is performed on sub-sampled datasets.

With few exceptions (Carl-son et al., 2010), machine learning models havebeen confined to IID datasets that lack the structurein time from which humans draw correlations aboutlong-range causal dependencies

how pretraining obfuscates ourability to measure generalization (Linzen, 2020)

but even com-plex simulation action spaces can be discretizedand enumerated.

models the listener’s desires and experiences explic-itly

Collecting data about rich natural sit-uations is often impossible.

Meanwhile, it is precisely human’sability to draw on past experience and make zero-shot decisions that AI aims to emulate

Second, current cross entropy training losses ac-tively discourage learning the tail of the distribu-tion properly, as statistically infrequent events aredrowned out (Pennington et al., 2014; Holtzmanet al., 2020).

it is unlikely that universal function approximatorssuch as neural networks would ever reliably positthat people, events, and causality exist without be-ing biased towards such solutions (Mitchell, 1980)

(which are usually thrown out beforethe dataset is released)

persistent enough to learn the effects of actions.

and active experimentation is keyto learning that effec

participatein lin-guistic activity, such as negotiation (Yang et al.,2019a; He et al., 2018; Lewis et al., 2017), collab-oration (Chai et al., 2017), visual disambiguation(Anderson et al., 2018; Lazaridou et al., 2017; Liuand Chai, 2015), or providing emotional support(Rashkin et al., 2019).

Framing, such as suggesting that achatbot speaks English as a second language

Robotics and embodiment are not available inthe same off-the-shelf manner as computer visionmodels.

(Liet al., 2019b; Krishna et al., 2017; Yatskar et al.,2016; Perlis, 2016)

Models must be ableto watch and recognize objects, people, and activi-ties to understand the language describing them

Learned, physical heuristics, such as thefact that a falling cat will land quietly, are general-ized and abstracted into language metaphors likeas nimble as a cat(Lakoff, 1980).

Language learning needs perception, because per-ception forms the basis for many of our semanticaxioms

As text pretraining schemes seem to be reach-ing the point of diminishing returns,

parked my car in the compact park-ing space because it looked (big/small) enough

Continuing to expandhardware, data sizes, and financial compute costby orders of magnitude will yield further gains, butthe slope of the increase is quickly decreasing.

cale in data andmodeling has demonstrated that a single represen-tation can discover both rich syntax and semanticswithout our help (Tenney et al., 2019).

You can’t learn language from the radio.

The futility of learning language from lin-guistic signal alone is intuitive, and mirrors thebelief that humans lean deeply on non-linguisticknowledge (Chomsky, 1965, 1980).

from their use by people to communi-cate

Natural language processing is a diverse field,and progress throughout its development hascome from new representational theories, mod-eling techniques, data collection paradigms,and tasks.

success-ful linguisticcommunicationrelies on a sharedexperience of the world. It is this shared expe-rience that makes utterances meaningful

share attention

Any smaller subset of these compe-tencies is not sufficient to develop proper language/communi-cation skills, and further, the development of language clearlybootstraps better motor and affordance learning and/or sociallearning.

Hisnot learnable at rate faster thanR

For simplicity of exposition, we have stated a definition corresponding todeterministicalgorithms, to avoidthe notational inconvenience required to formally define randomized algorithms in this contex

erP

That is,everynontrivial classHis eitheruniversally learnable at an exponential rate (but not faster), or isuniversally learnable at a linearrate (but not faster), or is universally learnable but necessarily with arbitrarily slow rates

for any learning algorithm, there is a realizable distributionPwhoselearning curve decays no faster than a linear rate (Schuurmans, 1997)

S({Oμ(x)})

2−Nq(h∗)eN(h∗m−log coshh∗)

argument [10] converts Eq. (1) withα= 1 into the state-ment that, for a large system,N→ ∞, the energy andentropy are exactly equal (up to a constant) to leadingorder inN.

Because the exponentαN1for language models, we can approximateN−αN≈1−αNlog(N)to obtainequation 4.1.

could be misleading if the models have not all been trained fully to convergence

which makes the interpretation ofL(N)difficult.

mattn

There we also show trends forthe training loss, which do not adhere as well to a power-law form, perhaps because of the implicit curriculumin the frequency distribution of easy and hard problems

the poor loss onthese modules would dominate the trends

easier problems will naturally appear more often than more difficult problems

We sample the default mixture of easy, medium, and hard problems, withouta progressive curriculum.

context length of3200tokens per image/caption pair

We revisit the question “Is a picture worth a thousand words?” by comparing the information-contentof textual captions to the image/text mutual information

but we will onlyapply it along the time dimensiont.

The key point of this work is that based on observing a single sample from a subpopulation, it isimpossible to distinguish samples from “borderline” populations from those in the “outlier” ones. Thereforean algorithm can only avoid the risk of missing “borderline” subpopulations by also memorizing examplesfrom the “outlier” subpopulations.

e over parameters and the function-space posterior co-variance. Red indicates the under-parameterized setting, yellowthe critical regime withp≈n, and green the over-parameterizedregime.

We see wide but shallow models overfit, providing low train loss, but high testloss and high effective dimensionality.

subspace and ensembling methods could beimproved through the avoidance of expensive com-putations within degenerate parameter regimes

w

Our theoryagain perfectly fits the experiments.

K

marginal training data point causes greater reduc-tion in relative error for low frequency modes than for highfrequency modes.

Each expert in the MoE layer receives a combinedbatch consisting of the relevant examples from all of the data-parallel input batches.

A prior over parametersp(w)combines with the functionalform of a modelf(x;w)to induce a distribution over func-tionsp(f(x;w)). It is this distribution over functions thatcontrols the generalization properties of the model; the priorover parameters, in isolation, has no meaning.

Distance between the truepredictive distribution and the approximation

coherent

As the effective dimensionality increases, so doesthe dimensionality of parameter space in which theposterior variance has contracted.

In the notation of Section 3, pointsω∈Ωrepresent possible samples. In our setting, each sam-ple represents a complete record of a machine learning experiment. An environmentespecifies adistributionPeon the spaceΩof complete records.In the setting of supervised deep learning, a complete record of an experiment would specify hy-perparameters, random seeds, optimizers, training (and held out) data, etc.

We measure a simple empirical statistic, thegradient noise scale3(essentially a measure of the signal-to-noise ratio of gradient across training examples),and show that it can approximately predict the largest efficient batch size for a wide range of tasks

non-zero entropy

overfitting

we stop training early when the test loss ceases to improve and optimize all models in the same way

Nincreases and the model begins to overfit

S

We find that generalization depends almost exclusively on thein-distribution validation loss, and does not depend on the duration of training or proximity to convergence

Although these models have been trained on the WebText2 dataset, their test loss on a variety of other datasetsis also a power-law inNwith nearly identical power, as shown in Figure 8.

(approximately twice the compute as the forwards pass)

To utilize both training time and compute as effectively as possible, it is best to train with a batchsizeB≈Bcrit

standard deviations

(x;W1,...,Wl,b1,...,bl)

Naturally, such an increase in the learning rate also increases the mean stepsE[∆w]. However,we found that this effect is negligible sinceE[∆w]is typically orders of magnitude lower than thestandard deviation.

−

〈O( ̄θ)〉=〈[[O[ ̄θ−η ̄∇LB(θ)]]]m.b.〉.

We omit thedβexp (−cγ) +bβlog(1δ)nterm since it does not change with changein random labels.

while ̃Hθ†l,φ[j,j] can change based onα-scaling Dinh et al. [2017], the effective curvature is scale invariant

(f) stays valid for the test error rate in (a)

Then, based on the ‘fast rate’ PAC-Bayes bound as before, we have the following result

Further, all the diagonal elementsdecrease as more samples are used for training.

Theorem 1

The bound provides a concrete realization of the notionof ‘flatness’ in deep nets [Smith and Le, 2018, Hochreiter and Schmidhuber, 1997, Keskar et al., 2017] andillustrates a trade-off between curvature and distance from initialization.

In spite of the dependency on the Hessian diagonal elements, which canbe changed based on re-parameterization without changing the function [Smith and Le, 2018, Dinh et al.,2017], the bound itself is scale invariant since KL-divergence is invariant to such re-parameterizations Klee-man [2011], Li et al. [2019].

∈Ck

because of the softmax operation.

the signs of f and 𝑓̃ f~\tilde{f} are the same.

f~\tilde{f} as 𝑓𝑉=𝜌𝑓̃ fV=ρf~f_V=\rho \tilde{f},

Our main results should also hold for SGD.

normalized weights Vk as the variables of interest

This mechanism underlies regularization in deep networks for exponential losses

Bahdanau et al.(2019) learn a reward function jointly with the action policybut does so using an external expert dataset whereas ouragent uses trajectories collected through its own exploration

other agents

Zero Sum

specific choice ofλ

z(xi)z(xj)|h

Z(θ)

Ziebart et al. (2008)

eθ>F(X,Y)

without interactionwith the expert

Attention: Mask

nlayerdmodel3dattn

Large models are more sample-efficient than small models, reaching the same level ofperformance with fewer optimization steps (Figure 2) and using fewer data points (Figure 4)

Theperformance penalty depends predictably on the ratioN0.74/D

Some preliminary numerical simulations show that thisapproach does predict high robustness and log scaling.However, it only makes any sense if transitions from onephenotype to another phenotype are memoryless.

LetPbe a row vectorspecifying the probability distribution over phenotypes. Wewant to find a stochastic transition matrixM(rows sum toone) such that

Mhas 1s on the diagonals,and 0s elsewhere, for example

Fano’s inequality)

Intrinsic motivations f

High-quality trajectories are trajectories where the agent collectsdescriptions from the social partner for goals that are rarely reached.

f memory-based sample efficient methods

We find that the object geometry makes a significantdifferences in how hard the problem is

When it comes to NNs, the regulariza-tion mechanism is also well appreciated in the literature,since they traditionally suffer from overparameterization,resulting in overfitting.

Therefore, we can get the following generalization bound:

They use on-average stability that does not imply generalization bounds with highprobability

forTďmstep

validation error which is used asan empirical estimate forRpw1q

our bound corroborates the intuition that whenever we start at a good location of the objectivefunction, the algorithm is more stable and thus generalizes better.

Rpw1q ́R‹

Whileit is known having a finite VC-dimension (Vapnik and Chervonenkis, 1991) or equivalentlybeing CVEEEloostable (Mukherjee et al., 2006) is necessary and sufficient for the EmpiricalRisk Minimization (ERM) to generalize,

position attribute.

The bounds based on`2-path normand spectral norm can be derived directly from the those based on`1-path norm and`2norm respectively

ExperimentsIn

min(Td;2S)

29

K(x)is an even

of a Teacher Gaussian process with covarianceKTand assume that they lie in theRKHS of the Student kernelKS, namely

If both kernels are Laplace kernels thenT=S=d+ 1andEMSEn1=d, whichscales very slowly with the dataset size in large dimensions. If the Teacher is a Gaussian kernel(T=1) and the Student is a Laplace kernel then= 2(1 + 1=d), leading to!2asd!1



We perform kernel classification via the algorithmsoft-margin SVM.

man

Importantly (i) Eq. (1) leads to a prediction for(d)that accurately matches our numerical study forrandom training data points, leading to the conjecture that Eq. (1) holds in that case as well.

s a result, various works on kernel regressionmake the much stronger assumption that the training points are sampled from a target function thatbelongs to thereproducing kernel Hilbert space(RKHS) of the kernel (see for example [Smola et al.,1998]). With this assumptiondoes not depend ond(for instance in [Rudi and Rosasco, 2017]= 1=2is guaranteed). Yet, RKHS is a very strong assumption which requires the smoothness ofthe target function to increase withd[Bach, 2017] (see more on this point below), which may not berealistic in large dimensions.

irreducible error (e.g.,Bayes error)

GMM on a dataset of previously sampled parametersconcatenated to their respective ALP measure.

nevertheless, the few re-maining ones must still differ in a finite fraction of bits fromeach other and from the teacher so that perfect generaliza-tion is still impossible. For aslightly above aconly the cou-plings of the teacher survive.

For a committeeof two students it can be shown that when the number ofexamples is large, the information gain does not decreasebut reaches a positive constant. This results in a much fasterdecrease of the generalization error. Instead of being in-versely proportional to the number of examples, the de-crease is now exponentially fast

n general, the baseline leaves the expected value of the update unchanged,but it can have a large

computevar1bbÂj

This suggests that the effect ofj(x)is to rotate the gradient field and move thecritical points, also seen in Fig. 4b.

sampling with replacement has better regularization

conservative

This implies that SGD implicitlyperforms variational inference with a uniform prior, albeit of a different loss than the one used tocompute back-propagation gradients

The second particularity is that since the computation of the rewardRpp;c;;oqis internal to themachine, it can be computed any time after the experimentpc;;oqand for any problempPP,not only the particular problem that the agent was trying to solve. Consequently, if the machineexperiments a policyin contextcand observeso(e.g. trying to solve problemp1), and storesthe resultspc;;oqof this experiment in its memory, then when later on it self-generates problemsp2;p3;:::;piit can compute on the fly (and without new actual actions in the environment) theassociated rewardsRp2pc;;oq;Rp3pc;;oq;:::;Rpipc;;oqand use this information to improveover these goalsp2;p3;:::;pi.

Although methods to learndisentangled representation of the world exist [25,26,27], they do not allow to distinguish featuresthat are controllable by the learner from features describing external phenomena that are outsidethe control of the agent.

We find that the full NTK has better approximation propertiescompared to other function classes typically defined for ReLU activations [5, 13, 15], which arise for instancewhen only training the weights in the last layer, or when considering Gaussian process limits of ReLUnetworks (e.g., [20, 24, 32]).

and we have left the activation kernel unchanged,K`=1M`A0`A0T`

(A`jJ`)

Second, we modified theinputs by zeroing-out all but the first input unit (Fig. 1 right).

for MAP inference, the learned representationstransition from the input to the output kernel, irrespective of the network width.

he representations in learned neural networks slowly transitionfrom being similar to the input kernel (i.e. the inner product of the inputs) to being similar to theoutput kernel (i.e. the inner product of one-hot vectors representing targets).

the covariance in the top-layer kernel induced by randomnessin the lower-layer weights.

e.g.compare performance in Garriga-Alonso et al. (2019) and Novak et al. (2019) against He et al.(2016) and Chen et al. (2018)).

enabling efficient and exact reasoning aboutuncertainty

significant new benchmark for performance of a pure kernel-based method on CIFAR-10, being 10% higher than the methods reported in [Novak et al., 2019]

Optimally, these parameters are chosen such that the true predictiveprocessP(t§jx§;S) is closest toQ(t§jx§;S) in relative entropy.

Bayes classiØer

our task is then to separate the structure from thenoise.

We know of no interesting real-world learningproblem which comes without any sort of prior knowledg

(theluckycase)

, such as cross entropy loss, encourage a larger outputmargin

the gap between predictions on the true label and andnext most confident label.

This is further consistent with recent experimental work showing that neuralnetworks are often robust to re-initialization but not re-randomization of layers (Zhang et al. [42]).

Kernels from single hidden layer randomly initializedReLUnetwork convergence to analytic kernel using Monte Carlo sampling (Msamples). See §I foradditional discussion

We observe that the empirical kernel^gives more accurate dynamics for finite width networks.