Notice how flat the second graph is along the 2x_1+2x_2 = 1 direction. (0 derivative throughout)

It seems flatness along some directions of the loss as a function of some representation indicates how good (dis-entangled) the representations themselves are. The next layer that operates on these representations can then discard this flat direction, and focus on the other one. Hence, the more directions the curve is flatter in (== the flatter the curve is), the more efficiently can the next layer learn.

Essentially, smoothness limits how many times can the network "turn" in a given small subset of the input. This limit then makes it so tha all the data in that subset must be fit by the function without wiggling around.

One can imaging how this can be used to create an equivalent problem by collapsing a lot of points into a single point. And lots of wiggly functions to a smooth variant.

If we go from ID data to OOD data via a covariate shift, this is true.

But not if a concept shift has happened when going from ID to OOD data.

Are we sure?

Brownian Dynamics

But there can be other fixed points also! In particular, \(\theta^*\) is a global minima. We may get stuck in local minima, or oscillate around them due to the changing nature of \(P_t(X_i; \theta)\).

This pattern is observed in other places in Deep Learning also. Wherein, introducing new/meta-variables and varying quantities often doesn't hurt convergence.

There seems to be something about the architecture/initialisation/optimisation in DL that seems to enable it.

That is, there exists a parameter setting \(\theta^*\) such that it can minimize the loss on each and every input. Meaning that it can predict the correct probabilities over \(Y\) for every \(x \in X\).

The \(\sigma\)-algebra of the probability space is closed under complements and countable intersections.

Also using countable additivity of measure space with the above fact, leads to the result.

As each event = \(\bar{\phi}\).

In particular, using the relation between the structure of the problem and how it relates to the hypothesis class we have chosen. The better the two structures align, the better the best performing model will be and it will be easier to find that model. Hence it is clear that more information helps in obtaining better bounds, but not clear why it will be difficult to calculate the \(inf_{g \in \mathcal{H}} L(g)\) and hence convert all the bounds obtained on excess risk to those on absolute risk.

It will be difficult for two reasons probably:

1. You will need to know the actual distribution \(p\), just samples from it, won't do. But maybe we can have a limiting case like bounds, where as \(#\)samples \(\rightarrow N\) the actual distribution \(p\) becomes known and the bound becomes valid.

2. Finding the optimal loss or \(g\) may be a computationally intractable problem.

Though, in some function classes the optimal loss may just be \(0\) ?

Very clean.

Very clean idea.

The prediction of \(f_{lin}\) at \(x\) is equal to the original value of \(f\) at \(x\) at parameter \(\theta_0\) plus the component of the task vector along the direction of gradient at \(x\). That is we add a factor corresponding to how well the gradient on the sample \(x\) aligns with the "average" gradient(task vector) direction.

Another useful way to make sense would be to,replace \(\tau\) with \(\Delta \tau\). And then see what the equation looks like. It is a first order approximation of \(f\) around \(\theta_0\).

For a particular input \(x\); if I add any linear combination of task vectors, only the component along the task vector corresponding to the task to which \(x\) belongs, determines the value of \(f\) at that point. That is, the other task vectors lie in the tangent space of \(f\) seen as function of parameters, at the point \(x\).

This means that \(g_t\) varies the same way as \(f\) varies when we take the parameters \(\alpha_t\mathbf{\tau_t}\) away from the original \(\theta_0\), and \(g_t\)'s value doesn't get effected when we move along any other task vector.

So, if I move to \(A=\theta_0 + \alpha_1\tau_1\) first, then all the variation in \(f\), due to that movement is captured by \(g_1\) and all other \(g_t\) stay the same.

Then if we move to \(\theta_0 + \alpha_1\tau_1 + \alpha_2\tau_2\) from \(A\), all the variation will be captured by \(g_2\) and the rest of \(g_t\) will stay same.

If moving along \(\tau_t\) only changes the value of \(f\) on \(\mathcal{D}_t\) and not on any other task's dataset, then the functions:

\(g_t(x;\alpha_t\tau_t) := f(x; \theta_0+\alpha_t\tau_t)\mathbb{1}[x \in \mathcal{D}_t]\)

will satisfy the condition required by equation \((4.)\).

Spatially localized in the input space. That is, functions that are 0 outside of \(\mathcal{D}_t\).

Do the results hold if we allow multiple mixing coefficients? One per task?

Moreover, the output is assumed to be linear in the parameters. And addition can be adding scaled versions of \(\tau_t\), too.

What makes a task transferable?

Conjecture 1: The diversity of input for a task is what matters about how transferable the task will be.

Three experiments for comparing transferability:

1. distributions of cosine similarities between samples => task diversity (?=>) transferability.

2. Transf.(Model trained on QQP) on one hand and Transf(model trained on MNLI with labels generated by QQP fine-tuned model)

3. Slowly shuffle larger proportion of labels. Does the task still stay transferable?

It is not necessary that the dashed line is bound on performance.

Here the equality is in the probabilistic sense.

A probabilistic notion of equality of two events \(E, F\) is to say that \(E\) and \(F\) are equal when \(E\subset F\) and \(F\subset E\).

Another one can be: Two events \(E, F\) are said to be equal if \(P(E\backslash F)=0\) and \(P(F\backslash E)=0\).

Both the definitions of equality are preserved under extensions as extensions are probability preserving. And both form equivalence relations on the underlying \(\sigma\)-algebra. The former is a refinement of the latter.

This is a sequence that decays to 0, as n increases. As we bring A closer to 0, this sequence decays faster and faster, with increasing \(n\).

The statement that "event \(E\) holds with overwhelming probability" means that the probability of this event is an upper bound on the faster and faster decaying sequences.

In \((iv.)\), instead of having sequences that decay faster and faster, we want that \(\mathbf{P}(E)\) is an upper bound on just one sequence that decays as a polynomial in \(n\).

In \((v.)\) this is further relaxed to \(\mathbf{P}(E)\) being an upper bound on some sequence that decays to 0.

Note that in the above, upper bounds are on the 1-decaying sequence.

Also, as \(E=E_n\), \(\mathbf{P}(E)\) is also a sequence of \(n\). So, in fact, we have that one sequence, is forming an upper bound on another.

In \((iii.)\), the sequence of proababilities is upper bound to any sequence that tends to 1 as an exponent of \(n\).

In \((iv.)\), the sequence of proababilities is upper bound to some sequence that tends to 1 as an exponent in \(n\).

In \((v.)\), the sequence of proababilities is upper bound to some sequence that tends to 1.

Also, the constant is omitted for brevity, but choosing an apt constant is also to be considered in above statements.

So that we don't think about set-theoretic constructs like cardinality which may not be a probabilistic property. One should be aware that the unions/intersections is not between sets, but between events.

Same permutation for both \(x, z\).

Generalization of zeroing out the features instead.

The RHS is the maximum difference between the empirical loss(averaged over all the tasks), and the actual loss(averaged over all the tasks) for any shared representation \(\mathbf{h}\) and any task specific mappings \(\mathbf{f}\).

The inequality holds as both \(b\), and \(d\) resolve to the difference of empirical and average losses, for the functions \((\hat{\mathbf{f}}, \hat{\mathbf{h}})\) and \((\mathbf{f}^, \mathbf{h}^)\) respectively.

As \(\hat{\mathbf{h}}\) and \(\hat{\mathbf{f}}\) are the minimizers for the data over using which \(\hat{L}\) is computed.

On the other hand \(\mathbf{h}^\) and \(\mathbf{f}^\) are the functions that minimize the actual centered training risk(\(L\)), and hence the empirical centered training risk(\(\hat{L}\)) may be larger when we use these values.

This is 0.

Distance between the actual and learned representations, under the function class \(\mathcal{F}\), and given the optimal fit over the real shared representation \(\mathbf{h}^*\).

1. \(\hat{\mathbf{h}}\) is the learned shared representation.

2. \(\tilde{\mathbf{f}}\) is the function that actually optimizes the performance for each task \(j \in [t]\) using the learned shared representation \(\hat{\mathbf{h}}\).

3. \(\hat{\mathbf{f}}\) is the learned approximation to \(\tilde{\mathbf{f}}\).

A division by \(n\) seems to be missing..

The training risk over and above the one attained by the actual underlying feature representation \(\mathbf{h}^\) and the optimal \(f_j^\) corresponding to it.

The actual underlying shared representation of the tasks.

How well can any function from the function class \(\mathcal{Q}\) match a set of completely random samples from \({+1, -1}\).

Rademacher distribution

https://crypto.stackexchange.com/questions/48638/whats-the-difference-between-polylogarithmic-and-logarithmic

1. If \(\mathbf{h}'=\mathbf{h}\) you can directly put \(f'=f_j\) and the distance will go to 0.

2. If \(\mathbf{h}'\ne\mathbf{h}\), but you can find some \(g \in \mathcal{F}\) such that \(g \circ \mathbf{h}'= \mathbf{h}\), you can still make the distance 0.

3. This metric can also be negative, given what \(\mathbf{f}\) and \(\mathbf{h}\) are.

\(\mathbf{x}\) influences the distribution over \(y\) only via the task specific representation \(f_j^ \circ \mathbf{h}^\)

This is a Lipschitz constant dependent on the function class \(\mathcal{F}\).

what happens to this definition if \(N \to \infty\) ?

The assumption here is that the value of \(\mathbf{q}\) at any point \(\mathbf{x}_i\) is constrained by its value on the other points. Because otherwise, if \(\mathbf{q}(\mathbf{x}_i)\) can go to \(\infty\) without any other value going to \(-\infty\); then complexity is \(\infty\).

The assumption that the tasks share a common design.

Low-dimensional being the keyword here.

Why is this valid?

Why is a deep network approximated with a linear projection?

The point being made is that the head performs poorly, and the feature extractor may still be giving appropriate features for the head to perform well.

Can be verified by probing the representations of the finetuned model.

Also, we can't guarantee sequential reads/writes in a B-Tree like structure, no matter how we store it in the memory.

A queue per consumer. To efficiently distribute all messages among various consumers. The per-consumer queue is maintained by each consumer, probably.

Kafka broker manages consumer groups, maintaining the state of each consumer, and distributes load amongst them.

On the other hand, if we make an in-memory cache ourselves, that cache will take half the memory, and another half will be taken by the pagecache of the OS, when it writes to the disk.

Though, we can probably free off our cache as we send the writes to OS. But as we don't control when the OS will flush those out to disk, we are only guaranteed to have access of half of the available memory for our (in-memory)cache.

That is, directly write to filesystem, OS will buffer your writes.

The disk is fast when writing/reading lots of data sequentially. But not so fast when writing/reading data in random fashion.

This is the divergence.

OS maintains a cache of writes, and only when enough are accumulated, are they sent to the disk. This allows for efficient writes.

There is one entry in OFT per open() call. Each call to open(), in any process will create a new entry in the OFT.

Each process maintains its own set of file descriptors in its PCB(process control block). These file descriptors, are just numbered pointers to entries in the OFT.

The fork() call creates a child process, and copies over the contents of its file descriptor list, to the child; without executing open()-open() in child process again and again. Instead it increments the refcnt of the entry in the OFT.

dup() call in a process, creates a new entry in that process' file descriptors list, and updates the refcnt. It returns the position of the new entry in the file descriptors list.

dup2() does the same as dup() but puts the new entry at a specified position. So that any other function accessing the entry at that position will now be redirected to the file referred to by the new entry.

See here too.

Returns a copied file descriptor that points to the same entry in the OFT. The new file descriptor has a new number.

Releases lock atomically.

Called by OS, when the application program requests a read/write.

Called by ISR, (ide_intr()).

Called by the process which wants to read/write.

That is, locations of RAM.

These are registers of the external device.

https://en.wikipedia.org/wiki/Database_engine

https://redis.com/blog/redis-enterprise-delivers-linear-scale-proven-time/

https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

https://www.ibm.com/topics/esb

No discounting or similar thing in their definition

that is, concurrent learning of reward predictor and the agent.

Expert users know how to break the decision problem correctly into possible answers and can possibly provide very fine-grained feedback, as a result.

We want to succeed in the absence of such detailed feedback also.

Like NP problems. Solution can be verified in polynomial time, but not generated.

Better than having a separate label for null?

using regexes

the amount of "gross" written and "net" written in the invoice.

D’Andecy et al. (2018) build upon this approach by incorporating an a-priori model of term-positions to their iterative layout- specific extraction model, significantly boosting performance on difficult fields.

This can be adjusted when only one document is available. For itf-df we need more than one documents though.

That is, suppose we are extracting a word that is at 1.5cm far along a line at -45 degrees from a node whose transcript is "Total".

Then we don't only consider a "Total" appearing at appearing in a particular location(say at the end of document), but all "Total"s occuring thoughout the document and cast a vote for a location that is 1.5cm along the -45 degree line from each of those locations.

In effect, the model has following assumptions:

1. A field to extract has particular position relative to all other words in the invoice/document. All these are constraints are for the field. (Alternatively, the constraints are the edges of the star graph.)

2. The position that satisfies most of these constraints is the one that gets extracted.

Additionally, the model is made better and more in-line with reality when the three kinds of weightings in section VI are done incrementally.

\(j\) iterates over all the nodes having same transcription as the \(i^{th}\) word, \(w_i\).

The document processing(structured information extraction) problem can then be broken into the two tasks:

1. Template identification(identigying templates from a single document; or grouping documents with same templates together).

2. OCR.

All (uni/bi/tri/quad)-grams from any value of a field, will have that field as their label in the final results dict.

This training data generation, generates {nGram, field} pairs, which can be used to improve and adapt the classifier. But it is less clear how to use this to improve OCR, or the character level language model in case we decide to use one.[Think more over this later..]

The agents are the various classes. The tasks are all the words with the label as that class.

We can use this data to further finetune our models. We would need extra step to find which model exactly went wrong. The architecture of our pipeline should be such that this step can be performed easily.

[Aside: Use large model like T5 and condition on natural language description of what exactly went wrong?]

There is a consistency issue with previous step:

1. The N-grams that are classified as belonging to a particular field, should all compose a single contiguous unit(like a word or a date or entire amount) that can be the value of that field. That is, the Classifier should output "amount" for all characters in "100.0o" for the post-processor to convert it to "100.00". This means that the classifier should have language modelling like abilities.

2. The post processor can be thought of as a character level language model, that is used to correct spellings/common mistakes of OCR*. In case of poor images, it may be necessary to guess the characters based on the output of the classifier. For example, when from an invoice, half of a zero has evaporated so it looks like "100.0C". In this case it will be wrong to expect OCR to predict 0 at the last position. The character level language model may also let it go undetected(considering C for Celsius, which occurs fairly commonly), unless the language model is conditioned on the predictions of the previous step, i.e., it knows that it is producing an amount not temperature.

An architecture that will allow us to handle these dependencies between the two tasks effeciently, can be constructed by considering few layers common for the two tasks of language modelling spell correction and classifying the n-grams.

Other approaches can be to consider ensuring proper images. Or perform some image enhancement/edge detection. Or to do data augmentation to involve such poor samples in training too. Or finetune a pre-trained vision model.

It can also be the case that there are examples with invoices where C stands for cents. In this case, the sample will be truly ambiguous. [Probably we can do a joint model-ling of replacements and classifier labels, with a crf that is aware of this?]

[*Probably we could have had a character level language model correct characters output by OCR(step 1), earlier? As this ould be character by character replacement we can preserve the features associated with various character spans. RegEx make sense in case we already know what we are searching for, and it has fixed format, like date and amounts.]

Can also be a CRF. CRF can be a linear chain CRF( only one dimension) or be a CRF in 2 directions too.

In case, all classes are not known a-priori, we can train a model that learns to extract {key: value} pairs instead.(Using just B-K, I-K, O, B-V, I-V tags, for example.)That is ,cast the problem as a sequence tagging one.

We would need new model to include different kinds of annotations(for example, list item data), though.

This step is necessary to find units with which features in the next step can be associated.

Converting to actual words, at this point will result in loss of correspondence between characters identified by OCR(to which features like size etc. can be associated) and the actual words.

The OCR for a character is independent of the structure/context in which the character occurs. Hence it makes sense to include this step as a separate.

Pre-specified by user? Or will be different based on the extraction from various templates?

What about list-item type data, rather than key-value data?

1. We don't need to write this __init__. If we already construct layers, activation and bias outside the class, we can pass them through the automatically constructed __init__(). Note that __init__() is also present, and the one we wrote will only be called if pass a single argument while initializing(the key).

2. We can also add in_dim, out_dim, hidden_dim, activation_fn as arguments in this __init__, rather than having fixed values, if we want.

3. All the fields specified in the annotation list are considered as the children of the PyTree defined by the class.

This is a problem because:

As we won't be writing our own tree_flatten and tree_unflatten(will be inherited from eqx.Module), all the attributes of self that are set in __init__, will be passed through these flatten and unflatten.

So, if we want to write a class like in Appendix A, that allows activation function as a class attribute(we could also have written an __init__() function that takes in the activation function, alongside key), we will ed up having a leaf in the PyTree, that is not a jnp.array but an activation function.

With the first argument(self) being a PyTree instance, having all the parameters(as jax.grad differentiates w.r.t. first argument, it is ideal that first argument contains all the parameters!).

As gradient of PyTree basically corresponds to calculating gradient w.r.t. all leaf nodes, and storing them in the same PyTree structure.

An expression is called referentially transparent if it can be replaced with its corresponding value (and vice-versa) without changing the program's behavior.

For example when ensuring that treedef of two arguments of a function are same, like in vmap example.

As the auxiliary data gets added to the treedef, it becomes essential that the auxiliary data also matches for vmap to be applied properly. So, we need to keep this mind while defining/registering new pytree node.

Gradient of loss with respect to inputs(rather than NN parameters) contains quite meaningful information that can be easily used to learn better NN parameters, via backprop.

Adjoint is also denoted as \(T^*\). What about the conjugation in the adjoint, though?

Nice result. variance of estimator, bounded below by 1/[information content]!! This is the meaning of "maximum information" in a sample.

scales proportional to \(n^{-1}\).

measures how sensitive expected value of estimator is to small changes in the underlying real distribution's real parameter \(\theta\).

Expected value of the estimator \(\hat{\theta}\) over all possible samples drawn from the "real" distribution\(f(x|\theta)\) as a function of the "real" parameter \(\theta\).

Here we suppose "ahh.. if \(\theta\) was our real parameter..", and then ask .. "what would the expected value of our estimator be with this \(\theta\)?"

Fisher information tells what is the maximum information we can get about parameter \(\theta\) from a sample \(X_1, ..., X_n\).

As this is the maximum information, our inference must have a variance related to this maximum information.

Consider the set of all \(x\) for which \(P(x)\) is false. Consider a minimal element of this set, \(y\). \(P(x)\) is true \(\forall x<y \).

Now, \(y=succ(z)\) for some \(z \in X\) xor \(y=sup([min(X), y))\). In both cases, we can apply one of the suppositions, 2 and 3 respectively, to conclude that \(P(y)\) is true, which is a contradiction.

Hence \(P(x)\) is true \(\forall x\in X\).

It is assumed \(min(X) \in [min(X), min(X))\).

not needed, as totally ordered is implied by existence of minimal elements for subsets consisting of only \(2\) elements of \(X\).

The proof doesn't require \(\mathcal{F}\) to be the collection of all subsets of \(X\). Hence, we can, for one, consider a \(\mathcal{F}\) to be a collection of mutually disjoint subsets of \(X\).

This is essential as \(f(\phi) \notin \phi\), no matter how you choose \(f\).

shape\(=n_L\)

shape\(=n_L\times n_L\)

\(C\) is defined to have a (functional) derivative at \(f_0\) if the map from \(\mathcal{F}\) to \(\mathbb{R}\) defined as:

\(\rho \longrightarrow \lim_{x \rightarrow 0} \frac{C[f_0+x\rho]-C[f_0]}{x}\)

is an element of \(\mathcal{F^*}\).

maps linear forms [functions from \(\mathcal{F}\) to \(\mathbb{R}\)] to functions on the data \(\in \mathcal{F}\).

or in the support of \(p^{in}\).

Note that \(||f||K=||f||{p^{in}}=0\) if \(f\) is \(0\) at all data points.

Or in otherwords, \(||f-g||K=||f-g||{p^{in}}=0\) if \(f\) and \(g\) are same on all the data points.

If \(K(x,x')\) is not positive definite for some \((x,x')\) pair,

\(\implies \exists v \in \mathbb{R}^{n_L} \backslash { \mathbf{0} } : v^TK(x,x')v \le 0\).

Setting \(f(x)=v \forall x \in \mathbb{R}^{n_0}\), we have \(||f||_K = \sqrt{\langle f, f \rangle_K}\) is not defined or 0.

whereas \( ||f||_{p^{in}} \) is still well defined and \(>0\) as \(v\ne\mathbf{0}\).

This means that if \(K(x,x')\) is not positive definite for some \((x,x')\), then the kernel \(K\) is not positive definite as per the definition here.

Moreover, if \(K(x,x')\) is positive definite for every \((x,x')\), then

\(\langle f,f \rangle > 0 \forall f \) that are non-zero in some region(\(\subseteq \mathbb{R}^{n_0}\)) of non-zero probability,

which means that \( ||f||{p^{in}} >0 \implies ||f||_K > 0 \forall p^{in}\). This means that K is positive definite w.r.t. \(||\dot||{p^{in}}\).

This means the current definition is equivalent to saying that K is positive definite iff \(K(x,x')\) is a positive definite \(n_L \times n_L\) matrix, for all \((x,x') \sim p^{in}\).

The second derivative of output of ANN with respect input of ANN may not be bounded?

The expected value of seminorm, then resolves to the empirical average over the entire dataset of \(N\) samples,

Seminorm satisfies: 1. Triangle inequality 2. absolute homogeneity, i.e., \(||s.f||=|s|.||f||\) for all scalars \(s\) and all functions \(f \in \mathcal{F}\). 3. 1 &2 => non-negativity, i.e., \(||f|| \ge 0 \forall f \in \mathcal{F}\).

For a seminorm to be a norm, it is additionally required that: 4. \(||f||=0 \implies f=0\).

(https://en.wikipedia.org/wiki/Seminorm#Definition)

The triangle inequality here follows from the fact that: 1. \(\langle v, u \rangle \le ||v||||u|| \forall v,u \in \mathbb{R}^{n_L}\) 2. and linearity of integration

Does it? [CONFIRM]

Equation 4?

First term is positive semidefinite as \(vv^T\) is and sum of PSD matrices is PSD.

Figure 5?

That is sharper minima.

This kind of analysis(comparing magnitude of the eigenvalues) tells us that sharper minima found by large batch gradient descent are indeed sharper in terms of the curvature or the eigenvalues of the Hessian.

That is, it is not the case that sharper minima just span lesser number of dimensions.

The number of dimensions spanned by the minima can be guessed from the number of positive eigenvalues.

In this case \(>M-N\) eigenvalues are expected to be trivial, i.e., \(0\).

when is this true?

Second derivative is 0 along most directions, at the final trained point in parameter space!

low loss regions of parameter space.

problem for the non-degenerate assumption.

If the Hessian is degenerate, then it has at least one 0 eigenvalue. That is, there is some subspace of directions along which the derivative stays 0. When the number of parameters, \(M >>\), the number of samples, \(N\); it is possible that all of the \(N\) samples allow gradients in the Kernel of Hessian only. In which case, we will converge to wrong solution..

*full batch

The green peaks occur at small plateaus just before big slingshots of weight norm. The sharpness increases first, indicating a big change in last layer norm is about to come.

why cosine distance? What happens with euclidean distance?

Is this sentence correct?

why does training loss retain low value when norm grows rapidly?

Probably because at first, the norm increase is just trying to increase the difference between logit of correct and other labels. But, as we have momentum in our optimizers, maybe it increases the norm too much. This leads to incompatibility between features available to last layer, and the classifying function implemented by the last layer. The features and classifier need to be made compatible again. This causes feature updates and wild loss function fluctuations to happen after norm growth?

so that the difference in the logit of the correct label and other labels increases.

continuously differentiable with L-lipschitz continuous gradient

missing square

Here, search engine can be thought of as searching whether the asked questions is answered in the available documents.

First stage corresponds to finding questions related to the actual questions, that are answered in the available documents and the answers to which, will help answer the actual final question.

The first step(and the last), although it looks simple at first glance, will be quite complex to perform as the questions become more and more complex.

For example, say, we want to answer a question like: "What would be the consequences of the Riemann hypothesis being false?", given only documents that explain Riemann Hypothesis and results derived thereof.

Then, we would need to reformulate the question as: what results follow from the Riemann Hypothesis? and then we can find answers for that from our search engine, which ,in turn, can be negated in the synthesis stage to answer the counterfactual question.

i.e., \(\frac{||W_{OV}^{h_1}W_{OV}^{h_2}||F}{||W{OV}^{h_1}||F||W{OV}^{h_2}||_F}\) was small for all pairs \((h_1, h_2)\).

for the present token, with the considered head on the logit..

Transformers may learn exact parsing of sentences.. rather than just using heuristics like word matching, bigrams etc.

It is so because, otherwise it will end up interfering with information transfer from other source tokens.

In case of higher layers, source token's embedding after first layer.

As \(A^h\) is in tensor product with \(W_UW_{OV}^hW_E\).

As the residual stream gets rotated, each of the matrices used to read and write to the residual stream will also be rotated in the opposite direction. The attention heads and other MLPs are kept, as is.

Rotating these bases to be as axes aligned as possible, could reveal new(or easily interpretable) insights.

What could be the analogue for similar relation(/information exchange) between more than 2 layers?

Equilibration in humans: http://penta.ufrgs.br/edu/telelab/3/equilibr.htm

The development or maintenance of an equilibrium.

related to abductive reasoning. Abductive reasoning is "making a probable conclusion from what you already know.

any of the new. The max in eqn 8 selects the constraint that is closest to being fully satisfied in the next few steps.

How does performance of adapters with PET look like?

Can be a test point outside of \(\mathcal{X}\).

Rather than using the expressions in (10) and (11), we can also obtain \(\theta_t\) from equation (8) and use that to compute output on arbitrary \(x\).

This is of dimension \(k \times k|\mathcal{D}|\).

\(f_t^{lin}(x)\) depends on both \(\theta_0\) and \(\theta_t\).

We take the gradient on both sides of equation (5), to get \(\nabla_{\theta_t} f_t^{lin}(\mathcal{X})=\) coefficient of \(\theta_t\) in (5), which is \(\nabla_\theta f_0(\mathcal{X}) |_{\theta=\theta_0}\).

The gradient on both sides of eqn (5) is taken w.r.t. the running parameters, \(\theta_t\), as the gradient in equation (2) is also w.r.t. the running parameters.

The second equality here is obtained by substituting for \(\dot{\theta_t}\) from equation (2).

standard parameterization samples directly from \(\mathcal{N}(0,\frac{\sigma_w}{\sqrt{n_l}})\). Thus, gradients are passed only upto \(W_{i,j}^l\). But the gradients in this paper, are passed back one step further, to \(w_{ij}^l\) and hence are scaled by a factor of \(\frac{\sigma_w}{\sqrt{n_l}}\), for all weights of the network.

The initialised plot of conditioning distribution, \(P(Y|X)\) is not aligned with the initialised contour plot, again.

The initialized values in the covariance matrix donot correspond to the contour plot on the left. Slightly perturbing the yellow eigenvectors will correct the covariance matrix.

Even though we have a lot of parameters, the complexity of function class \(\mathcal{F}\) is still low because the output of a the central function in \(\mathcal{F}\), [corresponding to the weights \(\matbb{W}^{(1)}\)] can only be perturbed in a linear way (specified in the definition of NTRF) for every input.

This simplicity of complexity of \(\mathcal{F}\), probably allows us to use VC-dimension like generalization bounds of \(\mathcal{\widetilde{O}}(\frac{1}{\sqrt{n}})\)

1. \(\mathbf{W}\) acts as a specification for perturbation to each and every weight of a neural network.

2. The perturbation in each sample's output is proportional to the sum over all weights of the network of: {perturbation of weight multiplied by the gradient of the output on that sample}.

This NTRF function class sort of tells about all the functions available in an \(\frac{R}{\sqrt{m}}\) neighborhood of \(\mathbf{W}^{(1)}\). This is true when \(\frac{R}{\sqrt{m}}\) is small enough that \(f(x;\mathbf{W}^{(1)})\) is approximately linear in the weights in \(\frac{R}{\sqrt{m}}\) neighborhood of \(\mathbf{W}^{(1)}\) for all inputs \(x\).

New thought-1.

\(c=1\) "epoch"

Starting from the \(\hat{w}\) that has been trained for \(B\) or \(0.75B\) time.

The cyclical and constant learning rate schedules are leading to much more aligned train-test losses than in Figure 1.

Because of the learning rate schedule and using SGD[not Adam].

Figure out when this is satisfied, exactly.

correspond to smaller batch size, larger learning rate.

A differential equation in which one or more terms is a stochastic process.

What are these?

Here, we are expecting the model to consolidate the following points:

1) A particular noun phrase that occurs at first argument position of a transitive verb, can also occur at the second.

2) love, which has been seen before as a transitive verb, must be a transitive verb this time too(not an abstract noun as in "Love is the greatest thing ever.")

3) "The boy", "the hedgehog" which occurred as NP before, must still be NP.

4) The second argument position of the transitive verb "love" was the NP that came later on in the sequence, so it must be the same this time too.

If the model considers semantic information too, the ability of an NP to occur at both argument positions of a transitive verb depends on what the NP is, as well as, what the transitive verb is. For e.g., you can't "eat" "water". You can only "drink" "water".

But here, we are assuming that the model tries to learn to generalize the grammatical information only.

translationally equivariant to shifts of size \(P\).

But we didn't train our network's prediction to be invariant to this \(\frac{1}{3}^{rd}\) factor!

This means, network will not detect/mis-detect objects of this smaller size, and lead to errors.

No re-scaling required after taking the RoC of this size, unlike training time, when we needed to re-scale RoC.

\(K_{test}^{image}\) is user specified.

\(K_{train} = K_{test} \le K_{test}^{image}\).

The difference between \(K_{test}\) and \(K_{test}^{image}\) allows to express bias towards important things being present at the centre of the image.

The final number of pixels an object occupies in a sampled RoC, only depends on the actual height of the object and its distance from the camera. It doesn't depend on the resolution of camera. This is to be expected as we are re-scaling to a fixed size(\(K_{train}\)).

The number of pixels occupied by the object in an original image, however would depend on the resolution of the image. In particular, \(r_1 = k\sqrt{HW}\frac{R}{Z}\), is dependent on both \(H\) and \(W\).

Meaning we are taking a square RoC out of a square image.

\(s\) is geometric mean of \(\frac{K_{train}}{H_{RoC}}\)(scaling factor in \(X\) direction) and \(\frac{K_{train}}{W_{RoC}}\)(scaling factor in \(Y\) direction).

Assuming that the entire \(r \times r\) object happens to be in the RoC. If a part of it is in RoC, then that part's pixels in both directions will be scaled by \(s\)

See here%20is,the%20size%20of%20the%20sensor.) for detailed explanation.

Not pixels, but can be converted to pixels.

If we took random RoC at testing time too, they would have had same distribution.

That is, initially, all information regarding the class is contained in the patch tokens, and the class token is independent of the patch tokens. As training progresses, the class token's final embedding becomes correlated with (hence dependent on) the patch tokens. It is, as if, information has travelled from the patch tokens to the class token.

*equivariant

Is this positive semi-definite, though? It must be for the square root to be valid. This positive semi-definiteness of Hessian seems to be an additional assumption here.

The minimum number of weights in any layer.

non-zero.

By multiplicity.

These are square roots of (Eigenvalues of \(A^TA\)). As A^TA is positive semi-definite always, the square roots are valid.

As we are at a minima, the second derivative in some direction must be positive. Otherwise, if all directions, have second derivative negative or zero, we are at a maxima or a completely flat region(in case all second derivatives on the diagonal are 0).

https://math.mit.edu/~stevenj/18.335/norm-equivalence.pdf

By the particular choice of \(\alpha\).

Intuitively, you can imagine each point, \(\theta'=(\theta'_1, \theta'_2)\) in the initial \(B_{\infty}(r, \theta)\) moving along the curve \((\alpha\theta'_1, \alpha^{-1}\theta'_2)\).

The \(\mathcal{L}_{\infty}\) ball is the ball constructed using the \(L^{\infty}\) norm on the parameter space. It is just a cube, or side length \(2r\).

Add 1 to prevent sharpness from exploding when \(f(x)\) is close to 0.

Provides mapping between \(n\)-dimensional model-parameter space and a \(p\)-dimensional manifold.

If this term was always 0, \(C_{\epsilon}\) would have been a perfect cube, with side length \(2\epsilon\).

This terms stretches the \(i^{th}\) edge on both sides by \(\epsilon|(A^{+}x)_i|\). As there is \(|.|\), no side will be shorter than \(2\epsilon\).

The equation of linear kernel is:

\(K(x,y) = x^{T}y+c\)

where \(c\) is some constant.

Simple scaling corresponds to transforming the space by multiplication with a diagonal matrix. Isotropic scaling corresponds to multiplication by a diagonal matrix, all of whose diagonal entries are same.

A linear transformation that preserves dot products between elements of the vector space. In 2-D and 3-D it consists of rotations and reflects or any combinations thereof.

The entries of the matrix corresponding to this operator are not like those in the covariance matrix. They are of the form \( [p(e_i, e'_j) - p(e_i)p(e'_j)]_{ij}\) where :

1. \(\{e_i\}\) and \(\{e'_j\}\) are basis vectors of \(\mathcal{X}\) and \(\mathcal{Y}\) respectively.
2. \(p(e_i, e'_j)\) is the probability of sampling the pair of elements corresponding to \(e_i, e'_j\) from \(\mathcal{X} \times \mathcal{Y}\).

Only symbolic for the operator. The expectation will be calculated actually, when a \(g\) is sent inside it, and the tensor product operator gets applied on \(g\).

This too is a linear operator.

In the finite case, it is similar to the tensor product of two vectors, which yields a matrix.

single element per Hilbert space. (most probably)

These must be true for all \(f\) and \(g\) respectively.

A countable complete orthonormal basis \(\{e_i\}_{i=0}^{\infty}\)for a Hilbert space has the following properties:

1. \(\langle e_i, e_j \rangle\) = 0 for \(i \neq j\).
2. \(||e_i|| = 1\)
3. The linear vector space spanned by the countable many basis vectors, must be dense in the Hilbert space.

The last point means that not every element of the Hilbert space may be written as a linear combination of the basis vectors, but, it can be approximated as closely as we want(in terms of the norm/metric derived from the inner product of the Hilbert Space) by some linear combination of the basis vectors.

In effect, while calculating the inner product in the Hilbert Space of linear operators, we are multiplying together corresponding elements of \(C(:= [c_{ij}])\) and \(D(:=[d_{ij}])\), and summing them together.

$$ \langle C,D \rangle_{HS} := \sum_{i,j} c_{ij}d_{ij}$$

where \(i, j\) belong to a countable index.

Because both \(\mathcal{G}\) and \(\mathcal{F}\) admit countable orthonormal bases, we just have summing over two countable sets while calculating this norm.

The linear operator \(C\) can be imagined as a matrix with a countable number of rows and columns. (Possibly infinite)

In topology, a topological space is called separable if it has a countable, dense subset.

A Hilbert space is separable, if it has a dense countable subset too. Together with Zorn's Lemma, this means that the Hilbert space admits a countable orthonormal basis.

Letting \(C(\mathcal{X})\) denote the space of continuous bounded functions on compact domain \(\mathcal{X}\), we call a kernel \(k\) ''universal'' if \(k(x,\cdot)\) is continuous for all \(x\) and the RKHS induced by \(k\) is dense in \(C(\mathcal{X})\).