this is a note

Path parameters:

• Identify a resource uniquely

Request body:

• Send and receive data via the REST API, often specifically to upload data.
• Needs to adhere to REST API principles to be RESTful. Therefore for POST/PUT, needs to send the whole resource in the body.

Query:

• Mainly used for filtering. If you send a query asking for a large number of resources, use query parameters to filter the set of resources that you want. Example: You send a request to api/pictures/category/cat. You will now get all pictures with cats which could be millions. You can put in the query more specific parameters to clarify your request, such as: api/pictures/category/cat?color=black&breed=korat. Now you will get the subset of pictures of cats which havethe color black and are korat cats.

Note: threads take turns executing tasks. They're not actually running in parallel, just switching between each other very fast within a CPU core.

In Python use threads for I/O, not for heavy CPU computations

<mark>P-value says the probability of seeing something</mark> assuming null hypothesis is true

To perform the actual test, we do a z-test or a t-test. For t-tests you need to know the population standard deviation, which is often not possible. T-test only requires standard deviation of a sample, which is more realistic.

P value is defined as: <mark>probability of observing this effect, if the null hypothesis is true(i.e the commonly accepted claim about a population). </mark>

Why?

Covariance: How much does X vary from its mean when Y varies from its mean.

• Notice that x_i and y_i can be +ve or -ve, therefore their product will also be +ve(x_i deviates in +ve direction away from mean and y_i also deviates in +ve direction away from mean or they both deviate negatively away from the mean) or -ve(x_i deviates +vely and y_i deviates -vely from the mean and viceversa). We then sum all of the products up. A +ve covariance will indicate that on average the values had a +ve linear relationship, -ve indicates a -ve linear relationship, and 0 indicates that all products cancelled out which means they were equally +ve and -ve relationships, and therefore no overall relatonship.

average prediction

given the same learning rate between 2 examples, as learning rate also affects the learning.

$$w = w - grad*lr$$

.

• LSTM's resolve vanishing gradient problem of RNN, can be seen as evolution of RNN(i.e better)
• Well suited on time series data.

performs the same operation on every input of data,except it also previous outputs(called context) as another input.

Description:

First.it takes the X(0) from the sequence of input and then it outputs h(0) which together with X(1) is the input for the next step. So, the h(0) and X(1) is the input for the next step. Similarly, h(1) from the next is the input with X(2) for the next step and so on

1. we train enough to get to a good area in the loss function.
2. We have a high learning rate(but not too high) so we can explore our surroundings and stumble upon nearby high performing minima. We periodically save the weights(every x epochs)
3. We average the weights . As a result, the averaged weights will be centered around the loss. See left picture below

Training with a small batch size has a regularizing effect, and like most regularizers, can lead to very good generalization. In general, batch size 1 has the best generatlization:

Small batches can offer a regularizing effect (Wilson and Martinez, 2003), perhaps due to the noise they add to the learning process. Generalization error is often best for a batch size of 1. Training with such a small batch size might require a small learning rate to maintain stability because of the high variance in the estimate of the gradient. The total runtime can be very high as a result of the need to make more steps, both because of the reduced learning rate and because it takes more steps to observe the entire training set. (Deep Learning Book, p276)

• Random initialization
• Noise from sampling mini-batches

causes neural networks with the same architecture to converge to <mark>different local minima, but very similar error rates.</mark>.

All layers declared inside of a neural network's __init__ method or as part of nn.Sequential automatically have their parameters set up with requires_grad=True and is_leaf=True. Autograd will therefore automatically store their gradients during a backward() call

the assumption in "Train longer, generalize better" is that it is due to making fewer updates: "we conducted experiments to show empirically that the "generalization gap" stems from the relatively small number of updates rather than the batch size, and can be completely eliminated by adapting" the training regime they use.

Wrap your equation between 2$ on each side. $$Example$$.

Another key point is that test distribution and train distribution are not always identical, therefore if you're in a flat area and test distribution shifts, you will still be in the flat area of the loss function but more around the edges. If you're in a sharp minima and distribution shifts, you're kicked out of it and end up somewhere higher up on the loss surface.

You should start with CIFAR-10 and MNIST only to get some initial results, but to see if those ideas hold up more broadly, test them on more realistic datasets like ImageWoof, ImageNet.

RAdam reduces the variance of the adaptive learning rate early on Source

• Their range is [0.0, 0.1]. They then do a couple of orders of magnitude through the range: [1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6].
• Logic for this is is once you find the best magnitude for your architecture/data distribution combination, you can then tune further by selecting a weight decay along that specific magnitude, this time by grid searching through the much smaller range.

Before using a hyperparmeter, first test that it will add value to your model. Only after that decide on value for it. If you don't want to use the default values, which you probably shouldn't given that hyperparameter values are very dependent on model architecture and dataset, then do something like grid search or randomized search through a model to find the best hyperparameter

can also use loss: If training loss decreases while validation loss increases -> overfitting.

i.e you can also set other hyperparameters(weight decay, learning rate, momentum, etc) per layer instead of whole network

l2 regularization can be done either by adding a penalty to the loss function and also directly to the weights through weight decay

This is very similar to what a snapshot ensemble does, except that a snapshot ensemble doesn't average out the weights at the end. Instead, it uses each network it saved as part of an ensemble during inference.

Pathological curvature is a big problem for gradient descent and makes it significantly slow down: for a ravine, instead of going straight down through it, it oscillates sideways a lot since that's the direction where the gradient is larger, as it is immediately steeper: