epsilon greedy

the crux

so rewards, and reward functions can be defined over S x A x S

no, on Q-learning the greedy action in the bellman equation is taken before the update, but the next step's action is generated from the updated Q.

whereas in SARSA with a greedy policy, the same greedy action is used in the update equation, and it is also used taken to generate the next state

because the policy for which Q estimates is the one that is greedy w.r.t Q, but the policy generating the samples can be anything

see 5.9, this is per-decision importance sampling

usually a function of the likelihood function plus some penalty for model complexity

sum of t independent innovations

due to having uncorrelated innovations

applying the inverse, renders the process an infinite order MA process

geometric series

function on eta_t

how does this affect the process over time?

e.g. changing interest rates on the long term economy

more data than parameters

S_t is a weighted average of white noise

actor-critic with state value baseline update, with discounting!

del ln(A|S, theta) is actually CrossEntropy

modified TD error in terms of regular TD error

A_{t+1}^t = I

we've achieved something special here - the recursive definition of theta_{t+1} on theta_t

this is already computationally quite nice, but these jokers want to incorporate the last term into the modified TD error

lambda return = lambda weighted sum of all n-step returns up to time t+1

the dutch trace update rule

weighted sum of all available n-step lambda returns from t=1

assuming the special linear case, with a discrete state space and where the feature vector is a one-hot encoding, the TD-error multipled by the eligbility vector is the effect of the current TD-error on each state, with the effect amplified (or de-amplified) by each states' recency of occurence.

the current TD error contributes less and less to those states that occur furhter back in time (using the linear case helps, where the eligibility trace is a sum of past, fading state input vectors).

sum of all remaining n-step returns

should this be inside the bracket??

yes, because rewards can propagate in from both sides now, making the optimal n shorter?

G(t:t+n) needs all rewards from time t+1

at the current time step t+n, we need the states and actions from time = t (i.e. n steps back)

use the most up-to-date version

hmm sure starting states can have an impact?

interesting, read!

V(terminal state) = 0

notice that the multiplier of the gradient here: G_t / pi(a|s) is positive, meaning we are always going in the same direction as the gradient. using a baseline G_t - v(S_t) allows us to revers this direction if G_t is lower than the baseline

actor critic algorithm one step TD:

very similar to box 199 but without h(s)

TD(0) update

same as REINFORCE MC baseline, but with the sampled G replaced with a bootstrapped G

constant baseline?

termination because discounting by gamma is equivalent to a non-disounted case, but with termination probability gamma

TODO: read this and find out what the philosophy behind parts-based model is??

what is this?

TODO: read

What? Clarify

super interesting, but hardly applicable. do rad though!

what

seems interesting, "iteratively integrate the orthogonality constraint in CNN training"

read paper again to see if any good ideas for architecture

Bottleneck layers do a 1x1 convolution to reduce the dimensionality, before a 3x3 convolution, to save computation

https://medium.com/@erikgaas/resnet-torchvision-bottlenecks-and-layers-not-as-they-seem-145620f93096

someething that came up whilst looking through papers in attention: https://arxiv.org/pdf/1709.01507.pdf squeeze-and-excitation

Parts-based paper, interesting approach