2 Matching Annotations
  1. May 2023
    1. It turns out that backpropagation is a special case of a general techniquein numerical analysis called automatic differentiat

      Automatic differentiation is a technique in numerical analysis. That's why Real Analysis is an important Mathematics area that should be studied if one wants to go into AI research.

  2. Feb 2023
    1. Skip connections serve two purposes. The first is that they help keep the gradient smooth, which is a big help for backpropagation. Attention is a filter, which means that when it’s working correctly it will block most of what tries to pass through it.