A new import is the _LRScheduler which we will use to implement our learning rate finder.

We will also show how to initialize the weights of our neural network and how to find a suitable learning rate using a modified version of the learning rate finder.

Gonnet, G., Stewart, J., Lafleur, J., Keith, S., McLellan, M., Jiang-Gorsline, D., & Snider, T. (2021). Analysis of feature influence on Covid-19 Death Rate Per Country Using a Novel Orthogonalization Technique. MetaArXiv. https://doi.org/10.31222/osf.io/4kw2n

Ward, K. P. (2021). Parenting During COVID-19: A Sentiment Analysis of Reddit Data. PsyArXiv. https://doi.org/10.31234/osf.io/4ukmd

作者提出了一种新的Alrao优化算法，让网络中每个 unit 或 feature 都各自从不同级别的随机分布中采样获得其自己的学习率。该算法没有额外计算损耗，可以更快速达到理想 lr 下的SGD性能，用来测试 DL 模型很棒！

深度神经网络训练(收敛/泛化性能)的近似Fisher信息矩阵表征，可自动优化mini-batch size/learning rate

挺有趣的 paper，提出了从 Fisher 矩阵抽象出新的量用来衡量训练过程中的模型表现，来优化mini-batch sizes and learning rates | 另外 paper 中的figure画的很好看 | 作者认为逐步增加batch sizes的传统理解只是partially true，存在逐步递减该 size 来提高 model 收敛和泛化能力的可能。