2 Matching Annotations
  1. Nov 2021
  2. lilianweng.github.io lilianweng.github.io
    How to Train Really Large Models on Many GPUs?
    1
    1. sherlockliao 09 Nov 2021
      It partitions optimizer state, gradients and parameters across multiple data parallel processes via a dynamic communication schedule to minimize the communication volume.

      ZeRO-DP 的原理是什么？

      ZeRO GPU memory Data Parallel
    Visit annotations in context

    Tags

    Annotators

    URL

    lilianweng.github.io/lil-log/2021/09/24/train-large-neural-networks.html
  3. Sep 2017
  4. tenso.rs tenso.rs
    TensorFire
    1
    1. dckc 01 Sep 2017
      webgl tensorflow deep-learning data parallel
    Visit annotations in context

    Tags

    Annotators

    URL

    tenso.rs/