Multi-GPU Parallelism

  • Here is a quick overview of the four different paradigms for multi-GPU training.
    • Data parallelism
    • Model parallelism
    • Pipeline parallelism
    • Tensor parallelism
  • Credits to Sebastian Raschka for the infographic below.

Citation

If you found our work useful, please cite it as:

@article{Chadha2020DistilledModelCompression,
  title   = {Model Compression},
  author  = {Chadha, Aman},
  journal = {Distilled AI},
  year    = {2020},
  note    = {\url{https://aman.ai}}
}