Here are the key points:
- Deep learning was enabled by advancements in hardware, specifically GPUs that provided enough compute power to train large models on big datasets in a reasonable amount of time.
- Since AlexNet in 2012, the compute demand for state-of-the-art deep learning models has increased exponentially, by around 10⁶x. NVIDIA GPU performance has increased ~1000x over the past decade to meet this demand, while the rest has come from scaling up the number of GPUs and training time.
- To continue improving hardware performance, key techniques include using optimal number representations like log numbers, exploiting sparsity, scaling at a finer granularity like per-vector instead of per-layer, and specialized instructions to amortize overhead costs.
- Deep learning software stacks like NVIDIA’s are critical, demonstrated by large gaps in performance on benchmarks like MLPerf. Building the software takes huge effort.
- Accelerators can achieve higher efficiency through techniques like reduced precision, massive parallelism, optimized memory, and amortized overhead. But programmable solutions like GPUs have advantages too.
- Future directions include better sparse execution, optimal clipping for scaling, and 3D memory stacking to increase bandwidth and reduce energy. An accelerator prototype shows potential for 10x gains beyond current GPUs.
