How to Speed Up Transformer Training Using NVIDIA Apex (FusedAdam, FusedLayerNorm) and Native torch.amp
In this tutorial, we work by means of an implementation of NVIDIA Apex, specializing in the elements that also matter in fashionable GPU coaching workflows. Instead of treating Apex as a normal mixed-precision library, we separate the older elements from the still-useful ones and check them instantly. We start by checking the CUDA runtime, constructing…
