Transformer-engine

Transformer Engine is an open-source, high-performance library developed by NVIDIA designed to accelerate the training speeds of Transformer models on modern Hopper and Blackwell GPUs. It utilizes intelligent FP8 mixed-precision mathematical quantization tracking to maximize tensor core throughput while retaining structural model accuracy.