NVIDIA Introduces X-Token: Projection-Guided Cross-Tokenizer KD That Outperforms GOLD by +3.82 Average Points on Llama-3.2-1B
Knowledge distillation (KD) transfers “darkish data” from a big instructor mannequin to a smaller pupil. The pupil learns from the instructor’s full output chance distribution over tokens, not simply appropriate solutions. This is finished through per-position Kullback–Leibler (KL) divergence over next-token chance distributions. This formulation requires a shared tokenizer. A practitioner dedicated to Llama-3.2-1B can…
