Four open supply tasks dominate LLM fine-tuning at present. Unsloth, Axolotl, TRL, and LLaMA-Factory all wrap the identical underlying PyTorch and Hugging Face stack. They diverge on the place they spend engineering effort.

Unsloth rewrites kernels. Axolotl composes parallelism methods. TRL defines the coach APIs the others construct on. LLaMA-Factory optimizes for breadth of mannequin protection and zero-code operation.

This comparability covers three axes engineers truly hit: coaching throughput, peak VRAM, and multi-GPU scaling.

Understand every framework

• TRL is the reference implementation layer. It ships SFTTrainer, DPOTrainer, GRPOTrainer, KTOTrainer, RewardCoach, and RLOOTrainer. Axolotl and LLaMA-Factory each name into it. The present secure launch line is v1.8.0.
• Unsloth replaces elements of the modeling code with hand-written Triton kernels. Backpropagation steps are manually derived slightly than autograd-generated. Hugging Face’s own writeup notes accuracy degradation is 0% versus commonplace QLoRA, as a result of no approximations are launched.
• Axolotl is a YAML-driven wrapper over Transformers, PEFT, TRL, Accelerate, and DeepSpeed. Its differentiator is composability of parallelism methods, not kernel work.
• LLaMA-Factory is an ACL 2024 system demonstration paper with a Gradio net UI known as LlamaBoard. The repository covers 100+ LLMs and VLMs.

Speed

Unsloth: kernel-level features on a single GPU

Unsloth's published benchmarks present 2x coaching velocity for Llama 3.1 8B and Llama 3.3 70B. The setup used the Alpaca dataset, batch measurement 2, and gradient accumulation 4. QLoRA ran at rank 32 on all linear layers.

The MoE results are bigger. Unsloth fine-tuned unsloth/gpt-oss-20b-BF16 on an NVIDIA B200. It studies 712.33 ms per step at 8K context, versus 5,226.86 ms for Transformers v5. That is a 7.3x hole. At 4K the hole is 4.82x, and at 1K it is just 1.37x.

The pattern course is model-dependent, and Unsloth's docs scope this declare to gpt-oss. There, the speedup grows with sequence size, credited to Flex Attention and the MoE kernels.

Qwen3-30B-A3B on B200 runs the opposite manner. Its reported speedup falls from 1.7x at 1K to 1.1x at 16K. Memory financial savings transfer the other way, rising from about 2% to fifteen%.

Qwen3-30B-A3B on H100 reaches as much as 1.77x. GLM-4.7-Flash on RTX PRO 6000 reaches 2.1x. A collaboration with AMD measured Llama-3.1-8B LoRA SFT at 2.07 s/step. TRL plus FlashAttention-2 took 2.87 s/step, a 1.39x hole with matching loss curves.

Axolotl: kernels borrowed, parallelism native

Axolotl added custom Triton kernels and autograd functions for LoRA in February 2025, explicitly citing Unsloth as inspiration. They are opt-in by way of lora_mlp_kernel, lora_qkv_kernel, and lora_o_kernel.

Recent release notes add SonicMoE LoRA assist. It delivers as much as 1.45x speedup and 30% reminiscence discount over a grouped_mm baseline. That determine is for Qwen3.5-35B-A3B 8-bit LoRA on a single H100 SXM.

Axolotl additionally ships FlashAttention 2/3/4, xFormers, Flex Attention, SageAttention, Liger Kernel, Cut Cross Entropy, and ScatterMoE.

TRL: the baseline everybody measures towards

TRL is often the reference level slightly than the winner on uncooked single-GPU throughput. It compensates with breadth of reminiscence and velocity levers documented in Reducing Memory Usage and Speeding Up Training.

Those levers embrace packing, padding-free batching, truncation, Liger Kernel, and vLLM sleep mode for GRPO. TRL additionally has a first-party Unsloth integration, so the 2 usually are not mutually unique.

LLaMA-Factory: velocity by delegation

LLaMA-Factory doesn't write its personal kernels. It exposes different individuals's work by way of config flags.

Setting use_unsloth: true prompts the Unsloth patch. The mission's changelog studies 170% relative velocity from that path. Unsloth's long-sequence coaching is listed at 117% velocity and 50% reminiscence. It additionally helps enable_liger_kernel: true and FlashAttention-2 through flash_attn: fa2.

VRAM

Reported reminiscence flooring

Unsloth publishes a VRAM requirements table sorted by parameter depend. It lists 6 GB for an 8B mannequin in 4-bit QLoRA and 41 GB for 70B. LoRA at 16-bit prices 22 GB and 164 GB for a similar fashions.

LLaMA-Factory's README hardware table covers the identical regime for 4-bit QLoRA. It lists 6 GB at 7B, 24 GB at 30B, and 48 GB at 70B. Full bf16 fine-tuning of 70B is listed at 600 GB.

Both tables describe minimums. Batch measurement, sequence size, and optimizer selection transfer the true quantity.

Context size is the sharper differentiator

Peak VRAM at a set context is much less fascinating than the utmost context a given VRAM price range permits. Unsloth's context length benchmarks for Llama 3.1 8B QLoRA at rank 32 and batch measurement 1 are stark.

GPU VRAM	Unsloth context	Transformers + FA2 context
8 GB	2,972	OOM
16 GB	40,724	2,551
24 GB	78,475	5,789
48 GB	191,728	15,502
80 GB	342,733	28,454

Unsloth attributes this to its gradient checkpointing algorithm mixed with Apple's Cut Cross Entropy. For Llama 3.3 70B on an 80 GB A100, it studies 89,389 tokens. The FA2 baseline reaches 6,916.

The MoE reminiscence story

MoE coaching is the place reminiscence conduct has shifted most in 2026. Unsloth studies gpt-oss-20b fine-tuning inside 12.8 GB, whereas Qwen3-30B-A3B at 16-bit LoRA wants 63 GB.

Its B200 gpt-oss run used 47.43 GB at 8K context the place Transformers v5 used 73.80 GB. At 16K, Transformers v5 went out of reminiscence and Unsloth used 55.13 GB.

The mechanism is a split-LoRA formulation. PEFT materializes the LoRA delta throughout all specialists earlier than the MoE matmul. Unsloth reorders the operations as a substitute, which is mathematically an identical however avoids the materialization.

Axolotl assaults the identical drawback otherwise. Its MoE expert quantization quantizes knowledgeable weights throughout mannequin loading, releasing the unique bf16 tensor instantly.

The cause is a Transformers v5 change. Expert layers moved from nn.Linear to fused nn.Parameter 3D tensors. bitsandbytes might now not quantize them on load. Axolotl's docs report GLM-4.7-Flash QLoRA dropping from roughly 127 GiB to roughly 23 GiB reserved reminiscence with quantize_moe_experts: true.

Multi-GPU

This is the place the rating inverts. Unsloth's single-GPU lead doesn't carry over.

Axolotl: the deepest parallelism matrix

Axolotl's multi-GPU guide provides three mutually unique sharding methods: DeepSpeed ZeRO levels 1 by way of 3, FSDP, and DDP. FSDP2 is the really helpful path, and FSDP1 is deprecated.

On prime of these, its N-D Parallelism guide composes knowledge, tensor, context, and knowledgeable parallelism by way of PyTorch's DeviceMesh. The documented assist matrix confirms FSDP+TP, HSDP+TP, FSDP+CP, FSDP+TP+CP, and FSDP+EP.

Two mixtures are explicitly unsupported. Expert parallelism can't compose with TP or CP in v1. Pure DDP can't compose with them both.

Axolotl's sequence parallelism makes use of the ring-flash-attention library. Its published H100 benchmark for Llama 3.1 8B QLoRA helps the tradeoff.

SP diploma	Max context	Context scaling	Tokens/sec	Speedup / GPU depend
1	17,408	1.00x	9,104	100.0%
2	34,816	2.00x	15,806	86.8%
4	66,560	3.82x	12,314	33.8%
8	129,024	7.41x	11,096	15.2%

Context scales near linearly. Throughput effectivity collapses. On 4090s at SP diploma 8, the identical benchmark information 0.88x speedup. Training obtained slower whereas context reached 32,768 tokens.

Axolotl additionally helps multi-node coaching by way of torchrun and Ray.

TRL: two sequence-splitting backends

TRL's distributed training guide attracts a clear distinction. Context Parallelism means Ring Attention on FSDP2. Sequence Parallelism means ALST/Ulysses on DeepSpeed.

Ring Attention wants Accelerate 1.11.0+, makes use of cp_size with cp_backend="torch", and at present helps SDPA solely. FlashAttention isn't supported on that path. Sequences should divide by cp_size * 2.

ALST/Ulysses wants DeepSpeed 0.18.1+ and Accelerate 1.12.0+. It makes use of sp_size with sp_backend="deepspeed" and works with FlashAttention-2. It is bounded by consideration head depend, requiring num_heads >= sp_size.

TRL's steering is restricted. Ring Attention fits 1M+ token sequences and restricted community topology. Ulysses fits NVLink or InfiniBand interconnects and sequences as much as roughly 500k tokens.

TRL's personal Ring Attention benchmark fine-tuned Qwen3-8B throughout 1, 2, 4, and 8 H100 GPUs. At 8 GPUs, context lengths above 300k tokens turned trainable.

LLaMA-Factory: commonplace engines with Megatron

LLaMA-Factory's distributed training docs cowl DDP, DeepSpeed, and FSDP, together with FSDP2 and Ray for single-node and multi-node runs. The docs additionally describe DeepSpeed AutoTP, which mixes tensor parallelism with ZeRO.

Its most consequential 2025 addition was a Megatron-core coaching backend by way of mcore_adapter. That opens a real large-scale pretraining path.

The FSDP+QLoRA path fine-tunes a 70B mannequin on two 24 GB GPUs. That is the most cost effective documented path to 70B on this comparability.

The friction level is the interface. Distributed configuration lives in YAML and CLI, not in LlamaBoard. Dev groups that adopted LLaMA-Factory for its zero-code UI lose that property once they scale previous one GPU.

Unsloth: the open hole

Unsloth's multi-GPU documentation states that multi-GPU works by way of Accelerate and DeepSpeed, giving entry to FSDP and DDP. It additionally states the method is advanced and requires guide setup, with official assist nonetheless being introduced.

The sensible route is speed up launch practice.py or torchrun --nproc_per_node N_GPUS practice.py. For fashions too massive for one GPU, device_map = "balanced" splits the mannequin throughout units.

Unsloth's PyPI listing marks multi-GPU as obtainable with main enhancements pending. The Studio changelog describes preliminary automated multi-GPU allocation for inference and coaching as of March 2026.

Read collectively, the place is evident. Unsloth helps multi-GPU. It doesn't but provide the composable parallelism matrix that Axolotl and TRL doc.

Where each breaks

• Unsloth breaks once you want tensor, context, or knowledgeable parallelism as first-class config. It additionally breaks when your mannequin isn't in its supported listing, since features come from architecture-specific kernels.
• Axolotl breaks on the educational curve. You configure FSDP2 versus DeepSpeed, SP diploma, and divisibility constraints. Those constraints span GPU depend, sequence size, and consideration heads.
• TRL breaks on defaults. It provides you appropriate primitives, not tuned ones. You provide the Accelerate config, the reminiscence optimizations, and the parallelism plan.
• LLaMA-Factory breaks on the UI boundary. Its abstraction is great as much as one node and skinny above it.

Choosing

• Single shopper GPU, supported structure, LoRA or QLoRA: Unsloth. The context-length headroom alone justifies it.
• Two to eight GPUs, lengthy context, full fine-tuning or RLHF pipelines: Axolotl. FSDP2 plus sequence parallelism is the best-documented path.
• Custom coaching loops, novel post-training algorithms, tight Hugging Face coupling: TRL. You are constructing on the layer the others wrap.
• Broadest mannequin protection, non-engineer operators, quickest first run: LLaMA-Factory. Then transfer to CLI once you scale.
• These selections usually are not unique. LLaMA-Factory can run Unsloth as a backend. TRL ships an Unsloth integration. Axolotl calls TRL trainers internally.

Key Takeaways

• Unsloth wins single-GPU velocity and context size; multi-GPU stays its documented weak level.
• Axolotl ships the deepest parallelism matrix: FSDP2, DeepSpeed, TP, CP, and EP composable through DeviceMesh.
• TRL is the primitive layer others wrap, now with Ring Attention and ALST/Ulysses sequence splitting.
• LLaMA-Factory trades depth for breadth: 100+ fashions, zero-code UI, Megatron backend, CLI-only distributed setup.
• Sequence parallelism scales context near-linearly however throughput effectivity drops sharply previous 4 GPUs.

The put up Unsloth vs Axolotl vs TRL vs LLaMA-Factory: A Fine-Tuning Framework Comparison on Speed, VRAM, and Multi-GPU appeared first on MarkTechPost.