|

Meet oLLM: A Lightweight Python Library that brings 100K-Context LLM Inference to 8 GB Consumer GPUs via SSD Offload—No Quantization Required

ByRicardo

oLLM is a light-weight Python library constructed on prime of Huggingface Transformers and PyTorch and runs large-context Transformers on NVIDIA GPUs by aggressively offloading weights and KV-cache to quick native SSDs. The undertaking targets offline, single-GPU workloads and explicitly avoids quantization, utilizing FP16/BF16 weights with FlashAttention-2 and disk-backed KV caching to preserve VRAM inside 8–10 GB whereas dealing with up to ~100K tokens of context.

But What’s new?

(1) KV cache learn/writes that bypass mmap to scale back host RAM utilization; (2) DiskCache assist for Qwen3-Next-80B; (3) Llama-3 FlashAttention-2 for stability; and (4) GPT-OSS reminiscence reductions via “flash-attention-like” kernels and chunked MLP. The desk printed by the maintainer studies end-to-end reminiscence/I/O footprints on an RTX 3060 Ti (8 GB):

  • Qwen3-Next-80B (bf16, 160 GB weights, 50K ctx) → ~7.5 GB VRAM + ~180 GB SSD; famous throughput “≈ 1 tok/2 s”.
  • GPT-OSS-20B (packed bf16, 10K ctx) → ~7.3 GB VRAM + 15 GB SSD.
  • Llama-3.1-8B (fp16, 100K ctx) → ~6.6 GB VRAM + 69 GB SSD.

How it really works

oLLM streams layer weights straight from SSD into the GPU, offloads the eye KV cache to SSD, and optionally offloads layers to CPU. It makes use of FlashAttention-2 with on-line softmax so the complete consideration matrix isn’t materialized, and chunks giant MLP projections to sure peak reminiscence. This shifts the bottleneck from VRAM to storage bandwidth and latency, which is why the oLLM undertaking emphasizes NVMe-class SSDs and KvikIO/cuFile (GPUDirect Storage) for high-throughput file I/O.

Supported fashions and GPUs

Out of the field the examples cowl Llama-3 (1B/3B/8B), GPT-OSS-20B, and Qwen3-Next-80B. The library targets NVIDIA Ampere (RTX 30xx, A-series), Ada (RTX 40xx, L4), and Hopper; Qwen3-Next requires a dev construct of Transformers (≥ 4.57.0.dev). Notably, Qwen3-Next-80B is a sparse MoE (80B whole, ~3B energetic) that distributors sometimes place for multi-A100/H100 deployments; oLLM’s declare is that you possibly can execute it offline on a single client GPU by paying the SSD penalty and accepting low throughput. This stands in distinction to vLLM docs, which recommend multi-GPU servers for a similar mannequin household.

Installation and minimal utilization

The undertaking is MIT-licensed and obtainable on PyPI (pip set up ollm), with a further kvikio-cu{cuda_version} dependency for high-speed disk I/O. For Qwen3-Next fashions, set up Transformers from GitHub. A brief instance within the README reveals Inference(...).DiskCache(...) wiring and generate(...) with a streaming textual content callback. (PyPI presently lists 0.4.1; the README references 0.4.2 adjustments.)

Performance expectations and trade-offs

  • Throughput: The maintainer studies ~0.5 tok/s for Qwen3-Next-80B at 50K context on an RTX 3060 Ti—usable for batch/offline analytics, not for interactive chat. SSD latency dominates.
  • Storage strain: Long contexts require very giant KV caches; oLLM writes these to SSD to preserve VRAM flat. This mirrors broader business work on KV offloading (e.g., NVIDIA Dynamo/NIXL and group discussions), however the strategy continues to be storage-bound and workload-specific.
  • Hardware actuality examine: Running Qwen3-Next-80B “on client {hardware}” is possible with oLLM’s disk-centric design, however typical high-throughput inference for this mannequin nonetheless expects multi-GPU servers. Treat oLLM as an execution path for large-context, offline passes quite than a drop-in substitute for manufacturing serving stacks like vLLM/TGI.

Bottom line

oLLM pushes a transparent design level: preserve precision excessive, push reminiscence to SSD, and make ultra-long contexts viable on a single 8 GB NVIDIA GPU. It gained’t match data-center throughput, however for offline doc/log evaluation, compliance overview, or large-context summarization, it’s a realistic approach to execute 8B–20B fashions comfortably and even step up to MoE-80B for those who can tolerate ~100–200 GB of quick native storage and sub-1 tok/s technology.

Check out the GITHUB REPO here. Feel free to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Also, be at liberty to comply with us on Twitter and don’t neglect to be part of our 100k+ ML SubReddit and Subscribe to our Newsletter.

The publish Meet oLLM: A Lightweight Python Library that brings 100K-Context LLM Inference to 8 GB Consumer GPUs via SSD Offload—No Quantization Required appeared first on MarkTechPost.

Similar Posts