Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows
In this tutorial, we deploy the 1-bit Bonsai-27B language mannequin utilizing the PrismML fork of llama.cpp, which gives the specialised CUDA kernels required to decode the mannequin’s Q1_0_g128 GGUF quantization format. We start by validating the GPU runtime, putting in the required Python dependencies, compiling the CUDA-enabled inference binaries, and downloading the compressed mannequin weights from Hugging Face. We then take a look at the mannequin via llama-cli, launch an OpenAI-compatible native inference server, and work together with it via a reusable Python shopper that helps commonplace completions, streamed responses, multi-turn conversations, and code era. We additionally study optionally available configurations for throughput benchmarking, quantized key-value caching, long-context inference, speculative decoding, and multimodal extensions.
import os
import sys
import time
import json
import shutil
import subprocess
import multiprocessing
WORK_DIR = "/content material"
REPO_URL = "https://github.com/PrismML-Eng/llama.cpp"
REPO_DIR = os.path.be a part of(WORK_DIR, "llama.cpp")
BUILD_DIR = os.path.be a part of(REPO_DIR, "construct")
BIN_DIR = os.path.be a part of(BUILD_DIR, "bin")
HF_REPO = "prism-ml/Bonsai-27B-gguf"
MODEL_FILE = "Bonsai-27B-Q1_0.gguf"
MODEL_PATH = os.path.be a part of(WORK_DIR, MODEL_FILE)
SERVER_HOST = "127.0.0.1"
SERVER_PORT = 8080
SERVER_URL = f"http://{SERVER_HOST}:{SERVER_PORT}"
GEN_PARAMS = {"temperature": 0.7, "top_p": 0.95, "top_k": 20}
CTX_SIZE = 8192
N_GPU_LAYERS = 99
USE_KV_Q4 = False
def sh(cmd, verify=True, **kw):
"""Run a shell command, streaming output to the pocket book."""
print(f"n$ {cmd}")
return subprocess.run(cmd, shell=True, verify=verify, **kw)
print("=" * 70)
print("[1/7] Checking surroundings")
print("=" * 70)
gpu = subprocess.run("nvidia-smi --query-gpu=identify,reminiscence.complete --format=csv,noheader",
shell=True, capture_output=True, textual content=True)
if gpu.returncode != 0:
sys.exit("No GPU detected. In Colab: Runtime -> Change runtime sort -> GPU (T4).")
print(f"GPU detected: {gpu.stdout.strip()}")
print("Bonsai-27B wants solely ~5.2 GB peak at 4K context — any Colab GPU works.")
sh("pip -q set up huggingface_hub requests")
We configure the Colab workspace, mannequin repository, server endpoint, inference parameters, context measurement, and GPU offloading settings required all through the tutorial. We outline a reusable shell-command perform and confirm that the runtime exposes a suitable NVIDIA GPU earlier than persevering with. We then set up the Hugging Face Hub and HTTP shopper dependencies wanted for mannequin retrieval and API communication.
print("=" * 70)
print("[2/7] Building PrismML llama.cpp fork with CUDA (cached after 1st run)")
print("=" * 70)
if not os.path.isdir(REPO_DIR):
sh(f"git clone --depth 1 {REPO_URL} {REPO_DIR}")
else:
print("Repo already cloned — skipping.")
cli_bin = os.path.be a part of(BIN_DIR, "llama-cli")
server_bin = os.path.be a part of(BIN_DIR, "llama-server")
bench_bin = os.path.be a part of(BIN_DIR, "llama-bench")
if not (os.path.exists(cli_bin) and os.path.exists(server_bin)):
jobs = multiprocessing.cpu_count()
sh(f"cmake -S {REPO_DIR} -B {BUILD_DIR} -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release")
sh(f"cmake --build {BUILD_DIR} -j{jobs} --target llama-cli llama-server llama-bench")
else:
print("Binaries already constructed — skipping.")
We clone the PrismML fork of llama.cpp, which gives the specialised kernels required for the mannequin’s Q1_0_g128 quantization format. We configure a CUDA-enabled launch construct with CMake and compile the command-line, server, and benchmarking executables. We additionally reuse beforehand generated binaries after they exist already, lowering repeated setup time in the identical Colab session.
print("=" * 70)
print("[3/7] Downloading weights from Hugging Face")
print("=" * 70)
from huggingface_hub import hf_hub_download
if not os.path.exists(MODEL_PATH):
downloaded = hf_hub_download(repo_id=HF_REPO, filename=MODEL_FILE,
local_dir=WORK_DIR)
print(f"Downloaded to: {downloaded}")
else:
print("Model already on disk — skipping.")
print(f"Model measurement on disk: {os.path.getsize(MODEL_PATH) / 1e9:.2f} GB")
We hook up with the Hugging Face Hub and obtain the Bonsai-27B GGUF mannequin into the Colab workspace. We skip the switch when the mannequin file is already accessible domestically, permitting subsequent runs to proceed extra effectively. We then calculate and show the deployed mannequin measurement to substantiate that the compressed weights are saved appropriately.
print("=" * 70)
print("[4/7] Smoke take a look at with llama-cli")
print("=" * 70)
sh(
f'{cli_bin} -m {MODEL_PATH} '
f'-p "Explain in two sentences why 1-bit quantization saves reminiscence." '
f'-n 128 -ngl {N_GPU_LAYERS} '
f'--temp {GEN_PARAMS["temperature"]} '
f'--top-p {GEN_PARAMS["top_p"]} --top-k {GEN_PARAMS["top_k"]} '
f'-no-cnv 2>/dev/null',
verify=False,
)
print("=" * 70)
print("[5/7] Starting llama-server (OpenAI-compatible API)")
print("=" * 70)
import requests
kv_flags = "-ctk q4_0 -ctv q4_0" if USE_KV_Q4 else ""
server_cmd = (
f"{server_bin} -m {MODEL_PATH} "
f"--host {SERVER_HOST} --port {SERVER_PORT} "
f"-ngl {N_GPU_LAYERS} -c {CTX_SIZE} {kv_flags}"
)
print(f"$ {server_cmd} (background)")
server_log = open(os.path.be a part of(WORK_DIR, "server.log"), "w")
server_proc = subprocess.Popen(server_cmd, shell=True,
stdout=server_log, stderr=server_log)
for _ in vary(120):
attempt:
if requests.get(f"{SERVER_URL}/well being", timeout=2).status_code == 200:
print("Server is up.")
break
besides requests.exceptions.RequestException:
move
time.sleep(2)
else:
server_proc.kill()
sys.exit("Server failed to begin — verify /content material/server.log")
We carry out a command-line smoke take a look at to confirm that the compiled runtime can load the quantized mannequin and generate a legitimate response. We then begin llama-server with full GPU layer offloading, the chosen context window, and optionally available quantized KV-cache settings. We repeatedly question the well being endpoint till the OpenAI-compatible inference service turns into prepared for shopper requests.
print("=" * 70)
print("[6/7] Talking to Bonsai-27B through the OpenAI-compatible API")
print("=" * 70)
def chat(messages, stream=False, max_tokens=512, **overrides):
"""Minimal OpenAI-compatible chat shopper for the native llama-server."""
payload = {
"mannequin": "bonsai-27b",
"messages": messages,
"max_tokens": max_tokens,
"stream": stream,
**GEN_PARAMS,
**overrides,
}
if not stream:
r = requests.submit(f"{SERVER_URL}/v1/chat/completions", json=payload)
r.raise_for_status()
return r.json()["choices"][0]["message"]["content"]
r = requests.submit(f"{SERVER_URL}/v1/chat/completions", json=payload, stream=True)
r.raise_for_status()
full = []
for line in r.iter_lines():
if not line or not line.startswith(b"knowledge: "):
proceed
chunk = line[len(b"data: "):]
if chunk == b"[DONE]":
break
delta = json.hundreds(chunk)["choices"][0]["delta"].get("content material", "")
full.append(delta)
print(delta, finish="", flush=True)
print()
return "".be a part of(full)
SYSTEM = {"function": "system", "content material": "You are a useful assistant"}
print("n--- 6a: fundamental completion ---")
reply = chat([SYSTEM, {"role": "user",
"content": "What is the capital of France? One sentence."}])
print(reply)
print("n--- 6b: math reasoning, streamed token-by-token ---")
chat([SYSTEM, {"role": "user",
"content": "A train travels 120 km at 80 km/h, then 90 km at "
"60 km/h. What is its average speed for the whole "
"trip? Show your reasoning briefly."}],
stream=True, max_tokens=700)
print("n--- 6c: multi-turn chat ---")
historical past = [SYSTEM]
for user_msg in ["My name is Priya and I love graph algorithms.",
"Suggest one project idea that combines my interest with LLMs.",
"What was my name again?"]:
historical past.append({"function": "consumer", "content material": user_msg})
reply = chat(historical past, max_tokens=300)
historical past.append({"function": "assistant", "content material": reply})
print(f"nUSER: {user_msg}nBONSAI: {reply}")
print("n--- 6d: code era ---")
print(chat([SYSTEM, {"role": "user",
"content": "Write a Python function that returns the n-th "
"Fibonacci number using memoization. Code only."}],
max_tokens=400))
We outline a reusable Python chat shopper that sends OpenAI-compatible requests to the domestically hosted Bonsai-27B server. We help each commonplace and server-sent-event streaming responses whereas making use of the configured temperature, top-p, and top-k sampling parameters. We then consider fundamental factual answering, mathematical reasoning, multi-turn conversational reminiscence, and Python code era.
print("=" * 70)
print("[7/7] Optional extras")
print("=" * 70)
RUN_BENCHMARK = False
if RUN_BENCHMARK:
sh(f"{bench_bin} -m {MODEL_PATH} -ngl {N_GPU_LAYERS}", verify=False)
print("""
NOTES & NEXT STEPS
------------------
* Long context: the mannequin helps as much as 262K tokens. On Colab, increase
CTX_SIZE and set USE_KV_Q4 = True (4-bit KV cache) — with it, 100K-token
contexts slot in roughly 6.8 GB peak, effectively inside a T4's 16 GB.
* Speculative decoding: the repo ships a DSpark drafter (Q4_1, ~1.79 GB)
that offers a lossless ~1.37x decode speedup on CUDA. See the PrismML
llama.cpp fork's README for the serving flags, and obtain the drafter
pack from the identical HF repo if you wish to attempt it.
* Vision: an optionally available ~0.63 GB mmproj pack provides picture enter; it is just
loaded when a picture arrives, so text-only serving by no means pays for it.
* Quality vs measurement: in order for you extra headroom, the ternary sibling
(prism-ml/Ternary-Bonsai-27B-gguf, ~5.9 GB, ~95% of FP16) is a drop-in
swap — simply change HF_REPO / MODEL_FILE above.
* Shutting down: run server_proc.kill() in a later cell to free the GPU.
""")
print("Done. The server remains to be working — name chat([...]) from new cells!")
We expose an optionally available benchmarking swap that measures prompt-processing and token-generation efficiency with the compiled llama-bench executable. We assessment superior deployment choices, together with long-context inference, 4-bit KV caching, speculative decoding, imaginative and prescient help, and a higher-capacity ternary mannequin variant. We end whereas retaining the server course of lively in order that we will proceed calling the chat perform from further Colab cells.
In conclusion, we established a full native inference workflow for working Bonsai-27B. We used the PrismML implementation to protect compatibility with the mannequin’s extremely compressed 1.125-bit weight illustration whereas retaining the total inference pipeline accessible via each command-line and OpenAI-compatible interfaces. We validated reasoning, conversational reminiscence, streaming era, and programming capabilities, whereas retaining management over sampling parameters, context size, GPU offloading, and KV-cache precision. This setup provides us a platform for experimenting with low-bit giant language fashions, evaluating their effectivity and output high quality, and integrating them into Python functions with out counting on an exterior hosted inference service.
Check out the Full Code here. Also, be happy to comply with us on Twitter and don’t neglect to affix our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to accomplice with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and many others.? Connect with us
The submit Deploying a 1-Bit Bonsai-27B Model with PrismML llama.cpp and OpenAI-Compatible Local Inference Workflows appeared first on MarkTechPost.
