IMDb Sentiment Analysis with DistilBERT LoRA, TF-IDF Baselines, Calibration, Interpretability, Robustness Testing, and Semi-Supervised Learning
In this tutorial, we develop an end-to-end sentiment evaluation workflow utilizing the Stanford NLP IMDb Large Movie Review Dataset and examine classical machine studying with parameter-efficient transformer fine-tuning. We start by establishing a reproducible surroundings and auditing the dataset for sophistication ordering, review-length skew, duplicate leakage, and preprocessing artifacts earlier than coaching a robust TF-IDF and Logistic Regression baseline. We then fine-tune DistilBERT with LoRA by PEFT, consider it utilizing accuracy, macro-F1, ROC-AUC, confusion matrices, and ROC curves, and look at threshold choice and likelihood calibration by Expected Calibration Error and reliability evaluation. Beyond headline metrics, we examine assured errors, efficiency throughout evaluate lengths, word-level occlusion saliency, and head-versus-tail truncation to know how the mannequin reaches its predictions and the place long-context limitations have an effect on efficiency. Finally, we use the unlabeled IMDb break up for confidence-based pseudo-labeling, examine the ensuing semi-supervised mannequin towards our baseline, and save the merged transformer for reusable sentiment inference.
import importlib.util, subprocess, sys, os, time, random, warnings, examine, hashlib
warnings.filterwarnings("ignore")
os.environ["TOKENIZERS_PARALLELISM"] = "false"
os.environ["WANDB_DISABLED"] = "true"
_REQUIRED = {
"transformers": "transformers",
"datasets": "datasets",
"peft": "peft",
"speed up": "speed up",
"sklearn": "scikit-learn",
}
_missing = [pkg for mod, pkg in _REQUIRED.items() if importlib.util.find_spec(mod) is None]
if _missing:
print(f"Installing: {', '.be part of(_missing)} ...")
subprocess.run([sys.executable, "-m", "pip", "install", "-q", *_missing], examine=True)
print("Done. (If imports fail under, restart the runtime and re-run.)n")
import numpy as np
import pandas as pd
import torch
import matplotlib.pyplot as plt
from datasets import load_dataset
from sklearn.feature_extraction.textual content import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import (accuracy_score, f1_score, roc_auc_score,
classification_report, confusion_matrix, roc_curve)
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
TrainingArguments, Trainer, DataCollatorWithPadding,
EarlyStoppingCallback, set_seed)
from peft import LoraConfig, get_peft_model, TaskType
def _disable_torchao_probe():
patched = []
strive:
import peft.import_utils as _piu
_piu.is_torchao_available = lambda: False
patched.append("peft.import_utils")
besides Exception:
go
for _name, _mod in checklist(sys.modules.gadgets()):
if _name.startswith("peft") and hasattr(_mod, "is_torchao_available"):
_mod.is_torchao_available = lambda: False
patched.append(_name)
return patched
strive:
import torchao as _tao
_v = getattr(_tao, "__version__", "?")
if tuple(int(x) for x in _v.break up(".")[:2]) < (0, 16):
print(f"[compat] torchao {_v} < 0.16 -> disabling PEFT's torchao probe: "
f"{', '.be part of(_disable_torchao_probe())}")
besides Exception:
_disable_torchao_probe()
SEED = 42
MODEL_NAME = "distilbert-base-uncased"
MAX_LEN = 256
N_TRAIN = 5000
N_EVAL = 2000
N_UNSUP = 3000
EPOCHS = 2
BATCH = 16
LR = 3e-4
FULL_RUN = False
if FULL_RUN:
N_TRAIN, N_EVAL, EPOCHS = 25000, 25000, 3
set_seed(SEED); random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print("=" * 79)
print(f"system={DEVICE} | torch={torch.__version__} | "
f"gpu={torch.cuda.get_device_name(0) if DEVICE=='cuda' else 'n/a'}")
print("=" * 79)
t0 = time.time()
uncooked = load_dataset("stanfordnlp/imdb")
print(uncooked, f"nloaded in {time.time()-t0:.1f}sn")
print("--- instance (truncated) ---")
print("label:", uncooked["train"][0]["label"], "|", uncooked["train"][0]["text"][:300], "...n")
first_labels = np.array(uncooked["train"]["label"][:5])
last_labels = np.array(uncooked["train"]["label"][-5:])
print(f"TRAP #1 - break up ordering: first 5 labels {first_labels}, "
f"final 5 labels {last_labels} -> ALWAYS shuffle earlier than subsampling.")
train_full = uncooked["train"].shuffle(seed=SEED)
test_full = uncooked["test"].shuffle(seed=SEED)
train_ds = train_full.choose(vary(min(N_TRAIN, len(train_full))))
eval_ds = test_full.choose(vary(min(N_EVAL, len(test_full))))
print(f" after shuffle+subsample: practice steadiness = "
f"{np.bincount(train_ds['label'])}, eval steadiness = {np.bincount(eval_ds['label'])}")
lens = np.array([len(t.split()) for t in train_full["text"]])
q = np.percentile(lens, [50, 75, 90, 95, 99])
print(f"nTRAP #2 - size (phrases): median={q[0]:.0f} p75={q[1]:.0f} p90={q[2]:.0f} "
f"p95={q[3]:.0f} p99={q[4]:.0f} max={lens.max()}")
print(f" ~{(lens > MAX_LEN*0.75).imply()*100:.1f}% of critiques exceed MAX_LEN={MAX_LEN} "
f"tokens (tough words->tokens issue 1.3). Section 9 measures what that prices.")
h_tr = {hashlib.md5(t.encode()).hexdigest() for t in uncooked["train"]["text"]}
h_te = {hashlib.md5(t.encode()).hexdigest() for t in uncooked["test"]["text"]}
print(f"nTRAP #3 - leakage: {len(h_tr & h_te)} actual duplicate critiques throughout "
f"practice/check; {len(uncooked['train'])-len(h_tr)} dupes inside practice itself.")
def clear(t):
return t.change("<br />", " ").change("<br/>", " ").strip()
plt.determine(figsize=(11, 3.2))
plt.subplot(1, 2, 1)
plt.hist(np.clip(lens, 0, 1000), bins=60)
plt.axvline(MAX_LEN, ls="--", coloration="okay", label=f"MAX_LEN={MAX_LEN}")
plt.title("Review size (phrases, clipped at 1000)"); plt.legend()
plt.subplot(1, 2, 2)
plt.bar(["neg", "pos"], np.bincount(uncooked["train"]["label"]))
plt.title("Train class steadiness (completely balanced)")
plt.tight_layout(); plt.present()
We configure the Colab surroundings, set up the required libraries, apply the PEFT–torchao compatibility repair, and set deterministic seeds for reproducible experiments. We load the Stanford IMDb dataset, shuffle and subsample the practice and check splits, and examine class steadiness, review-length distributions, duplicate leakage, and HTML artifacts. We additionally visualize evaluate lengths and label frequencies so we perceive the dataset construction earlier than constructing any fashions.
print("n" + "=" * 79 + "n3. TF-IDF BASELINEn" + "=" * 79)
Xtr = [clean(t) for t in train_ds["text"]]; ytr = np.array(train_ds["label"])
Xte = [clean(t) for t in eval_ds["text"]]; yte = np.array(eval_ds["label"])
t0 = time.time()
tfidf_clf = make_pipeline(
TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_features=300_000,
sublinear_tf=True, strip_accents="unicode"),
LogisticRegression(C=8.0, max_iter=2000, n_jobs=-1),
)
tfidf_clf.match(Xtr, ytr)
p_tfidf = tfidf_clf.predict_proba(Xte)[:, 1]
acc_tfidf = accuracy_score(yte, p_tfidf > 0.5)
auc_tfidf = roc_auc_score(yte, p_tfidf)
print(f"educated in {time.time()-t0:.1f}s -> acc={acc_tfidf:.4f} auc={auc_tfidf:.4f}")
vec, lr = tfidf_clf.steps[0][1], tfidf_clf.steps[1][1]
feats, coefs = np.array(vec.get_feature_names_out()), lr.coef_[0]
order = np.argsort(coefs)
print("nmost NEGATIVE n-grams:", ", ".be part of(feats[order[:12]]))
print("most POSITIVE n-grams:", ", ".be part of(feats[order[-12:]][::-1]))
print("n" + "=" * 79 + "n4. LoRA FINE-TUNINGn" + "=" * 79)
tok = AutoTokenizer.from_pretrained(MODEL_NAME)
def tokenize(batch):
return tok([clean(t) for t in batch["text"]], truncation=True, max_length=MAX_LEN)
tr_tok = (train_ds.map(tokenize, batched=True, remove_columns=["text"])
.rename_column("label", "labels"))
ev_tok = (eval_ds.map(tokenize, batched=True, remove_columns=["text"])
.rename_column("label", "labels"))
base = AutoModelForSequenceClassification.from_pretrained(
MODEL_NAME, num_labels=2,
id2label={0: "NEGATIVE", 1: "POSITIVE"},
label2id={"NEGATIVE": 0, "POSITIVE": 1},
)
lora_cfg = LoraConfig(
task_type=TaskType.SEQ_CLS,
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_lin", "v_lin"],
modules_to_save=["pre_classifier", "classifier"],
)
strive:
mannequin = get_peft_model(base, lora_cfg)
besides ImportError as e:
_disable_torchao_probe()
print(f"[compat] retrying after backend probe failure: {e}")
mannequin = get_peft_model(base, lora_cfg)
mannequin.print_trainable_parameters()
def compute_metrics(eval_pred):
logits, labels = eval_pred
probs = torch.softmax(torch.tensor(logits), dim=-1).numpy()[:, 1]
preds = (probs > 0.5).astype(int)
return {"accuracy": accuracy_score(labels, preds),
"f1_macro": f1_score(labels, preds, common="macro"),
"roc_auc": roc_auc_score(labels, probs)}
_ta = examine.signature(TrainingArguments.__init__).parameters
_eval_key = "eval_strategy" if "eval_strategy" in _ta else "evaluation_strategy"
ta_kwargs = dict(
output_dir="./imdb_lora", learning_rate=LR,
per_device_train_batch_size=BATCH, per_device_eval_batch_size=BATCH * 2,
num_train_epochs=EPOCHS, weight_decay=0.01, warmup_ratio=0.06,
logging_steps=50, save_strategy="epoch", save_total_limit=1,
load_best_model_at_end=True, metric_for_best_model="accuracy",
fp16=(DEVICE == "cuda"), report_to="none", seed=SEED,
)
ta_kwargs[_eval_key] = "epoch"
_tr = examine.signature(Trainer.__init__).parameters
_tok_key = "processing_class" if "processing_class" in _tr else "tokenizer"
coach = Trainer(
mannequin=mannequin, args=TrainingArguments(**ta_kwargs),
train_dataset=tr_tok, eval_dataset=ev_tok,
data_collator=DataCollatorWithPadding(tok),
compute_metrics=compute_metrics,
callbacks=[EarlyStoppingCallback(early_stopping_patience=2)],
**{_tok_key: tok},
)
t0 = time.time()
coach.practice()
print(f"nfine-tuned in {(time.time()-t0)/60:.1f} min")
We practice a robust TF-IDF and Logistic Regression baseline and examine probably the most influential optimistic and unfavorable n-grams to determine an interpretable reference level. We then tokenize the IMDb critiques and configure DistilBERT with LoRA adapters that replace solely a small subset of mannequin parameters whereas protecting the spine largely frozen. We use the Hugging Face Trainer with dynamic padding, early stopping, combined precision, and a number of analysis metrics to fine-tune the transformer effectively.
print("n" + "=" * 79 + "n5. EVALUATIONn" + "=" * 79)
pred_out = coach.predict(ev_tok)
p_lora = torch.softmax(torch.tensor(pred_out.predictions), dim=-1).numpy()[:, 1]
y_true = np.array(pred_out.label_ids)
yhat = (p_lora > 0.5).astype(int)
print(classification_report(y_true, yhat, target_names=["neg", "pos"], digits=4))
cm = confusion_matrix(y_true, yhat)
fig, ax = plt.subplots(1, 2, figsize=(11, 4))
ax[0].imshow(cm, cmap="Blues")
for i in vary(2):
for j in vary(2):
ax[0].textual content(j, i, cm[i, j], ha="middle", va="middle", fontsize=14)
ax[0].set_xticks([0, 1], ["pred neg", "pred pos"])
ax[0].set_yticks([0, 1], ["true neg", "true pos"]); ax[0].set_title("Confusion matrix")
for identify, p in [("TF-IDF", p_tfidf), ("DistilBERT+LoRA", p_lora)]:
fpr, tpr, _ = roc_curve(y_true, p)
ax[1].plot(fpr, tpr, label=f"{identify} (AUC={roc_auc_score(y_true, p):.4f})")
ax[1].plot([0, 1], [0, 1], "k--", lw=0.8)
ax[1].set_xlabel("FPR"); ax[1].set_ylabel("TPR"); ax[1].set_title("ROC"); ax[1].legend()
plt.tight_layout(); plt.present()
print("n" + "=" * 79 + "n6. THRESHOLD & CALIBRATIONn" + "=" * 79)
ths = np.linspace(0.05, 0.95, 91)
accs = [(y_true == (p_lora > t)).mean() for t in ths]
best_t = ths[int(np.argmax(accs))]
print(f"[email protected] = {accs[45]:.4f} | greatest threshold = {best_t:.2f} -> acc = {max(accs):.4f}")
def expected_calibration_error(probs, labels, n_bins=10):
"""ECE: |confidence - accuracy| averaged over confidence bins."""
conf = np.most(probs, 1 - probs)
right = (probs > 0.5).astype(int) == labels
bins = np.linspace(0, 1, n_bins + 1)
ece, xs, ys = 0.0, [], []
for lo, hello in zip(bins[:-1], bins[1:]):
m = (conf > lo) & (conf <= hello)
if m.sum() == 0:
proceed
ece += m.imply() * abs(conf[m].imply() - right[m].imply())
xs.append(conf[m].imply()); ys.append(right[m].imply())
return ece, np.array(xs), np.array(ys)
ece, cx, cy = expected_calibration_error(p_lora, y_true)
print(f"Expected Calibration Error = {ece:.4f} (0 = completely calibrated)")
plt.determine(figsize=(9, 3.2))
plt.subplot(1, 2, 1); plt.plot(ths, accs); plt.axvline(best_t, ls="--", coloration="r")
plt.xlabel("threshold"); plt.ylabel("accuracy"); plt.title("Threshold sweep")
plt.subplot(1, 2, 2); plt.plot([0.5, 1], [0.5, 1], "k--", lw=0.8)
plt.plot(cx, cy, "o-"); plt.xlabel("imply confidence"); plt.ylabel("empirical accuracy")
plt.title(f"Reliability diagram (ECE={ece:.3f})")
plt.tight_layout(); plt.present()
We consider the fine-tuned DistilBERT-LoRA mannequin utilizing classification metrics, a confusion matrix, and ROC curves whereas straight evaluating its ROC-AUC efficiency with the TF-IDF baseline. We sweep classification thresholds to find out whether or not the default likelihood cutoff of 0.5 offers the most effective accuracy on our analysis set. We additionally calculate Expected Calibration Error and assemble a reliability diagram to measure how carefully the mannequin’s predicted confidence corresponds to its precise correctness.
print("n" + "=" * 79 + "n7. ERROR ANALYSISn" + "=" * 79)
err = pd.DataFrame({
"textual content": eval_ds["text"], "y": y_true, "p_pos": p_lora,
"n_words": [len(t.split()) for t in eval_ds["text"]],
})
err["pred"] = (err.p_pos > 0.5).astype(int)
err["correct"] = err.pred == err.y
err["confidence"] = np.most(err.p_pos, 1 - err.p_pos)
print("--- 3 most CONFIDENT errors (the place the mannequin is confidently incorrect) ---")
for _, r in err[~err.correct].nlargest(3, "confidence").iterrows():
print(f"n[true={'pos' if r.y else 'neg'} pred={'pos' if r.pred else 'neg'} "
f"conf={r.confidence:.3f} words={r.n_words}]")
print(clear(r.textual content)[:400].change("n", " "), "...")
err["bucket"] = pd.qcut(err.n_words, 4, labels=["short", "med", "long", "v.long"])
by_len = err.groupby("bucket", noticed=True).agg(acc=("right", "imply"), n=("right", "dimension"))
print("n--- accuracy by evaluate size (truncation hurts lengthy critiques) ---")
print(by_len.to_string())
print("n" + "=" * 79 + "n8. OCCLUSION SALIENCYn" + "=" * 79)
infer_model = mannequin.merge_and_unload()
infer_model.to(DEVICE).eval()
@torch.no_grad()
def predict_proba(texts, bs=64):
out = []
for i in vary(0, len(texts), bs):
enc = tok([clean(t) for t in texts[i:i + bs]], truncation=True,
max_length=MAX_LEN, padding=True, return_tensors="pt").to(DEVICE)
out.append(torch.softmax(infer_model(**enc).logits, dim=-1)[:, 1].cpu().numpy())
return np.concatenate(out)
def occlusion(textual content, max_words=60):
phrases = clear(textual content).break up()[:max_words]
base = predict_proba([" ".join(words)])[0]
variants = [" ".join(words[:i] + phrases[i + 1:]) for i in vary(len(phrases))]
dropped = predict_proba(variants)
return phrases, base - dropped, base
pattern = err[err.correct].nlargest(1, "confidence").iloc[0]
phrases, contrib, base_p = occlusion(pattern.textual content)
print(f"P(optimistic) for the complete excerpt = {base_p:.3f} "
f"(true label = {'pos' if pattern.y else 'neg'})n")
high = np.argsort(np.abs(contrib))[-15:]
plt.determine(figsize=(7, 5))
plt.barh(vary(len(high)), contrib[top],
coloration=["tab:green" if contrib[i] > 0 else "tab:crimson" for i in high])
plt.yticks(vary(len(high)), [words[i] for i in high])
plt.xlabel("Δ P(optimistic) when the phrase is eliminated")
plt.title("Occlusion saliency — inexperienced pushes POSITIVE, crimson pushes NEGATIVE")
plt.tight_layout(); plt.present()
print("n" + "=" * 79 + "n9. HEAD vs TAIL TRUNCATIONn" + "=" * 79)
probe = err.nlargest(600, "n_words")
W = 180
head_txt = [" ".join(clean(t).split()[:W]) for t in probe.textual content]
tail_txt = [" ".join(clean(t).split()[-W:]) for t in probe.textual content]
yp = probe.y.values
acc_head = ((predict_proba(head_txt) > 0.5).astype(int) == yp).imply()
acc_tail = ((predict_proba(tail_txt) > 0.5).astype(int) == yp).imply()
print(f"on the {len(probe)} longest critiques, utilizing solely {W} phrases:")
print(f" first {W} phrases -> acc {acc_head:.4f}")
print(f" final {W} phrases -> acc {acc_tail:.4f}")
print(" Practical takeaway: if the tail wins, feed head+tail to the mannequin or "
"increase MAX_LEN, relatively than blindly truncating from the left.")
We look at the mannequin’s most assured incorrect predictions and group critiques by size to establish truncation-related failure patterns and tough examples. We merge the LoRA adapters into the underlying mannequin and apply leave-one-word-out occlusion to estimate which phrases push particular person predictions towards optimistic or unfavorable sentiment. We then examine predictions primarily based on the start and ending parts of lengthy critiques to find out the place the strongest sentiment info resides.
print("n" + "=" * 79 + "n10. PSEUDO-LABELLINGn" + "=" * 79)
unsup = uncooked["unsupervised"].shuffle(seed=SEED).choose(vary(N_UNSUP))
p_uns = predict_proba(unsup["text"])
maintain = (p_uns > 0.95) | (p_uns < 0.05)
pl_texts = [clean(t) for t, k in zip(unsup["text"], maintain) if okay]
pl_labels = (p_uns[keep] > 0.5).astype(int)
print(f"stored {maintain.sum()}/{N_UNSUP} pseudo-labels at conf>0.95 "
f"(steadiness: {np.bincount(pl_labels)})")
aug = make_pipeline(
TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_features=300_000,
sublinear_tf=True, strip_accents="unicode"),
LogisticRegression(C=8.0, max_iter=2000, n_jobs=-1),
).match(Xtr + pl_texts, np.concatenate([ytr, pl_labels]))
acc_aug = accuracy_score(yte, aug.predict(Xte))
print(f"TF-IDF baseline : {acc_tfidf:.4f}")
print(f"TF-IDF + pseudo-labels: {acc_aug:.4f} (Δ {acc_aug-acc_tfidf:+.4f})")
print("Caveat: beneficial properties are bounded by the instructor. Self-training additionally amplifies "
"the instructor's biases — all the time validate on clear, held-out knowledge.")
print("n" + "=" * 79 + "n11. SAVE & INFERn" + "=" * 79)
SAVE_DIR = "./imdb-distilbert-lora-merged"
infer_model.save_pretrained(SAVE_DIR); tok.save_pretrained(SAVE_DIR)
print(f"saved merged mannequin to {SAVE_DIR}/ (load with "
f"AutoModelForSequenceClassification.from_pretrained('{SAVE_DIR}'))")
demos = [
"A masterclass in tension. The final act left the whole theatre silent.",
"Two hours I will never get back. Wooden acting, incoherent plot.",
"It's not the disaster the trailer promised, but it never really lands either.",
]
for d, p in zip(demos, predict_proba(demos)):
print(f" P(pos)={p:.3f} -> {'POSITIVE' if p > 0.5 else 'NEGATIVE'} | {d}")
print("n" + "=" * 79)
print(f"SUMMARY (n_train={N_TRAIN}, n_eval={N_EVAL}, max_len={MAX_LEN})")
print("=" * 79)
print(pd.DataFrame([
{"model": "TF-IDF + LogReg", "accuracy": acc_tfidf, "roc_auc": auc_tfidf},
{"model": "TF-IDF + pseudo-labels", "accuracy": acc_aug, "roc_auc": float("nan")},
{"model": "DistilBERT + LoRA", "accuracy": accuracy_score(y_true, yhat),
"roc_auc": roc_auc_score(y_true, p_lora)},
]).to_string(index=False))
print("""
NEXT EXPERIMENTS
- Set FULL_RUN = True for the true 25k/25k benchmark (~40 min on a T4).
- Swap MODEL_NAME to 'roberta-base' (target_modules=['query','value']) or
'answerdotai/ModernBERT-base' for an 8k context window — no truncation.
- Head+tail truncation: first 128 + final 128 tokens, motivated by part 9.
- Ablate LoRA rank r in {4, 8, 16, 64} and plot accuracy vs trainable params.
- Replace the pseudo-label instructor with an ensemble and iterate self-training.
- Push to the Hub: huggingface_hub.login() then infer_model.push_to_hub(...).
""")
We use the fine-tuned transformer to generate high-confidence pseudo-labels for examples from IMDb’s unlabeled break up and add these examples to the TF-IDF coaching corpus. We examine the augmented classifier towards the unique baseline to measure whether or not semi-supervised self-training improves predictive accuracy. Finally, we save the merged DistilBERT mannequin and tokenizer, run sentiment inference on customized critiques, and summarize the efficiency of all fashions developed all through the tutorial.
In conclusion, we developed a rigorous sentiment classification pipeline that goes nicely past merely fine-tuning a transformer and reporting accuracy. We established a aggressive TF-IDF baseline, practice DistilBERT effectively with LoRA, and consider each predictive high quality and likelihood reliability whereas figuring out how evaluate size, truncation, and extremely assured errors affect real-world efficiency. We additionally interpreted particular person predictions by occlusion-based saliency, examined whether or not sentiment info is concentrated close to the start or finish of lengthy critiques, and prolonged supervised studying with high-confidence pseudo-labels from the unlabeled dataset.
Check out the FULL CODES here. Also, be happy to comply with us on Twitter and don’t neglect to hitch our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to accomplice with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so forth.? Connect with us
The publish IMDb Sentiment Analysis with DistilBERT LoRA, TF-IDF Baselines, Calibration, Interpretability, Robustness Testing, and Semi-Supervised Learning appeared first on MarkTechPost.
