|

ClawHub Security Signals: A Coding Guide to End-to-End Security Signal Analysis and Verdict Classification on the AI Skills Dataset

ByRicardo

In this tutorial, we use the ClawHub Security Signals dataset to look at how completely different safety scanners assess AI abilities and associated information. We load the dataset straight from the Hugging Face Parquet conversion to keep away from compatibility points with newer dataset metadata, then examine the primary columns, verdict distribution, scanner outputs, and severity labels. After exploring scanner disagreement and overlap patterns, we construct a sensible machine studying pipeline that mixes SKILL.md textual content with numerical scanner indicators to predict the last ClawScan verdict. It provides us an entire workflow for loading, analyzing, visualizing, and modeling safety sign information in a Colab-ready atmosphere.

Setting Up the Colab Environment and Imports for Security Signal Analysis

Copy Code
!pip -q set up -U "huggingface_hub>=0.23" pyarrow scikit-learn pandas numpy matplotlib seaborn
import warnings, numpy as np, pandas as pd
warnings.filterwarnings("ignore")
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_theme(model="whitegrid")
from huggingface_hub import HfApi, hf_hub_download
from sklearn.feature_extraction.textual content import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, FunctionTransformer
from sklearn.impute import SimpleImputer
from sklearn.metrics import (classification_report, confusion_matrix,
                            cohen_kappa_score, jaccard_score)
SAMPLE_SIZE = 20000
RANDOM_STATE = 42

We set up all the required libraries and import the primary packages wanted for information loading, evaluation, visualization, and machine studying. We additionally configure warnings and set the plotting model to maintain the pocket book output clear and readable. Finally, we outline the pattern measurement and random seed to make the experiment managed and reproducible.

Loading the ClawHub Security Signals Dataset from the Hugging Face Parquet Conversion

Copy Code
REPO = "OpenClaw/clawhub-security-signals"
REV  = "refs/convert/parquet"
print("Listing Parquet information on the Hub...")
api = HfApi()
all_files = api.list_repo_files(REPO, repo_type="dataset", revision=REV)
parquet_files = [f for f in all_files if f.endswith(".parquet")]
def load_split(break up):
   """Download + concat all Parquet shards for a given break up right into a DataFrame."""
   shards = [f for f in parquet_files
             if f.split("/")[-2] == break up]
   if not shards:
       increase ValueError(f"No parquet information for break up '{break up}'. Found: {parquet_files[:5]}")
   frames = []
   for f in shards:
       native = hf_hub_download(REPO, f, repo_type="dataset", revision=REV)
       frames.append(pd.read_parquet(native))
   return pd.concat(frames, ignore_index=True)
print("Downloading practice + take a look at splits (first run pulls the information)...")
train_df = load_split("practice")
test_df  = load_split("take a look at")
if SAMPLE_SIZE:
   train_df = train_df.pattern(min(SAMPLE_SIZE, len(train_df)),
                              random_state=RANDOM_STATE).reset_index(drop=True)
print(f"nTrain rows in use: {len(train_df):,} | Test rows: {len(test_df):,}")
print("Columns:", record(train_df.columns))

We join to the Hugging Face dataset repository and record the out there Parquet information from the transformed dataset department. We create a helper perform to obtain and mix the Parquet shards for every break up right into a single pandas DataFrame. We then load the practice and take a look at splits, optionally pattern the coaching information, and print the dataset measurement and column names.

Exploring Verdict Distribution and Scanner Agreement with Jaccard and Cohen’s Kappa

Copy Code
print("n=== ClawScan verdict distribution (practice) ===")
print(train_df["clawscan_verdict"].value_counts(normalize=True).mul(100).spherical(2))
print("n=== SkillSpector severity distribution ===")
print(train_df["skillspector_severity"].value_counts(dropna=False))
pattern = train_df.iloc[0]
print(f"nExample ability: {pattern['skill_slug']} (v{pattern['skill_version']})")
print(f"Verdict: {pattern['clawscan_verdict']} | Summary: {pattern['clawscan_summary']}")
print("SKILL.md (first 400 chars):n", str(pattern["skill_md_content"])[:400])
POSITIVE = {"suspicious", "malicious"}
def is_pos(sequence):
   return sequence.fillna("").isin(POSITIVE).astype(int)
an = train_df.copy()
an["vt_pos"]     = is_pos(an["virustotal_status"])
an["static_pos"] = is_pos(an["static_status"])
an["spec_pos"]   = is_pos(an["skillspector_status"])
print("n=== Scanner optimistic charges ===")
for col, identify in [("vt_pos","VirusTotal"),("static_pos","Static"),("spec_pos","SkillSpector")]:
   print(f"  {identify:12s}: {an[col].imply()*100:5.2f}% optimistic")
def sample(r):
   tags = []
   if r.vt_pos: tags.append("VT")
   if r.static_pos: tags.append("Static")
   if r.spec_pos: tags.append("SkillSpector")
   return "None" if not tags else " + ".be part of(tags)
an["pattern"] = an.apply(sample, axis=1)
print("n=== Positive-signal overlap patterns ===")
print(an["pattern"].value_counts(normalize=True).mul(100).spherical(2))
print("n=== Pairwise settlement (low = scanners examine completely different surfaces) ===")
pairs = [("vt_pos","static_pos","VT vs Static"),
        ("vt_pos","spec_pos","VT vs SkillSpector"),
        ("static_pos","spec_pos","Static vs SkillSpector")]
for a, b, label in pairs:
   j = jaccard_score(an[a], an[b], zero_division=0)
   ok = cohen_kappa_score(an[a], an[b])
   print(f"  {label:26s} Jaccard={j:.3f}  Cohen's kappa={ok:.3f}")

We carry out the primary exploratory evaluation on the ClawHub Security Signals dataset. We examine verdict distributions, severity labels, instance ability metadata, and the starting of a SKILL.md file to perceive the information construction. We additionally convert scanner outputs into optimistic flags and evaluate VirusTotal, static evaluation, and SkillSpector via optimistic charges, overlap patterns, Jaccard scores, and Cohen’s kappa.

Visualizing Verdict Distribution, Scanner Positive Rates, and Overlap Patterns

Copy Code
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
order = ["clean","suspicious","malicious"]
sns.countplot(information=train_df, x="clawscan_verdict", order=order, ax=axes[0,0], palette="viridis")
axes[0,0].set_title("ClawScan verdict distribution"); axes[0,0].set_yscale("log")
charges = {"VirusTotal":an["vt_pos"].imply(), "Static":an["static_pos"].imply(),
        "SkillSpector":an["spec_pos"].imply()}
axes[0,1].bar(charges.keys(), [v*100 for v in rates.values()], shade="#d95f02")
axes[0,1].set_title("Scanner optimistic charge (%)"); axes[0,1].set_ylabel("% flagged")
computer = an["pattern"].value_counts()
axes[1,0].barh(computer.index, computer.values, shade="#7570b3")
axes[1,0].set_title("Positive-signal overlap patterns"); axes[1,0].invert_yaxis()
sns.boxplot(information=train_df, x="clawscan_verdict", y="skillspector_score",
           order=order, ax=axes[1,1], palette="viridis")
axes[1,1].set_title("SkillSpector rating by verdict")
plt.tight_layout(); plt.present()

We create visualizations to make the dataset patterns simpler to perceive. We plot the ClawScan verdict distribution, scanner optimistic charges, positive-signal overlap patterns, and SkillSpector rating variations throughout verdict classes. These charts assist us rapidly see class imbalance, scanner conduct, and the relationship between numerical safety scores and last verdicts.

Building a Logistic Regression Pipeline on SKILL.md Text and Scanner Signals to Predict ClawScan Verdicts

Copy Code
TEXT_COL = "skill_md_content"
NUM_COLS = ["skillspector_score", "static_finding_count",
           "skillspector_issue_count", "virustotal_malicious_count"]
TARGET   = "clawscan_verdict"
def prep(df):
   out = df.copy()
   out[TEXT_COL] = out[TEXT_COL].fillna("").astype(str).str.slice(0, 6000)
   for c in NUM_COLS:
       out[c] = pd.to_numeric(out[c], errors="coerce")
   return out
train_p, test_p = prep(train_df), prep(test_df)
get_text = FunctionTransformer(lambda X: X[TEXT_COL].values, validate=False)
text_pipe = Pipeline([
   ("select", get_text),
   ("tfidf", TfidfVectorizer(max_features=20000, ngram_range=(1,2),
                             min_df=3, sublinear_tf=True)),
])
num_pipe = Pipeline([
   ("impute", SimpleImputer(strategy="constant", fill_value=0)),
   ("scale", StandardScaler()),
])
options = ColumnTransformer([
   ("text", text_pipe, [TEXT_COL]),
   ("num", num_pipe, NUM_COLS),
])
clf = Pipeline([
   ("features", features),
   ("model", LogisticRegression(max_iter=2000, C=4.0,
                                class_weight="balanced",
                                multi_class="multinomial")),
])
print("nTraining classifier (SKILL.md textual content + scanner numbers -> verdict)...")
clf.match(train_p[[TEXT_COL] + NUM_COLS], train_p[TARGET])
pred = clf.predict(test_p[[TEXT_COL] + NUM_COLS])
print("n=== Test-set classification report ===")
print(classification_report(test_p[TARGET], pred, digits=3))
cm = confusion_matrix(test_p[TARGET], pred, labels=order)
plt.determine(figsize=(6,5))
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", xticklabels=order, yticklabels=order)
plt.title("Confusion matrix (take a look at break up)"); plt.xlabel("Predicted"); plt.ylabel("Actual"); plt.present()
test_out = test_p[["skill_slug", TARGET, "clawscan_summary"]].copy()
test_out["pred"] = pred
errors = test_out[test_out[TARGET] != test_out["pred"]].head(8)
print("n=== Sample misclassifications ===")
for _, r in errors.iterrows():
   print(f"- {r['skill_slug']:35s} true={r[TARGET]:10s} pred={r['pred']:10s}")
print("nDone. Set SAMPLE_SIZE=None for the full dataset.")

We put together the textual content and numerical options for coaching a machine studying classifier. We construct a pipeline that makes use of TF-IDF options from SKILL.md content material, together with scanner-related numeric fields, and then trains a balanced logistic regression mannequin to predict the ClawScan verdict. We consider the mannequin utilizing a classification report, a confusion matrix, and pattern misclassifications to perceive the place the classifier performs effectively and the place it fails.

Conclusion

In conclusion, we accomplished an end-to-end evaluation of the ClawHub Security Signals dataset, from strong information loading to test-set analysis of a verdict classifier. We examined how VirusTotal, static evaluation, and SkillSpector indicators differ, visualized their patterns, and used each textual and numerical options to practice a balanced logistic regression mannequin. This workflow helps us perceive how safety verdicts are distributed, and additionally how a number of scanner indicators might be mixed right into a easy predictive system. We can prolong this additional by utilizing the full dataset, attempting stronger textual content fashions, or including deeper characteristic engineering round scanner summaries and ability metadata.

Check out the Full Codes with Notebook. Also, be at liberty to comply with us on Twitter and don’t overlook to be part of our 150k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to companion with us for selling your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar and so forth.? Connect with us

The put up ClawHub Security Signals: A Coding Guide to End-to-End Security Signal Analysis and Verdict Classification on the AI Skills Dataset appeared first on MarkTechPost.

Similar Posts