A Coding Guide to Implement Advanced Hyperparameter Optimization with Optuna using Pruning Multi-Objective Search, Early Stopping, and Deep Visual Analysis

In this tutorial, we implement a sophisticated Optuna workflow that systematically explores pruning, multi-objective optimization, customized callbacks, and wealthy visualization. Through every snippet, we see how Optuna helps us form smarter search areas, pace up experiments, and extract insights that information mannequin enchancment. We work with actual datasets, design environment friendly search methods, and analyze trial habits in a approach that feels interactive, quick, and intuitive. Check out the FULL CODES here.

Copy Code

import optuna
from optuna.pruners import MedianPruner
from optuna.samplers import TPESampler
import numpy as np
from sklearn.datasets import load_breast_cancer, load_diabetes
from sklearn.model_selection import cross_val_score, KFold
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
import matplotlib.pyplot as plt


def objective_with_pruning(trial):
   X, y = load_breast_cancer(return_X_y=True)
   params = {
       'n_estimators': trial.suggest_int('n_estimators', 50, 200),
       'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),
       'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 10),
       'subsample': trial.suggest_float('subsample', 0.6, 1.0),
       'max_features': trial.suggest_categorical('max_features', ['sqrt', 'log2', None]),
   }
   mannequin = GradientBoostingClassifier(**params, random_state=42)
   kf = KFold(n_splits=3, shuffle=True, random_state=42)
   scores = []
   for fold, (train_idx, val_idx) in enumerate(kf.cut up(X)):
       X_train, X_val = X[train_idx], X[val_idx]
       y_train, y_val = y[train_idx], y[val_idx]
       mannequin.match(X_train, y_train)
       rating = mannequin.rating(X_val, y_val)
       scores.append(rating)
       trial.report(np.imply(scores), fold)
       if trial.should_prune():
           elevate optuna.TrialPruned()
   return np.imply(scores)


study1 = optuna.create_study(
   course='maximize',
   sampler=TPESampler(seed=42),
   pruner=MedianPruner(n_startup_trials=5, n_warmup_steps=1)
)
study1.optimize(objective_with_pruning, n_trials=30, show_progress_bar=True)


print(study1.best_value, study1.best_params)

We arrange all of the core imports and outline our first goal perform with pruning. As we run the Gradient Boosting optimization, we observe Optuna actively pruning weaker trials and guiding us towards stronger hyperparameter areas. We really feel the optimization turning into sooner and extra clever because the examine progresses. Check out the FULL CODES here.

Copy Code

def multi_objective(trial):
   X, y = load_breast_cancer(return_X_y=True)
   n_estimators = trial.suggest_int('n_estimators', 10, 200)
   max_depth = trial.suggest_int('max_depth', 2, 20)
   min_samples_split = trial.suggest_int('min_samples_split', 2, 20)
   mannequin = RandomForestClassifier(
       n_estimators=n_estimators,
       max_depth=max_depth,
       min_samples_split=min_samples_split,
       random_state=42,
       n_jobs=-1
   )
   accuracy = cross_val_score(mannequin, X, y, cv=3, scoring='accuracy', n_jobs=-1).imply()
   complexity = n_estimators * max_depth
   return accuracy, complexity


study2 = optuna.create_study(
   instructions=['maximize', 'minimize'],
   sampler=TPESampler(seed=42)
)
study2.optimize(multi_objective, n_trials=50, show_progress_bar=True)


for t in study2.best_trials[:3]:
   print(t.quantity, t.values)

We shift to a multi-objective setup the place we optimize each accuracy and mannequin complexity. As we discover completely different configurations, we see how Optuna routinely builds a Pareto entrance, letting us evaluate trade-offs slightly than chasing a single rating. This gives us with a deeper understanding of how competing metrics work together with each other. Check out the FULL CODES here.

Copy Code

class EarlyStoppingCallback:
   def __init__(self, early_stopping_rounds=10, course='maximize'):
       self.early_stopping_rounds = early_stopping_rounds
       self.course = course
       self.best_value = float('-inf') if course == 'maximize' else float('inf')
       self.counter = 0
   def __call__(self, examine, trial):
       if trial.state != optuna.trial.TrialState.COMPLETE:
           return
       v = trial.worth
       if self.course == 'maximize':
           if v > self.best_value:
               self.best_value, self.counter = v, 0
           else:
               self.counter += 1
       else:
           if v < self.best_value:
               self.best_value, self.counter = v, 0
           else:
               self.counter += 1
       if self.counter >= self.early_stopping_rounds:
           examine.cease()


def objective_regression(trial):
   X, y = load_diabetes(return_X_y=True)
   alpha = trial.suggest_float('alpha', 1e-3, 10.0, log=True)
   max_iter = trial.suggest_int('max_iter', 100, 2000)
   from sklearn.linear_model import Ridge
   mannequin = Ridge(alpha=alpha, max_iter=max_iter, random_state=42)
   rating = cross_val_score(mannequin, X, y, cv=5, scoring='neg_mean_squared_error', n_jobs=-1).imply()
   return -score


early_stopping = EarlyStoppingCallback(early_stopping_rounds=15, course='decrease')
study3 = optuna.create_study(course='decrease', sampler=TPESampler(seed=42))
study3.optimize(objective_regression, n_trials=100, callbacks=[early_stopping], show_progress_bar=True)


print(study3.best_value, study3.best_params)

We introduce our personal early-stopping callback and join it to a regression goal. We observe how the examine stops itself when progress stalls, saving time and compute. This makes us really feel the ability of customizing Optuna’s move to match real-world coaching habits. Check out the FULL CODES here.

Copy Code

fig, axes = plt.subplots(2, 2, figsize=(14, 10))


ax = axes[0, 0]
values = [t.value for t in study1.trials if t.value is not None]
ax.plot(values, marker='o', markersize=3)
ax.axhline(y=study1.best_value, coloration='r', linestyle='--')
ax.set_title('Study 1 History')


ax = axes[0, 1]
significance = optuna.significance.get_param_importances(study1)
params = record(significance.keys())[:5]
vals = [importance[p] for p in params]
ax.barh(params, vals)
ax.set_title('Param Importance')


ax = axes[1, 0]
for t in study2.trials:
   if t.values:
       ax.scatter(t.values[0], t.values[1], alpha=0.3)
for t in study2.best_trials:
   ax.scatter(t.values[0], t.values[1], c='crimson', s=90)
ax.set_title('Pareto Front')


ax = axes[1, 1]
pairs = [(t.params.get('max_depth', 0), t.value) for t in study1.trials if t.value]
Xv, Yv = zip(*pairs) if pairs else ([], [])
ax.scatter(Xv, Yv, alpha=0.6)
ax.set_title('max_depth vs Accuracy')


plt.tight_layout()
plt.savefig('optuna_analysis.png', dpi=150)
plt.present()

We visualize every little thing we have now run thus far. We generate optimization curves, parameter importances, Pareto fronts, and parameter-metric relationships, which assist us interpret your complete experiment at a look. As we study the plots, we achieve perception into the place the mannequin performs finest and why. Check out the FULL CODES here.

Copy Code

p1 = len([t for t in study1.trials if t.state == optuna.trial.TrialState.PRUNED])
print("Study 1 Best Accuracy:", study1.best_value)
print("Study 1 Pruned %:", p1 / len(study1.trials) * 100)


print("Study 2 Pareto Solutions:", len(study2.best_trials))


print("Study 3 Best MSE:", study3.best_value)
print("Study 3 Trials:", len(study3.trials))

We summarize key outcomes from all three research, reviewing accuracy, pruning effectivity, Pareto options, and regression MSE. Seeing every little thing condensed into just a few traces offers us a transparent sense of our optimization journey. We now really feel assured in extending and adapting this setup for extra superior experiments.

In conclusion, we have now gained an understanding of how to construct highly effective hyperparameter optimization pipelines that reach far past easy single-metric tuning. We mix pruning, Pareto optimization, early stopping, and evaluation instruments to kind a whole and versatile workflow. We now really feel assured in adapting this template for any future ML or DL mannequin we would like to optimize, realizing we now have a transparent and sensible blueprint for high-quality Optuna-based experimentation.

Check out the FULL CODES here. Feel free to try our GitHub Page for Tutorials, Codes and Notebooks. Also, be happy to observe us on Twitter and don’t neglect to be part of our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

The publish A Coding Guide to Implement Advanced Hyperparameter Optimization with Optuna using Pruning Multi-Objective Search, Early Stopping, and Deep Visual Analysis appeared first on MarkTechPost.