Estimated reading time: 12 minutes
Difficulty: Intermediate
Prerequisites: Getting Started with GASearchCV
Early Stopping and Callbacks for Hyperparameter Search
Callbacks hook into the search loop and run after each generation. They can stop the search early, log progress, save checkpoints, or trigger any custom action.
Callbacks are evaluated in three phases:
on_start— once before generation 0on_step— after every generation. ReturnTrueto stop the searchon_end— once after the final generation (or after a stop)
Prerequisites
- Completed Basic Usage
Using Callbacks
Pass a single callback or a list to fit:
from sklearn_genetic.callbacks import ConsecutiveStopping, ProgressBar
search.fit(X_train, y_train, callbacks=[
ProgressBar(),
ConsecutiveStopping(generations=5, metric="fitness_best"),
])If any callback signals to stop, the search ends at the end of that generation.
Built-in Callbacks
ProgressBar
Displays a tqdm progress bar showing generation progress.
from sklearn_genetic.callbacks import ProgressBar
callback = ProgressBar()No parameters required. The bar length equals the configured number of generations.
ConsecutiveStopping
Compares the current value with each of the previous N values and stops if it improves on none of them. Score metrics follow the search criterion: lower values improve a minimizing search and higher values improve a maximizing search. For fitness_std, an increase in dispersion counts as improvement in either direction.
from sklearn_genetic.callbacks import ConsecutiveStopping
callback = ConsecutiveStopping(generations=5, metric="fitness")| Parameter | Type | Default | Description |
|---|---|---|---|
generations | int | — | Stop if the metric has not improved in this many consecutive generations |
metric | str | "fitness" | History column to track: "fitness", "fitness_best", "fitness_max", "fitness_min", "fitness_std" |
ThresholdStopping
Stops once the metric reaches or exceeds a target value.
from sklearn_genetic.callbacks import ThresholdStopping
callback = ThresholdStopping(threshold=0.98, metric="fitness_max")| Parameter | Type | Default | Description |
|---|---|---|---|
threshold | float | — | Stop when the metric ≥ this value |
metric | str | "fitness" | History column to watch |
DeltaThreshold
Stops when the spread (max − min) across the last N generations falls below a threshold — useful for detecting convergence.
from sklearn_genetic.callbacks import DeltaThreshold
callback = DeltaThreshold(threshold=0.001, generations=5, metric="fitness_min")| Parameter | Type | Default | Description |
|---|---|---|---|
threshold | float | — | Stop when max(metric[-N:]) - min(metric[-N:]) ≤ threshold |
generations | int | 2 | Window size for the comparison |
metric | str | "fitness" | History column to watch |
TimerStopping
Stops after a wall-clock time limit.
from sklearn_genetic.callbacks import TimerStopping
callback = TimerStopping(total_seconds=300) # stop after 5 minutes| Parameter | Type | Description |
|---|---|---|
total_seconds | float | Maximum elapsed seconds before starting another candidate batch; an in-flight batch can finish after the limit |
Pass it to fit to cap a long search by wall-clock time — handy for notebook and local experiments where the search space may be larger than expected:
from sklearn_genetic.callbacks import TimerStopping
callbacks = [TimerStopping(total_seconds=60)]
search.fit(X_train, y_train, callbacks=callbacks)The search checks the budget between candidate batches, including batches within a generation. Final selection and refit can extend the total fit time.
LogbookSaver
Saves the generation logbook to a file after each generation. Useful for long runs — if the search is interrupted, you can inspect progress so far.
from sklearn_genetic.callbacks import LogbookSaver
callback = LogbookSaver(checkpoint_path="./logbook.pkl")Restore with joblib:
from joblib import load
logbook = load("./logbook.pkl")
print(logbook)ModelCheckpoint
Saves the search configuration, generation history and runtime state, including cached candidate scores, archived elites and random generator states. Each successful write replaces the previous snapshot. See the checkpoint recipe to resume a search.
from sklearn_genetic.callbacks import ModelCheckpoint
callback = ModelCheckpoint(checkpoint_path="./checkpoint.pkl")Inspect the saved state:
import pickle
with open("./checkpoint.pkl", "rb") as f:
checkpoint = pickle.load(f)
estimator_state = checkpoint["estimator_state"]
logbook = checkpoint["logbook"]Live training plot
Use the plotting extra to watch best and average fitness as each generation finishes. The same callback works for GASearchCV and GAFeatureSelectionCV.
from sklearn_genetic.plots import LivePlot
callback = LivePlot(save_path="./training-progress.png")
search.fit(X, y, callbacks=callback)Install with pip install sklearn-genetic-opt[plot]. The Matplotlib window updates during fitting when a graphical backend is available. save_path rewrites the current PNG after each generation, including in headless runs. The callback keeps the figure open for inspection after fitting.
For a custom callback, use plot_generation_metrics(logbook) to draw the same generation traces from the logbook received by on_step. After fitting, use plot_search_overview to inspect convergence, diversity, optimizer events and the best candidates or selected features.
The legacy TensorBoard callback remains importable for existing users who install TensorFlow separately. TensorFlow is no longer included in any extra.
Full Example
This example uses ThresholdStopping and DeltaThreshold together — it stops when accuracy exceeds 0.98 OR when the improvement between consecutive generations drops below 0.001:
from sklearn.datasets import load_digits
from sklearn.metrics import accuracy_score
from sklearn.model_selection import StratifiedKFold, train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn_genetic import EvolutionConfig, GASearchCV, PopulationConfig, RuntimeConfig
from sklearn_genetic.callbacks import DeltaThreshold, ThresholdStopping
from sklearn_genetic.space import Categorical, Continuous, Integer
data = load_digits()
X, y = data["data"], data["target"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
param_grid = {
"min_weight_fraction_leaf": Continuous(0, 0.5),
"criterion": Categorical(["gini", "entropy"]),
"max_depth": Integer(2, 20),
"max_leaf_nodes": Integer(2, 30),
}
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
evolved_estimator = GASearchCV(
estimator=DecisionTreeClassifier(),
cv=cv,
scoring="accuracy",
param_grid=param_grid,
evolution_config=EvolutionConfig(
population_size=16,
generations=30,
tournament_size=3,
elitism=True,
crossover_probability=0.9,
mutation_probability=0.05,
),
population_config=PopulationConfig(initializer="smart"),
runtime_config=RuntimeConfig(n_jobs=-1, verbose=True),
)
evolved_estimator.fit(X_train, y_train, callbacks=[
ThresholdStopping(threshold=0.98, metric="fitness_max"),
DeltaThreshold(threshold=0.001, metric="fitness"),
])
y_predict_ga = evolved_estimator.predict(X_test)
print(evolved_estimator.best_params_)
print("Accuracy:", accuracy_score(y_test, y_predict_ga))Tips & Gotchas
- Callbacks see the same metric names that appear in the generation log:
fitness,fitness_std,fitness_best,fitness_max,fitness_min. ConsecutiveStoppingwatchesfitnessby default (mean population fitness). Usemetric="fitness_best"if you want to stop when the overall best score stops improving.TimerStoppingstops scheduling new batches after the limit; an in-flight CV batch and final selection/refit can still complete.- When multiple callbacks are combined, any single one returning
Truestops the search.
See Also
- Writing Custom Callbacks — build your own stopping logic
- Reproducibility & Checkpointing — use LogbookSaver and ModelCheckpoint
- Cross-Validation in Hyperparameter Search — understand the metrics callbacks watch
- Advanced Optimizer Control — diversity control and stagnation recovery
- API: Callbacks — full callback reference