Skip to content

Estimated reading time: 12 minutes
Difficulty: Intermediate
Prerequisites: Getting Started with GASearchCV

Early Stopping and Callbacks for Hyperparameter Search ​

Callbacks hook into the search loop and run after each generation. They can stop the search early, log progress, save checkpoints, or trigger any custom action.

Callbacks are evaluated in three phases:

  1. on_start — once before generation 0
  2. on_step — after every generation. Return True to stop the search
  3. on_end — once after the final generation (or after a stop)

Prerequisites ​

Using Callbacks ​

Pass a single callback or a list to fit:

python
from sklearn_genetic.callbacks import ConsecutiveStopping, ProgressBar

search.fit(X_train, y_train, callbacks=[
    ProgressBar(),
    ConsecutiveStopping(generations=5, metric="fitness_best"),
])

If any callback signals to stop, the search ends at the end of that generation.

Built-in Callbacks ​

ProgressBar ​

Displays a tqdm progress bar showing generation progress.

python
from sklearn_genetic.callbacks import ProgressBar

callback = ProgressBar()

No parameters required. The bar length equals the configured number of generations.


ConsecutiveStopping ​

Compares the current value with each of the previous N values and stops if it improves on none of them. Score metrics follow the search criterion: lower values improve a minimizing search and higher values improve a maximizing search. For fitness_std, an increase in dispersion counts as improvement in either direction.

python
from sklearn_genetic.callbacks import ConsecutiveStopping

callback = ConsecutiveStopping(generations=5, metric="fitness")
ParameterTypeDefaultDescription
generationsint—Stop if the metric has not improved in this many consecutive generations
metricstr"fitness"History column to track: "fitness", "fitness_best", "fitness_max", "fitness_min", "fitness_std"

ThresholdStopping ​

Stops once the metric reaches or exceeds a target value.

python
from sklearn_genetic.callbacks import ThresholdStopping

callback = ThresholdStopping(threshold=0.98, metric="fitness_max")
ParameterTypeDefaultDescription
thresholdfloat—Stop when the metric ≥ this value
metricstr"fitness"History column to watch

DeltaThreshold ​

Stops when the spread (max − min) across the last N generations falls below a threshold — useful for detecting convergence.

python
from sklearn_genetic.callbacks import DeltaThreshold

callback = DeltaThreshold(threshold=0.001, generations=5, metric="fitness_min")
ParameterTypeDefaultDescription
thresholdfloat—Stop when max(metric[-N:]) - min(metric[-N:]) ≤ threshold
generationsint2Window size for the comparison
metricstr"fitness"History column to watch

TimerStopping ​

Stops after a wall-clock time limit.

python
from sklearn_genetic.callbacks import TimerStopping

callback = TimerStopping(total_seconds=300)  # stop after 5 minutes
ParameterTypeDescription
total_secondsfloatMaximum elapsed seconds before starting another candidate batch; an in-flight batch can finish after the limit

Pass it to fit to cap a long search by wall-clock time — handy for notebook and local experiments where the search space may be larger than expected:

python
from sklearn_genetic.callbacks import TimerStopping

callbacks = [TimerStopping(total_seconds=60)]

search.fit(X_train, y_train, callbacks=callbacks)

The search checks the budget between candidate batches, including batches within a generation. Final selection and refit can extend the total fit time.


LogbookSaver ​

Saves the generation logbook to a file after each generation. Useful for long runs — if the search is interrupted, you can inspect progress so far.

python
from sklearn_genetic.callbacks import LogbookSaver

callback = LogbookSaver(checkpoint_path="./logbook.pkl")

Restore with joblib:

python
from joblib import load

logbook = load("./logbook.pkl")
print(logbook)

ModelCheckpoint ​

Saves the search configuration, generation history and runtime state, including cached candidate scores, archived elites and random generator states. Each successful write replaces the previous snapshot. See the checkpoint recipe to resume a search.

python
from sklearn_genetic.callbacks import ModelCheckpoint

callback = ModelCheckpoint(checkpoint_path="./checkpoint.pkl")

Inspect the saved state:

python
import pickle

with open("./checkpoint.pkl", "rb") as f:
    checkpoint = pickle.load(f)

estimator_state = checkpoint["estimator_state"]
logbook = checkpoint["logbook"]

Live training plot ​

Use the plotting extra to watch best and average fitness as each generation finishes. The same callback works for GASearchCV and GAFeatureSelectionCV.

python
from sklearn_genetic.plots import LivePlot

callback = LivePlot(save_path="./training-progress.png")
search.fit(X, y, callbacks=callback)

Install with pip install sklearn-genetic-opt[plot]. The Matplotlib window updates during fitting when a graphical backend is available. save_path rewrites the current PNG after each generation, including in headless runs. The callback keeps the figure open for inspection after fitting.

For a custom callback, use plot_generation_metrics(logbook) to draw the same generation traces from the logbook received by on_step. After fitting, use plot_search_overview to inspect convergence, diversity, optimizer events and the best candidates or selected features.

The legacy TensorBoard callback remains importable for existing users who install TensorFlow separately. TensorFlow is no longer included in any extra.

Full Example ​

This example uses ThresholdStopping and DeltaThreshold together — it stops when accuracy exceeds 0.98 OR when the improvement between consecutive generations drops below 0.001:

python
from sklearn.datasets import load_digits
from sklearn.metrics import accuracy_score
from sklearn.model_selection import StratifiedKFold, train_test_split
from sklearn.tree import DecisionTreeClassifier

from sklearn_genetic import EvolutionConfig, GASearchCV, PopulationConfig, RuntimeConfig
from sklearn_genetic.callbacks import DeltaThreshold, ThresholdStopping
from sklearn_genetic.space import Categorical, Continuous, Integer

data = load_digits()
X, y = data["data"], data["target"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)

param_grid = {
    "min_weight_fraction_leaf": Continuous(0, 0.5),
    "criterion": Categorical(["gini", "entropy"]),
    "max_depth": Integer(2, 20),
    "max_leaf_nodes": Integer(2, 30),
}

cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)

evolved_estimator = GASearchCV(
    estimator=DecisionTreeClassifier(),
    cv=cv,
    scoring="accuracy",
    param_grid=param_grid,
    evolution_config=EvolutionConfig(
        population_size=16,
        generations=30,
        tournament_size=3,
        elitism=True,
        crossover_probability=0.9,
        mutation_probability=0.05,
    ),
    population_config=PopulationConfig(initializer="smart"),
    runtime_config=RuntimeConfig(n_jobs=-1, verbose=True),
)

evolved_estimator.fit(X_train, y_train, callbacks=[
    ThresholdStopping(threshold=0.98, metric="fitness_max"),
    DeltaThreshold(threshold=0.001, metric="fitness"),
])

y_predict_ga = evolved_estimator.predict(X_test)
print(evolved_estimator.best_params_)
print("Accuracy:", accuracy_score(y_test, y_predict_ga))

Tips & Gotchas ​

  • Callbacks see the same metric names that appear in the generation log: fitness, fitness_std, fitness_best, fitness_max, fitness_min.
  • ConsecutiveStopping watches fitness by default (mean population fitness). Use metric="fitness_best" if you want to stop when the overall best score stops improving.
  • TimerStopping stops scheduling new batches after the limit; an in-flight CV batch and final selection/refit can still complete.
  • When multiple callbacks are combined, any single one returning True stops the search.

See Also ​

Released under the MIT License.