Development version

You are reading the latest (development) docs. For stable documentation, see stable.

Tune SGDRegressor

Time: 5 min | Difficulty: Intermediate

What This Solves

SGDRegressor is a linear model that is sensitive to feature scaling and hyperparameter selection. This recipe demonstrates a two-stage optimization workflow using genetic algorithms:

1. Feature selection with GAFeatureSelectionCV
2. Hyperparameter tuning with GASearchCV

The example uses a synthetic regression dataset generated with make_regression(), performs feature scaling using StandardScaler, optimizes the model using RMSE during cross-validation, and evaluates the final model using the R² score on the test set.

Recipe

from sklearn.datasets import make_regression
from sklearn.model_selection import KFold, train_test_split
from sklearn.linear_model import SGDRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import r2_score
from sklearn.pipeline import Pipeline

from sklearn_genetic import GASearchCV, GAFeatureSelectionCV
from sklearn_genetic.space import Categorical, Continuous, Integer
from sklearn_genetic.config import (
    RuntimeConfig,
    EvolutionConfig,
    OptimizationConfig,
)

# Generate a synthetic regression dataset
X, y = make_regression(
    n_samples=1000,
    n_features=20,
    noise=0.1,
    random_state=42,
)

# Split the data
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
)

# Apply CV on each fold and then apply Standard scaler on the training fold
# Using the statistics (i.e mean, standard deviation) of the training fold
# Transform both the training and validation fold.
# Next, use this transformed training fold data to fit the SGD Regressor
# Evaluate on the validation fold

pipe = Pipeline(
    [
        ("scaler", StandardScaler()),
        ("sgd", SGDRegressor(random_state=42)),
    ]
)

cv = KFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)

# Stage 1: Feature Selection
selector = GAFeatureSelectionCV(
    estimator=pipe,
    cv=cv,
    scoring="neg_root_mean_squared_error",
    evolution_config=EvolutionConfig(
        population_size=20,
        generations=15,
        elitism=True,
    ),
    optimization_config=OptimizationConfig(
        diversity_threshold=1.0,
        sharing_alpha=5.0,
    ),
    runtime_config=RuntimeConfig(
        n_jobs=-1,
        verbose=True,
    ),
    random_state=42,
    local_search_steps=3,
)

selector.fit(X_train, y_train)

mask = selector.support_

print(
    f"Stage 1: {mask.sum()} features selected "
    f"(from {X_train.shape[1]})"
)

X_train_sel = X_train[:, mask]
X_test_sel = X_test[:, mask]

# Stage 2: Hyperparameter Tuning
param_grid = {
    "sgd__alpha": Continuous(1e-6, 1e-2, distribution="log-uniform"),
    "sgd__l1_ratio": Continuous(0.0, 1.0),
    "sgd__max_iter": Integer(500, 5000),
    "sgd__tol": Continuous(1e-6, 1e-2),
    "sgd__eta0": Continuous(1e-4, 1.0, distribution="log-uniform"),
    "sgd__n_iter_no_change": Integer(5, 50),
    "sgd__penalty": Categorical([
        "l2",
        "l1",
        "elasticnet",
    ]),
    "sgd__learning_rate": Categorical([
        "constant",
        "optimal",
        "invscaling",
        "adaptive",
    ]),
    "sgd__average": Categorical([
        True,
        False,
    ]),
}

ga = GASearchCV(
    estimator=pipe,
    param_grid=param_grid,
    scoring="neg_root_mean_squared_error",
    cv=cv,
    evolution_config=EvolutionConfig(
        population_size=20,
        generations=15,
        elitism=True,
    ),
    runtime_config=RuntimeConfig(
        n_jobs=-1,
        verbose=True,
    ),
    random_state=42,
)

ga.fit(X_train_sel, y_train)

pred = ga.predict(X_test_sel)

print("R² on Test Set:", round(r2_score(y_test, pred), 4))
print("Best Parameters:", ga.best_params_)

Key Points

• Scale the features before optimization: SGDRegressor is sensitive to the scale of the input features. Always apply StandardScaler before feature selection and hyperparameter tuning.
• Two-stage optimization: First select informative features using GAFeatureSelectionCV, then tune the estimator using GASearchCV.
• Cross-validation metric: The genetic algorithms optimize the model using neg_root_mean_squared_error.
• Final evaluation: The tuned model is evaluated on the held-out test set using the R² score.

See Also

• Tune for MAE — MAE as alternative metric
• Tune for RMSE — custom RMSE scorer
• LightGBM Regressor — fastest training option