How many epochs without improvement are tolerated before stopping.

Effect: Too low: stops too early (during temporary fluctuations). Too high: overfitting happens anyway.

💡 patience=3 for short training runs. patience=5 for long training runs with fluctuations.

Minimum improvement in validation loss to count as "better".

Effect: Prevents stopping when only minimal noise slightly improves the metric.

💡 min_delta=0.001 is a good starting value.

Which metric is monitored.

Effect: val_loss is more stable and direct. val_accuracy can plateau while val_loss is still decreasing.

💡 Always monitor val_loss, not val_accuracy.

Whether the best checkpoint is automatically loaded when stopping.

Effect: Without: last checkpoint (after overfitting). With: best checkpoint loaded.

💡 Always True. Only exception: if the last checkpoint is explicitly desired.

Each model gives one class prediction. The most common class wins (like an election).

# Combining 3 model predictions:
predictions = [model1.predict(x),  # "positive"
               model2.predict(x),  # "positive"  
               model3.predict(x)]  # "negative"
from collections import Counter
final = Counter(predictions).most_common(1)[0][0]
# → "positive" (2 of 3 models)

Use case: When class labels (not probabilities) are available

Softmax probabilities of all models are averaged. Then the highest probability determines the class.

# Averaging probabilities:
probs_1 = model1.predict_proba(x)  # [0.8, 0.2]
probs_2 = model2.predict_proba(x)  # [0.6, 0.4]
probs_3 = model3.predict_proba(x)  # [0.9, 0.1]

avg_probs = np.mean([probs_1, probs_2, probs_3], axis=0)
# → [0.77, 0.23]
final_class = np.argmax(avg_probs)  # class 0 = "positive"

Use case: When softmax probabilities are available – almost always better than majority voting

Different checkpoints from the same training run are combined. Cost-effective – no separate training needed.

# Last N checkpoints from one training run:
checkpoints = [
    'model_epoch_8.pt',
    'model_epoch_9.pt', 
    'model_epoch_10.pt'
]
# Average predictions from all checkpoints
ensemble_probs = np.mean([
    load_and_predict(ckpt, x) 
    for ckpt in checkpoints
], axis=0)

Use case: When multiple full training runs are too expensive – reuses existing checkpoints

Same model, but different random seeds or slightly different hyperparameters. More diversity = better.

# 5 runs with different seeds:
models = []
for seed in [42, 123, 456, 789, 1337]:
    model = train_with_seed(seed, lr=2e-5)
    models.append(model)

# Ensemble of the 5 models:
ensemble_pred = np.mean([
    m.predict_proba(x) for m in models
], axis=0)

Use case: Competitions, when maximum performance is needed

A meta-model learns to optimally combine the outputs of the base models. More complex but more powerful.

# Stacking procedure:
# 1. Train base models (cross-validation)
# 2. Use base model predictions as features
# 3. Train a meta-model on these features

# Simplest meta-learner: logistic regression
from sklearn.linear_model import LogisticRegression
meta_features = np.column_stack([
    model1.predict_proba(X_val),
    model2.predict_proba(X_val),
    model3.predict_proba(X_val)
])
meta_model = LogisticRegression().fit(meta_features, y_val)

Use case: When base models are very different (different architectures)