Interpreting Loss Curves
The heart of training monitoring
The loss curve is your most important diagnostic tool. A trained eye can tell within seconds whether training is healthy, problems are emerging, or immediate action is needed.
Anatomy of a loss curve
X-axis: Time
Training steps or epochs. Further right means longer training. Steps give more granularity.
Y-axis: Loss value
The model's error. Lower = better. Goal: drop to a low, stable level as quickly as possible.
Training Loss (purple)
Error on training data. Calculated after every batch. Often somewhat noisy due to stochasticity.
Validation Loss (dashed)
Error on validation data. Smoother than training loss. The decisive indicator of real performance.
The 4 fundamental loss curve patterns
Good training: train and val loss decrease together
Good training: Both curves drop steadily and converge to a low value. Small, stable gap between training and validation loss. It flattens out at the end (plateau).
Overfitting: val loss rises again
Overfitting: Training loss keeps dropping, validation loss starts to rise. The divergence point marks the optimal training time (early stopping!)
Underfitting: both curves stagnate at a high level
Underfitting: Both curves stagnate at a high level. The model isn't learning the patterns. Causes: LR too low, too few epochs, wrong model.
Learning rate too high: loss jumps around uncontrollably
Unstable training (LR too high): Strong oscillation, no convergence. The optimizer keeps overshooting the minimum. Solution: reduce LR.
Benchmarks for "good" loss values
Note: Absolute values depend heavily on task and model. Trend and comparison matter more than absolute numbers.
Reading loss curves in FrameTrain
In the FrameTrain monitoring panel you see training loss and validation loss in real time. The dashboard shows both steps and epochs on the X-axis. Base your actions on the validation loss — not the training loss!
Recognizing Good Training
The signs of successful, stable training
The ideal scenario
Good training: train and val loss decrease together
Checklist: 6 signs of good training
Both losses drop noticeably in the first epochs
Shows active learning. If losses barely drop from epoch 1 → underfitting, LR too low, or a data problem.
Training loss ≈ validation loss (small, stable gap)
Typically val loss is 5–15% higher. A huge gap means overfitting. No gap means underfitting or identical data (data leakage!).
Curves flatten out toward the end (plateau)
The model is approaching its ceiling on this dataset. Normal and desired.
No large jumps (spikes) in the loss
Occasional small spikes are normal. Large, recurring spikes mean LR too high or faulty data.
Validation accuracy rises monotonically or nearly so
Small fluctuations are normal. A long-term upward trend shows real improvement.
Validation loss reaches a minimum and stabilizes
That's the optimal checkpoint. The ideal model is saved here.
Understanding training phases
Phase 1: Rapid learning (epochs 1–3)
Loss drops steeply. The model learns the most obvious patterns. Validation loss closely follows training loss.
Phase 2: Refinement (epochs 4–8)
Loss drops more slowly. Subtler patterns are learned. The generalization gap may widen slightly.
Phase 3: Convergence / plateau (from epoch 8+)
Barely any more improvement. Training loss minimally below validation loss. A good time to stop.
Phase 4 (if trained too long): Overfitting
Training loss keeps dropping, validation loss rises. From here: load the best checkpoint.
When is training optimally finished?
Validation loss flattens out, only minimal fluctuation (±0.001)
→ Give it 2–3 more epochs of patience, then stop. Load the best checkpoint.
Validation loss rises for the first time
→ That was the best epoch. Load the best checkpoint and stop.
Validation loss rises for 3 epochs in a row
→ Early stopping triggered. The best checkpoint is optimal.
Training loss << validation loss (large gap)
→ Overfitting. Either revert to the best checkpoint or abort training.
Recognizing Overfitting
When the model memorizes the training data
Overfitting is the most common problem when training deep networks. The model learns the training data so well that it also memorizes noise and random artifacts. On new, unseen data it then performs noticeably worse.
Overfitting: val loss rises again
Clear warning signs
Training loss keeps dropping, validation loss rises
CriticalLarge, growing gap between training and validation loss
WarningTraining accuracy near 100%, validation accuracy noticeably lower
CriticalModel performs very well on training data, poorly on new data
CriticalValidation loss forms a U-curve after initially dropping
WarningWhy does overfitting happen? (Causes)
Too little training data
Few examples means the model memorizes them instead of learning patterns. Solution: collect more data or use augmentation.
Too many training epochs
Long training grinds the training data in ever deeper. Early stopping would have been the solution.
Model too complex for the amount of data
Too many parameters for little data means too much capacity for memorization. Use a smaller model or LoRA with a small rank.
No dropout / too little regularization
Without regularization the model learns more freely. Apply more dropout or weight decay.
Dataset too homogeneous
Dataset contains duplicates or very similar examples. Deduplication helps.
Data leakage
Test data ended up in training: the model has "cheated". Metrics look good, real performance is poor.
The bias-variance tradeoff
Overfitting and underfitting are the two sides of the classic bias-variance tradeoff:
Underfitting
High bias
Low variance
Model too simple
Ideal
Bias ↔ variance
Balanced
Sweet spot
Overfitting
Low bias
High variance
Model too complex
Mild overfitting is normal
Recognizing Underfitting
When the model doesn't learn the patterns
Underfitting means the model fails to capture the underlying patterns in the data. It performs poorly on both training and validation data. It's the opposite of overfitting.
Underfitting: both curves stagnate at a high level
Warning signs
High training loss that barely changes across epochs
The worst sign — the model is learning practically nothing.
High validation loss correlates with training loss (both high)
Both curves lie close together, but at a high level.
Low training accuracy (< 60–70% on simple tasks)
Can't even classify its own training data properly.
Loss curve flattens very early without dropping much
Stagnation after a few steps — no further learning progress.
Val loss and train loss are nearly identical, but both high
No overfitting, but also no learning — the model is too simple.
Causes and direct fixes
Learning rate too low
Signs: Loss drops <0.01 per epoch
→ Fix: Increase LR by a factor of 3–10. Start at 1e-5, try 3e-5, then 1e-4.
Too few epochs
Signs: Training loss still dropping at the last epoch
→ Fix: Double the epochs. The loss curve shows whether there's still potential.
Model too small / too simple
Signs: Even long training doesn't improve it
→ Fix: Choose a larger base model: BERT → RoBERTa → DeBERTa, or a 7B model instead of 1B.
Wrong optimizer
Signs: SGD without momentum on transformers
→ Fix: Use AdamW instead of SGD. AdamW is the standard for all transformer training.
Dropout too high
Signs: Train and val loss both high and similar
→ Fix: Set dropout to 0.0 for testing, then increase gradually.
Wrong data formatting
Signs: Loss is constant from the start (no drop)
→ Fix: Check the dataset format carefully: is tokenization correct? Are labels correct (0-indexed vs. 1-indexed)?
Vanishing gradients in very deep networks
Signs: Gradients near 0 in early layers
→ Fix: Check layer normalization, ensure residual connections, review initialization.
Quickly diagnosing underfitting
Unstable Training
Diagnosing loss spikes, explosions, and chaos
Unstable training shows up as strong fluctuations, sudden jumps, or a complete loss crash. These patterns have clear causes and solvable fixes.
Learning rate too high: loss jumps around uncontrollably
Instability patterns and what they mean
Strong oscillation: loss jumps up and down every epoch
Learning rate too highThe optimizer keeps "jumping" past the minimum and oscillates further. No convergence possible.
→ Reduce LR by a factor of 5–10. Add an LR warmup.
Single large spike → training recovers
Bad batch / outlier in dataA faulty or extremely difficult batch caused a temporary spike. Gradient clipping helps.
→ Enable gradient clipping (max_grad_norm=1.0). Check the dataset for outliers.
Loss explodes and becomes NaN or Inf
Gradient explosionGradients grow exponentially until numeric values become unstable. Training crashes.
→ Set gradient clipping to 1.0. Drastically reduce LR. Disable mixed precision or switch to fp32.
Loss suddenly rises after many good epochs
LR spike from a wrong scheduler / warmup too longA misconfigured LR scheduler can suddenly increase the LR. Or: a very unusual batch after long stability.
→ Check the scheduler configuration. Load the best checkpoint. Gradient clipping.
Regular, gentle waves in the loss
Cosine annealing with restarts (SGDR)Not a problem! Cyclical LR schedulers intentionally produce cyclical loss patterns. Each cycle converges anew.
→ Only if undesired: choose a different scheduler. Otherwise: expected behavior.
Gradient clipping – the most important safeguard
Gradient clipping caps the maximum norm of the gradients. If gradients get too large, they're scaled proportionally. Default value: max_grad_norm = 1.0
Without gradient clipping: Gradient = [0.01, 0.02, ..., 500.0] ← explosion! → weight update huge → training crashes (NaN) With gradient clipping (max_norm = 1.0): Gradient norm = sqrt(sum(g²)) = 500.01 If norm > 1.0: gradient = gradient × (1.0 / norm) → gradient is scaled, direction is preserved → safe update, training stays stable In FrameTrain: set to 1.0 by default.
FrameTrain protects automatically