An artificial neural network (ANN) is loosely inspired by the human brain. It consists of layers of neurons connected to one another. Each connection has a weight – the knowledge learned during training.

How a single neuron works

Each neuron receives inputs from all neurons in the previous layer, multiplies them by weights, sums everything up, and applies an activation function:

Neuron computation (simplified):
  output = activation(w₁·x₁ + w₂·x₂ + ... + wₙ·xₙ + b)

  w₁...wₙ = weights (learned during training)
  x₁...xₙ = inputs from the previous layer
  b        = bias (offset value)
  activation = non-linear function (ReLU, sigmoid, etc.)

Example with ReLU:
  output = max(0, w₁·x₁ + w₂·x₂ + b)
  → everything < 0 becomes 0 → the network learns "fires / doesn't fire"

Activation functions

Key layer types in modern networks

Depth vs. width: why depth is better

A deep network (many layers) learns hierarchical representations: lower layers learn simple patterns (edges, syllables), middle layers combine them (shapes, words), upper layers recognize complex concepts (faces, meanings). Width alone isn't enough.

Since the paper "Attention Is All You Need" (Vaswani et al., 2017), the transformer architecture has dominated the AI world. All modern LLMs – GPT-4, LLaMA, Mistral, BERT – are based on it. The key: the self-attention mechanism.

Self-attention: the core idea

Self-attention lets every token in the text "look at" every other token to understand its context. Three projections (query, key, value) determine how strongly each token attends to others:

For each token i:
  Query_i  = Token_i × W_Q    (what am I looking for?)
  Key_j    = Token_j × W_K    (what do I offer?)
  Value_j  = Token_j × W_V    (what is my content?)

Attention-Score(i,j) = softmax(Query_i · Key_j / √d_k)
Output_i = Σ_j [Attention-Score(i,j) × Value_j]

→ Each token gets an output that is a
  weighted combination of all other tokens.

Multi-head attention

Instead of a single attention computation, h parallel heads are used. Each head learns to attend to a different type of relationship: head 1 → syntactic dependencies, head 2 → coreference, head 3 → semantic similarity, etc.

Anatomy of a transformer block

Modern LLMs at a glance

VRAM requirements for LLM fine-tuning

Behind every training run are two fundamental algorithms: backpropagation (computes how wrong the model is and why) and gradient descent (adjusts the parameters in the right direction).

The complete learning cycle

The learning rate: the slider of learning

The learning rate (LR) determines how large each step downhill is. Too large: you overshoot the minimum and bounce around wildly. Too small: you almost never get there.

Weight update:
  W_new = W_old - α × ∂L/∂W

  α (alpha) = learning rate (e.g. 0.00002 = 2e-5)
  ∂L/∂W    = gradient (derivative of the loss w.r.t. W)

Example:
  W_old = 0.5, gradient = 0.3, LR = 0.01
  W_new = 0.5 - 0.01 × 0.3 = 0.5 - 0.003 = 0.497

Stochastic Gradient Descent (SGD) vs. Batch GD

Why local minima aren't a problem (for large LLMs)

In low-dimensional spaces (2D or 3D), there are many local minima you can get stuck in. In high-dimensional parameter spaces with billions of parameters, it's fundamentally different: almost all critical points are saddle points or flat regions, not true local minima. Moreover, all "good" local minima are about as good as the global minimum.

Vanishing & exploding gradients

In very deep networks, gradients can vanish during the backward pass (→ weights barely change, training stagnates) or explode (→ weights become huge, training breaks down). Modern architectures address this through: