AIC4P04

Neural Network Training Fundamentals
📌 Training a neural network involves learning weights and biases using data, specifically in supervised learning where examples are paired with class labels.
🧠 For a multi-class classification problem with $C$ classes, the network requires $C$ neurons in the output layer, each associated with a class.
📝 The training set consists of $(x, y)$ pairs, where $x$ is the input vector and $y$ is the target, often represented as a one-hot binary vector of length $C$.

Loss Functions for Optimization
⚖️ Training minimizes an objective function composed of a loss function that measures the mismatch between network outputs ($f$) and targets ($y$).
📉 Two popular loss functions discussed are Mean Squared Error (MSE), which computes the squared norm of the difference between output and target, and Cross-Entropy.
⭐ Cross-Entropy requires network outputs to be in the interval $[0, 1]$ and sum to 1 (probabilistic normalization, e.g., using Softmax), penalizing strongly when the output for the correct class is small.

Prediction and Binary Classification
🎯 In multi-class prediction, the simplest decision rule is taking the max of the network output, assigning the class associated with the highest score (e.g., $0.7$).
👤 For binary classification (two classes), one can use a single output neuron with a sigmoid activation (output in $[0, 1]$), using 0.5 as a common decision threshold.
➖ Binary classification with a single output neuron requires targets to be scalar values in $[0, 1]$, necessitating the use of the specialized Binary Cross-Entropy loss function instead of standard Cross-Entropy.

Optimization via Gradient Descent and Backpropagation
🔄 Neural networks are trained using gradient-based optimization, specifically Gradient Descent, which updates parameters in the direction of the negative gradient of the loss function.
➗ The update rule involves subtracting the product of the learning rate ($\rho$, step size) and the gradient (derivative of the objective function w.r.t. the weight/bias) from the current parameter value.
⚙️ Backpropagation is the efficient algorithm used to compute the gradient of the objective function with respect to all weights and biases in a multi-layer network.

Backpropagation Mechanics
🔗 Backpropagation relies on two key ingredients: the Chain Rule for differentiating composite functions (layers) and a two-stage process: the forward stage (computing and storing neuron outputs) and the backward stage (computing derivatives from output to input).
💾 During the backward stage, intermediate values called deltas are computed and stored for each neuron to ensure efficient differentiation and avoid recalculation.

Key Points & Insights
➡️ For multi-class problems with $C$ classes, define $C$ output neurons, using one-hot vectors for targets $y_i$.
➡️ When outputs are normalized probabilistically (sum to 1), Cross-Entropy is preferred as it provides a stronger penalization for incorrect, low-probability predictions than MSE.
➡️ Gradient descent updates parameters using the rule: $\text{parameter}_{\text{new}} = \text{parameter}_{\text{old}} - \rho \times \frac{\partial L}{\partial \text{parameter}}$.
➡️ Backpropagation is critical as it efficiently calculates the necessary gradients ($\frac{\partial L}{\partial w}$ and $\frac{\partial L}{\partial b}$) using the Chain Rule across the network layers.

📸 Video summarized with SummaryTube.com on Dec 23, 2025, 11:29 UTC