AdaBoost - A Step by Step Explanation

AdaBoost Algorithm Steps (Conceptual Overview)
📌 The initial step involves setting the weight for every row in the dataset equally, calculated as $1/N$, where $N$ is the total number of samples.
🔧 In Stage 1, a weak classifier (like a decision stump with depth 1) is trained on the weighted data to find the split that maximizes the information gain or minimizes the entropy.
⚖️ After training the first model (Model 1), its performance is evaluated to calculate $\text{alpha}_1$, which represents the weight of this model in the final prediction, dependent on its error rate ($\text{error}_1$).
🔄 The core of AdaBoost is weight updating: misclassified samples receive increased weight, and correctly classified samples receive decreased weight, preparing the dataset for the next stage.

Weight Calculation and Error Metrics
📈 The weight update formula for misclassified points is: $\text{New Weight} = \text{Old Weight} \times e^{\text{alpha}}$, and for correctly classified points: $\text{New Weight} = \text{Old Weight} \times e^{-\text{alpha}}$.
📉 The error rate ($\text{error}_t$) for a classifier at stage $t$ is the sum of the weights of all misclassified samples.
➗ To ensure the total weight sums to 1 after updating, all new weights must be normalized by dividing by the sum of all updated weights.
🧮 The model weight ($\text{alpha}_t$) is calculated using the formula $\text{alpha}_t = \frac{1}{2} \log \left( \frac{1 - \text{error}_t}{\text{error}_t} \right)$, which inversely correlates high error with low model weight.

Resampling and Iteration (Boosting)
🔄 After weight updates, a new dataset is created via sampling with replacement proportional to the new weights (this is called sub-sampling or boosting).
🎲 This resampling process involves creating cumulative weight ranges and generating random numbers between 0 and 1 to select which samples will be included in the new training set for the subsequent stage ($t+1$).
♻️ This entire process (training a weak classifier, calculating $\text{alpha}$, updating weights, and resampling) is repeated for the desired number of stages (e.g., 50 times for 50 trees/stumps).

Key Points & Insights
➡️ The goal of selecting the weak classifier at each stage is to find the one that yields the maximum decrease in entropy (or maximum information gain).
➡️ The $\text{alpha}$ value quantifies the reliability of a specific model; a lower error rate leads to a higher $\text{alpha}$, giving that model more influence in the final prediction.
➡️ Boosting inherently focuses subsequent classifiers on the data points that previous classifiers struggled with by increasing their weights.
➡️ The resampling technique ensures that the next model focuses disproportionately on the hard-to-classify samples, driving iterative improvement.

📸 Video summarized with SummaryTube.com on Nov 27, 2025, 17:57 UTC