AdaBoost Hyperparameters | GridSearchCV in Adaboost

XGBoost Hyperparameters Overview
📌 XGBoost is a relatively simple algorithm compared to other boosting methods, primarily requiring tuning of only three to four main hyperparameters.
⚙️ The Base Estimator is typically a Decision Tree with max\_depth=1 (Decision Stump); while other algorithms like Logistic Regression or SVM can be used, Decision Trees provide the best practical results (99.9% of the time).
❌ K-Nearest Neighbors (KNN) is generally incompatible because it lacks support for sample weighting, a requirement for boosting algorithms.

Key Hyperparameters Detailed
🔢 N\_estimators controls the maximum number of base models trained; if a perfect fit is achieved earlier, the process terminates early.
⚖️ Learning Rate applies a weight to each subsequent classifier; tuning it involves a trade-off with `n_estimators`.
💨 The default algorithm is 'SAMME.R', which typically converges faster than 'SAMME' and achieves lower test error with fewer iterations.

Impact of N\_estimators on Model Performance
📉 Setting `n_estimators=1` results in a model that is too simplistic, leading to underfitting.
🔥 Increasing `n_estimators` significantly (e.g., to 500 or 1500) leads to complex decision boundaries and clear signs of overfitting, taking longer to train.
🎯 The goal is to find an optimum value for `n_estimators` to avoid both underfitting and excessive overfitting.

Role of Learning Rate and Shrinkage
📐 The learning rate scales the calculated coefficient ($\alpha$) for each weak classifier; by default, $\alpha$ is calculated as $\frac{1}{2} \log \left(\frac{1-\text{error}}{\text{error}}\right)$.
📉 Decreasing the learning rate (e.g., to 0.1) reduces $\alpha$, which dampens the amplitude of sample weight updates in each step (shrinkage).
🐌 Slower learning due to a reduced learning rate helps dramatically reduce overfitting while allowing for a higher `n_estimators`.

Hyperparameter Tuning with Grid Search CV
📊 A default model achieved an initial accuracy of 78% on a custom classification dataset.
✅ Applying GridSearchCV by testing various combinations of `n_estimators` (up to 1500), low learning rates ($\leq 1$), and algorithms yielded a maximum accuracy of 83%.
✨ The optimal parameters found through tuning were `n_estimators=500`, learning\_rate=0.1, and the 'SAMME.R' algorithm.

Key Points & Insights
➡️ The combination of `n_estimators` and `learning_rate` is crucial for unlocking the true power of AdaBoost.
➡️ Lowering the learning rate (using shrinkage) mitigates overfitting that results from increasing the number of estimators.
➡️ Use GridSearchCV with a defined parameter grid focusing on `n_estimators` and `learning_rate` to systematically optimize model performance from an initial baseline (e.g., 78% to 83% accuracy).

📸 Video summarized with SummaryTube.com on Nov 28, 2025, 07:17 UTC