Stanford CS229: Machine Learning - Linear Regression and Gradient Descent | Lecture 2 (Autumn 2018)

Linear Regression Fundamentals
📌 Linear regression is introduced as one of the simplest supervised learning algorithms for regression problems where the output $Y$ is a continuous value (e.g., house price).
🏡 The lecture uses house size (in square feet) versus price (in thousands of dollars) from Portland, Oregon, as a motivating example to fit a straight line to the data.
⚙️ The process of supervised learning involves feeding a training set to a learning algorithm to output a hypothesis function $h(x)$ used for making predictions on new inputs.

Hypothesis Representation and Parameters
📐 For linear regression with one feature $X$ (size), the hypothesis is represented as $h(\mathbf{x}) = \theta_0 + \theta_1 X$.
📊 For multiple features ($X_1$=size, $X_2$=bedrooms), the hypothesis generalizes to $h(\mathbf{x}) = \theta_0 + \theta_1 X_1 + \theta_2 X_2$, concisely written using a dummy feature $X_0=1$ as $h(\mathbf{x}) = \sum_{j=0}^{n} \theta_j X_j$.
🎛️ $\theta$ (Theta) represents the parameters of the learning algorithm, and the algorithm's job is to select these parameters to minimize prediction error across the $M$ training examples.

Cost Function and Optimization
📉 The goal is to minimize the cost function $J(\theta)$, defined as one-half the sum of squared errors across all $M$ training examples:
$$J(\theta) = \frac{1}{2} \sum_{i=1}^{M} (h_{\theta}(\mathbf{x}^{(i)}) - y^{(i)})^2$$
⛰️ Gradient Descent is presented as an iterative algorithm to find $\theta$ that minimizes $J(\theta)$ by repeatedly taking steps in the direction of the steepest descent (negative gradient).
🏃 The update rule for each parameter $\theta_j$ in gradient descent is: $\theta_j := \theta_j - \alpha \frac{1}{M} \sum_{i=1}^{M} (h_{\theta}(\mathbf{x}^{(i)}) - y^{(i)}) X_j^{(i)}$, where $\alpha$ is the learning rate.

Gradient Descent Variants and Normal Equation
📦 Batch Gradient Descent calculates the gradient using the entire training set ($M$ examples) in every iteration, which is slow for very large datasets (e.g., hundreds of millions of examples).
⚡ Stochastic Gradient Descent (SGD) updates the parameters based on the error of one randomly chosen training example at a time, resulting in a noisy but much faster convergence path for large datasets.
✔️ For the special case of linear regression, the Normal Equation provides a closed-form, one-step solution to find the optimal $\theta$ without iteration: $\mathbf{\theta} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$.

Key Points & Insights
➡️ Hypothesis Structure is Key: Designing any ML algorithm requires clear decisions on data structure, hypothesis representation, and how the hypothesis represents that structure.
➡️ Learning Rate Tuning: For gradient descent, the learning rate $\alpha$ requires empirical tuning; if $J(\theta)$ increases, $\alpha$ is too large; common starting points include $0.01$ for scaled features, often tested on an exponential scale ($0.01, 0.02, 0.04, \dots$).
➡️ Batch vs. Stochastic: For small datasets, use Batch Gradient Descent for guaranteed convergence without worrying about parameter oscillation; for large datasets, Stochastic Gradient Descent is preferred due to the high cost of scanning terabytes of data for a single batch update.
➡️ Normal Equation Advantage: The Normal Equation is highly efficient for linear regression as it finds the global optimum in one matrix calculation step, avoiding the need for iterative tuning associated with gradient descent.

📸 Video summarized with SummaryTube.com on Jan 16, 2026, 15:32 UTC