Regression
Statistics & ProbabilityRegression is a statistical method that models the relationship between variables, typically to predict the value of one variable from another.
Definition
Regression is a method for finding the best line (or curve) through data points on a scatter plot. The line can be used to predict one value based on another.
Example
If you know that every hour of studying tends to add $5$ points to a score, regression finds that exact relationship so you can predict a score from any number of hours studied.
Key Insight
Regression turns the pattern you see in a scatter plot into a mathematical equation you can use for predictions.
Definition
Linear regression finds the equation of the line that best fits the data by minimizing the sum of squared residuals (vertical distances from each point to the line). The result is the least-squares regression line: $\hat{y} = b_0 + b_1 x$, where $b_1 = r(s_y/s_x)$ and $b_0 = \bar{y} - b_1\bar{x}$.
Example
Data: average daily temperature ($x$) and ice cream sales ($y$). Regression line: $\hat{y} = -50 + 3.2x$. For a day with temperature $85$ degrees: predicted sales $= -50 + 3.2(85) = \$222$. Residual = actual - predicted.
Key Insight
The slope $b_1$ tells you: for each $1$-unit increase in $x$, $y$ is predicted to increase by $b_1$ units. The intercept $b_0$ is the predicted $y$ when $x = 0$ (which may not be meaningful in context).
Definition
Ordinary least squares (OLS) regression estimates $\beta$ in $Y = X\beta + \epsilon$ by minimizing $\|Y - X\beta\|^2$, yielding $\hat{\beta} = (X^TX)^{-1}X^TY$. Under the Gauss-Markov theorem, OLS is BLUE (best linear unbiased estimator) when errors are i.i.d. with zero mean and constant variance. The $R^2$ statistic measures goodness of fit.
Example
In simple linear regression, $R^2 = r^2$ (the squared correlation). In multiple regression, $R^2 = 1 - RSS/TSS$ where RSS is the residual sum of squares and TSS is the total sum of squares. Adjusted $R^2$ penalizes for additional predictors: $R^2_{adj} = 1 - \frac{RSS/(n-p-1)}{TSS/(n-1)}$.
Key Insight
The Gauss-Markov theorem guarantees OLS is BLUE under mild assumptions but does not require normality. Adding the normality assumption for errors enables exact t and F inference. Violating assumptions (heteroscedasticity, autocorrelation) requires robust or generalized least squares.