Regression
Regression is a statistical method that models the relationship between variables, typically to predict the value of one variable from another.
Definition
Regression is a method for finding the best line (or curve) through data points on a scatter plot, which can be used to predict one value based on another. Linear regression finds the equation of the line that best fits the data by minimizing the sum of squared residuals (vertical distances from each point to the line), yielding the least-squares regression line $\hat{y} = b_0 + b_1 x$, where $b_1 = r(s_y/s_x)$ and $b_0 = \bar{y} - b_1\bar{x}$. Formally, ordinary least squares (OLS) regression estimates $\beta$ in $Y = X\beta + \epsilon$ by minimizing $\|Y - X\beta\|^2$, yielding $\hat{\beta} = (X^TX)^{-1}X^TY$; under the Gauss-Markov theorem, OLS is BLUE (best linear unbiased estimator) when errors are i.i.d. with zero mean and constant variance, and the $R^2$ statistic measures goodness of fit.
Example
If every hour of studying tends to add $5$ points to a score, regression finds that exact relationship so you can predict a score from any number of hours studied. For average daily temperature ($x$) and ice cream sales ($y$), a regression line $\hat{y} = -50 + 3.2x$ predicts sales of $-50 + 3.2(85) = \$222$ for a day at $85$ degrees, with the residual being actual minus predicted. In simple linear regression, $R^2 = r^2$ (the squared correlation); in multiple regression, $R^2 = 1 - RSS/TSS$, and adjusted $R^2$ penalizes for additional predictors: $R^2_{adj} = 1 - \frac{RSS/(n-p-1)}{TSS/(n-1)}$.
Key Insight
Regression turns the pattern you see in a scatter plot into a mathematical equation you can use for predictions: the slope $b_1$ tells you that for each $1$-unit increase in $x$, $y$ is predicted to increase by $b_1$ units, while the intercept $b_0$ is the predicted $y$ when $x = 0$ (which may not be meaningful in context). The Gauss-Markov theorem guarantees OLS is BLUE under mild assumptions but does not require normality; adding the normality assumption for errors enables exact t and F inference, while violating assumptions (heteroscedasticity, autocorrelation) requires robust or generalized least squares.