Line of Best Fit
Statistics & ProbabilityThe line of best fit is the straight line that most closely follows the pattern of data points on a scatter plot, minimizing overall error.
Formula
\hat{y} = b_0 + b_1 x
Definition
The line of best fit is the straight line drawn through a scatter plot that comes as close as possible to all the data points. It summarizes the overall trend of the data.
Example
On a scatter plot of hours studied vs. test scores, the line of best fit goes through the middle of the point cloud, showing the general upward trend.
Key Insight
The line of best fit does not have to pass through any data point. It is positioned to minimize the overall distance between itself and all the points.
Definition
The line of best fit (least-squares regression line) is the line $\hat{y} = b_0 + b_1 x$ that minimizes the sum of squared residuals: $\sum (y_i - \hat{y}_i)^2$. The slope $b_1 = r(s_y/s_x)$ and intercept $b_0 = \bar{y} - b_1\bar{x}$. The line always passes through $(\bar{x}, \bar{y})$.
Example
For data with $\bar{x}=5$, $\bar{y}=80$, $s_x=2$, $s_y=10$, $r=0.8$: $b_1 = 0.8(10/2) = 4$. $b_0 = 80 - 4(5) = 60$. Line: $\hat{y} = 60 + 4x$. Prediction for $x=7$: $\hat{y} = 60 + 28 = 88$.
Key Insight
The line of best fit is only valid for interpolation (predicting within the range of x-values used to build it). Extrapolating far beyond the data range is unreliable because the linear relationship may not hold.
Definition
The least-squares regression line is derived by solving the normal equations $(X^TX)\beta = X^TY$. For simple linear regression, the unique solution exists when there is variation in $x$. The fitted values lie on the line and the residuals are orthogonal to all predictor vectors, a geometric property of projection onto the column space of $X$.
Example
Geometrically, OLS projects the response vector $Y$ onto the column space of $X$. The fitted values $\hat{Y} = HY$ where $H = X(X^TX)^{-1}X^T$ is the hat (projection) matrix. The residuals $e = Y - \hat{Y} = (I-H)Y$ are in the orthogonal complement of the column space of $X$.
Key Insight
The hat matrix $H$ has diagonal entries $h_{ii}$ (leverages) that measure how much influence each observation has on its own fitted value. High-leverage points with large residuals are the most influential and are diagnosed with Cook's distance.