What Is Leverage and Why It Matters?

Sat, 12 Sep 2026 00:00:00 +0000

Adapted from NUS DSA3361 Tutorial 4 — Inferential Data Analytics.

When we first encounter regression, we often feel that estimating $\hat{\beta}$ and obtaining the prediction equation $\hat{y}=X\hat{\beta}$ should be enough. But courses usually do not stop there. We continue by looking more closely at the hat matrix $H$, leverage values $h_{ii}$, and related ideas.

Prerequisite: the matrix form of linear regression, especially

$\hat{\beta}=(X^TX)^{-1}X^Ty\;\;\Rightarrow\;\;\hat{y}=X\hat{\beta}\;\;\Rightarrow\;\;H=X(X^TX)^{-1}X^T$.

We will answer two questions that may be on your mind:

Why do we call $h_{ii}=H_{ii}$ leverage values? Or why it measures leverage?
Why does $h_{ii}$ matter when we analyse residuals?

For simplicity, we will focus entirely on simple linear regression (SLR). Suppose that we have five observations: $(x_1,y_1), \ldots,(x_5,y_5),$ and that the $x$’s have already been standardized, so that $\operatorname{avg}(x)=0$ and $\operatorname{std}(x)=1$.

Under this standardization, the two regression coefficients can be written as

\[ \hat{\beta}_1=\frac{1}{5}\sum_{j=1}^{5}x_jy_j\;\;\text{and}\;\; \hat{\beta}_0=\bar{y}=\frac{1}{5}\sum_{j=1}^{5}y_j. \]

Now imagine changing the response of observation $i$: replace $y_i$ with $y_i+\Delta$, while leaving all the other values unchanged. Then

\[ \hat{\beta}_0\;\;\Rightarrow\;\;\hat{\beta}_0+\frac{\Delta}{5}\;\;\text{and}\;\; \hat{\beta}_1\;\;\Rightarrow\;\;\hat{\beta}_1+\frac{x_i\Delta}{5}. \]

If $x_i\approx 0$, then moving $y_i$ up or down has almost no effect on the slope. It only shifts the regression line a little. But if $|x_i|$ is large, the same change $\Delta$ produces a much larger change in the slope: $\Delta\hat{\beta}_1=x_i\Delta / 5$.

This is the intuitive origin of the word leverage: the farther a point is from the centre of the $x$-values, the more leverage it has to pull and rotate the regression line.

But how much power can a point’s leverage really have? $h_{ii}$ gives us a useful answer: it measures how much the point pulls the fitted line toward itself.

Why $h_{ii}$ Measures Leverage?

Suppose we have already estimated $\hat{\beta}$. If we want fitted values for all five original observations, the prediction equation $\hat{y}=X\hat{\beta}$ tells us to plug the original $X$ back in:

\[ \begin{pmatrix}\hat{y}_1\\\hat{y}_2\\\hat{y}_3\\\hat{y}_4\\\hat{y}_5\end{pmatrix}=\begin{pmatrix}1&x_1\\1&x_2\\1&x_3\\1&x_4\\1&x_5\end{pmatrix}\hat{\beta}= X(X^TX)^{-1}X^T\begin{pmatrix}y_1\\y_2\\y_3\\y_4\\y_5\end{pmatrix}=H\begin{pmatrix}y_1\\y_2\\y_3\\y_4\\y_5\end{pmatrix}. \]

In this example, $H$ and $h_{ii}$ are quite intuitive (I computed them for you):

\[ H=\frac{1}{5} \begin{pmatrix} 1+x_1^2 & 1+x_1x_2 & \cdots & 1+x_1x_5 \\ 1+x_2x_1 & 1+x_2^2 & \cdots & 1+x_2x_5 \\ \vdots & \vdots & \ddots & \vdots \\ 1+x_5x_1 & 1+x_5x_2 & \cdots & 1+x_5^2 \end{pmatrix} \;\;\Rightarrow\;\; h_{ii}=\frac{1+x_i^2}{5}. \]

If we focus on one of these fitted values, say the one for observation 1, then

\[ \hat{y}_1=h_{11}y_1+\cdots+h_{15}y_5 =\frac{(1+x_1^2)y_1+(1+x_1x_2)y_2+\cdots+(1+x_1x_5)y_5}{5}. \]

Now the connection is visible:

\[ y_1\text{ changes by }\Delta \;\;\Rightarrow\;\; \hat{y}_1\text{ changes by }\frac{1+x_1^2}{5}\Delta. \]

Clearly, the farther $x_1$ is from the centre of the $x$-values, the more $\hat{y}_1$ responds to a change in $y_1$, and the stronger this self-pull becomes. For example:

If $x_1=2$, then $h_{11}=(1+2^2)/5=1$. Moving $y_1$ up by 1 unit changes $\hat{y}_1$ by 1.
If all other $x_i=-0.5$, so other $h_{ii}=(1+0.5^2)/5=0.25$. Moving $y_i$ up by 1 unit changes $\hat{y}_i$ by only 0.25.

The following figure may help make this idea more intuitive:

So we see that $h_{ii}$ measures how strongly observation $i$ can pull the fitted line toward itself. The larger $h_{ii}$ is, the more strongly the observation can shift the line, just like the point with $x_1=2$ moves the line noticeably above, pulling $\hat{y}_1$ closer to $y_1$, making its residual $e_1=y_1-\hat{y}_1$ look smaller. High-leverage points can therefore partly hide their own residuals. This leads us to our final topic:

Why does $h_{ii}$ matter when we analyse residuals?

If $(x_1,y_1)$ is a high-leverage point, its fitted value $\hat{y}_1$ is pulled toward $y_1$. Its prediction may therefore look unusually accurate, with a smaller residual $e_1=y_1-\hat{y}_1$. With a little matrix algebra, we can see why the residual is also less variable:

\[ \begin{aligned} e&=y-\hat{y}=Iy-X\hat{\beta}\\ &=Iy-X(X^TX)^{-1}X^Ty=(I-H)y. \end{aligned} \]

Therefore, we find $\operatorname{Cov}(e)=\sigma^2(I-H)$ or specifically $\operatorname{Var}(e_1)=\sigma^2(1-h_{11})$. The larger the leverage $h_{11}$, the smaller the variance of the first observation’s residual.

How about the others? If we add up all five leverage values, we get

\[ \sum_{i=1}^{5}h_{ii} =\sum_{i=1}^{5}\frac{1+x_i^2}{5} =\frac{5+\sum_{i=1}^{5}x_i^2}{5} =\frac{5+5}{5} =2. \]

Mini-Challenge: Why is $\sum_{i=1}^{5}x_i^2=5$?

\[ \text{Initial assumption:}\;\; \operatorname{std}(x)=1 \;\;\Rightarrow\;\; \frac{1}{5}\sum_{i=1}^{5}x_i^2=1 \;\;\Rightarrow\;\; \sum_{i=1}^{5}x_i^2=5. \]

The total leverage power of all observations is fixed and limited! If $(x_1,y_1)$ has more power to pull the regression line, the other points have less power to pull it. As a result, the other residuals collectively have larger variance. This leads to the question:

If we want to compare residuals, directly comparing the raw values $e_1,\ldots,e_5$ is not quite fair, because their variability depends on its leverage.

The fix is quite simple. Since we know that $\operatorname{Var}(e_i)=\sigma^2(1-h_{ii})$, we can standardize each residual by its own leverage-adjusted variability, replacing the unknown $\sigma$ with the sample estimate $s$. This gives the studentized residual:

\[ r_i:= \frac{e_i}{s\sqrt{1-h_{ii}}}\;\;\text{where}\; s=\sqrt{\frac{\sum_{j=1}^{n}e_j^2}{n-p}}\;\text{is an estimate of real std }\sigma. \]

The resulting $r_i$ has some very useful properties. For any observation $i$, we have approximately $E(r_i)=0$ and $\operatorname{Var}(r_i)=1$. If we also assume normally distributed errors, then approximately $r_i\sim N(0,1)$. This makes studentized residuals convenient for checking model assumptions, such as constant variance and normality.

That is why the model-diagnostics sections in DSA3361 (05 Model Diagnostics.pdf) and ST3131 (Topic 3 To Model Adequacy Checking.pdf) pay particular attention to $r_i$.

ST3131 Regression Tutorials

Sat, 25 Apr 2026 00:00:00 +0000

I served as a teaching assistant for ST3131 in AY25/26 Semester 2. While going through the course with students, I received many thoughtful questions, and I decided to organize some of them here as supplementary notes and slides.

Please note that these are NOT the official materials for the current offering of the course. If you rely on any of the material here, please do so at your own discretion. For official guidance, always refer to your course instructor and the current Canvas page.

Linear Regression Basics

If you are confused by how $\hat{\beta}$, $\mathrm{se}(\hat{\beta})$, the t-value, or confidence intervals are computed in R, the following references may be helpful:

This SLR table for simple linear regression. It also includes a summary of ANOVA.
This MLR table for multiple linear regression. It also includes a summary of ANOVA.

Tutorial 3: Some Extensions

The difference between the Confidence Interval (CI) and the Prediction Interval (PI)
Another way to interpret the coefficient $\hat{\beta}$ in the model
Proving that $\mathrm{SampCor}(x,y)=\sqrt{R^2}$ in simple linear regression

If you are interested, you can read this Tutorial 3 extension note.

Tutorial 6: Model Assumptions

What regression model assumptions mean
How to deal with possible violations in practice

In this note, we use a dataset as a concrete example to show how these assumptions can be checked.

Tutorial 7 & 8: Multicollinearity

Multicollinearity can create difficulties in computation, whether in R or Python.

These Tutorial 7 slides introduce what multicollinearity is and why it causes trouble.
These Tutorial 8 slides introduce how to detect multicollinearity and some common remedies.

Tutorial 10: Logistic Regression

Linear regression (SLR/MLR) is used when the response variable $y$ is continuous. But what should we use when facing a classification problem, for example when $y$ takes only two values such as male/female?

A natural extension is logistic regression. These Tutorial 10 slides give a quick introduction to the model, including how to fit it, perform hypothesis testing, and make predictions.

Linear Regression | Zhu Xuelin