DSA3361 | Zhu Xuelin

Tutorial 4 · Can Regression Recover the Truth?

Sun, 13 Sep 2026 00:00:00 +0000

Adapted from NUS DSA3361 Tutorial 4.

Suppose we have 100 observations, $(x_1,y_1),\ldots,(x_{100},y_{100})$, and they look roughly like $y\approx\beta_0+\beta_1x$. You may already know how to use simple linear regression to estimate $\hat\beta_0$ and $\hat\beta_1$, giving us the fitted equation

$$ \hat y=\hat\beta_0+\hat\beta_1x. $$

We can use this equation to predict $y$ for a new $x$. But have you ever wondered:

Are these computed numbers reliable or not?

Let’s make this question concrete with a familiar example: the formula for a falling object.

Suppose $y_i$ is the distance an object has fallen, and $x_i=t_i^2$, where $t_i$ is the time since it started falling. If you remember a little high-school physics, you may recognise that, ignoring air resistance and starting from rest,

$$ y=\frac12gt^2=0+\frac12g x. $$

This relationship does not come from statistics or machine learning. It comes from physics. So the true intercept is $0$, and the true slope is $\frac12g$, which is about $4.9$.

But suppose you have forgotten the value of $g$, and you want to estimate it. We can always grab a small ball, let it fall, and record its falling distance and time. This gives us a dataset. Then we fit an SLR model and obtain the fitted line $\hat y=\hat\beta_0+\hat\beta_1x$. Since the true slope is $\frac12g$, twice the fitted slope gives us an estimate of gravity: $\hat g=2\hat\beta_1$.

Let’s see what this looks like in practice. We will build the dataset step by step: collect times, measure the ideal heights, add inevitable measurement error, and then fit a line.

Try the experiment a few more times and keep track of what you get. Did you notice that $\hat\beta_1$ changes a little each time?

How should we understand this variation?

Neither our timing nor our distance measurements can be perfectly accurate. In other words, the two sides of the physics equation $y=\frac12gt^2$ will never be recorded exactly in our dataframe.

For this exercise, let’s put all of that measurement mess into one error term, $\epsilon_i$. We can write the model as

\[y_i=\beta_0+\beta_1x_i+\epsilon_i.\]

The errors are different each time we run the experiment. That is why the fitted line changes slightly when we collect a new dataset, and also why the points do not sit perfectly on one straight line.

In short, measurement error in each experiment means that our estimates $\hat\beta_0$ and $\hat\beta_1$ will not be exactly equal to the true values $\beta_0=0$ and $\beta_1=g/2$. But we still hope that a useful scientific method behaves sensibly: when the measurement error is not too large, OLS—or linear regression more generally—should tend to recover the underlying parameters reasonably well.

That is what we will investigate next. Let’s create a world where we know the truth and see how well regression can recover it.

Generate a world where we know the truth

We generate 100 sample points from

\[ Y_i=-1+0.5X_i+\epsilon_i,\qquad i=1,\ldots,100. \]

Here’s what each part means:

$X_i$ is the observed predictor, generated from $N(0,1)$.
$\epsilon_i$ is a random error term generated from $N(0,0.25)$.
The true intercept is $\beta_0=-1$, and the true slope is $\beta_1=0.5$.

Let’s create these vectors in Python:

import numpy as np

np.random.seed(123)
x = np.random.normal(loc=0, scale=1, size=100)
eps = np.random.normal(loc=0, scale=np.sqrt(0.25), size=100)
y = -1 + 0.5 * x + eps

With np.random.seed(123), the realised values (rounded to two decimal places) are below:

$$ \begin{aligned} \texttt{x}=\begin{bmatrix}x_1\\x_2\\\vdots\\x_{100}\end{bmatrix} = \begin{bmatrix}-1.09\\1.00\\\vdots\\-0.38\end{bmatrix} \;\;\text{and}\;\; \texttt{eps}=\begin{bmatrix}\epsilon_1\\\epsilon_2\\\vdots\\\epsilon_{100}\end{bmatrix} = \begin{bmatrix}0.32\\-0.99\\\vdots\\-0.11\end{bmatrix} \end{aligned} $$

Applying the true relation $Y_i=-1+0.5X_i+\epsilon_i$, we obtain the response vector $y=(y_1,\ldots,y_{100})^\mathsf{T}$. This is the world where we know the truth.

Use the true model to fit the data.

Then we can fit the linear model and print the two estimates:

import statsmodels.api as sm

X = sm.add_constant(x)
model = sm.OLS(y, X).fit()
beta0_hat, beta1_hat = model.params
print(f"beta0_hat = {beta0_hat:.4f}")
print(f"beta1_hat = {beta1_hat:.4f}")

So, using the data $(x_i,y_i)_{i=1}^{100}$, our fitted line is $\hat y=-1.0095+0.4917x$. Looking at the coefficients, it is pretty close to the true relationship $Y=-1+0.5X+\epsilon$. Nice! Let’s plot the data together with the true relationship and our fitted line.

import matplotlib.pyplot as plt

plt.scatter(x, y, color="black", alpha=0.75, label="Observed data")
y_pred = model.fittedvalues # Compute predicted y
order = np.argsort(x)
x_sorted, y_pred_sorted = x[order], y_pred[order]
plt.plot(x_sorted, y_pred_sorted, color="red", linestyle="--", label="Fitted line")

plt.xlabel("x")
plt.ylabel("y")
plt.legend()
plt.show()

We can use the function model.summary() below to check the model’s $R^2$. It also prints quantities with deeper statistical meaning, such as the $t$- and $F$-test statistics, AIC, and BIC. If you are curious about what all these numbers mean, you are very welcome to take ST3131 — by the end of that course, you will know this whole table inside out 😊.

# Recall: model = sm.OLS(y, X).fit()model.summary()

OLS Regression Results

Dep. Variable	y	R-squared	0.567
Model	OLS	Adj. R-squared	0.562
Method	Least Squares	F-statistic	128.3
Prob (F-statistic)	1.66e-19	Log-Likelihood	-69.520
No. Observations	100	Df Residuals	98
Df Model	1	Covariance Type	nonrobust
AIC	143.0	BIC	148.3

	coef	std err	t	P>\|t\|
const	-1.0095	0.049	-20.603	<0.001
x1	0.4917	0.043	11.325	<0.001

What if our used model is wrong?

In practice, for a dataset we have collected, who knows the true relationship leh? We usually try several plausible models and compare how they perform. For our example, the true relationship is $Y=-1+0.5X+\epsilon$. Now suppose we accidentally choose a quadratic model,

\[ \hat y=\hat\beta_0+\hat\beta_1x+\hat\beta_2x^2, \]

and let’s see what happens. The data $x$ and $y$ are already available from above, so we can create the extra predictor $x^2$, fit the model, and inspect its summary:

# Recall: x and y are already generated above.

x1 = x
x2 = x**2

X_predictors = np.column_stack([x1, x2])

X = sm.add_constant(X_predictors)

model2 = sm.OLS(y, X).fit()

OLS Regression Results

Dep. Variable	y	R-squared	0.568
Model	OLS	Adj. R-squared	0.559
Method	Least Squares	F-statistic	63.66
Prob (F-statistic)	2.19e-18	Log-Likelihood	-69.436
No. Observations	100	Df Residuals	97
Df Model	2	Covariance Type	nonrobust
AIC	144.9	BIC	152.7

	coef	std err	t	P>\|t\|
const	-0.9925	0.065	-15.315	<0.001
x1	0.4929	0.044	11.277	<0.001
x2	-0.0134	0.033	-0.403	0.688

Therefore, the fitted quadratic model is $ \hat y=-0.9925+0.4929x-0.0134x^2.$ The $t$-test suggests that the coefficient of $x^2$, $\beta_2$, is not significantly different from $0$ (Curious? ST3131 covers this😊), and the adjusted $R^2$ does not improve. In other words, adding a more complex term did not meaningfully improve the model.

How can we plot the fitted quadratic curve?

Just like what we did for the linear model: get the fitted pairs $(x_i,\hat y_i)$ from model2.fittedvalues → sort them by $x_i$ from smallest to largest → connect the fitted points from left to right.

y_pred2 = model2.fittedvalues

order = np.argsort(x)

plt.scatter(x, y, color="black", alpha=0.75, label="Observed data")

plt.plot(
 x[order],
 y_pred2[order],
 color="red",
 linestyle="--",
 label="Fitted quadratic curve"
)

plt.xlabel("x")
plt.ylabel("y")
plt.legend()
plt.show()

The fitted curve looks also straight here. But what we actually get is a curve.

Var(ε) = 0.2500 β̂₂ = −0.0134

ŷ = −0.9925 + 0.4929x − 0.0134x²

Takeaway: what did we do today?

Today, we started with a world where the true relationship was known:

\[ Y_i=-1+0.5X_i+\epsilon_i. \]

We then did three things:

We generated data from this known relationship.
We fitted the correct linear model and found that $\hat\beta_0=-1.01$ and $\hat\beta_1=0.49$ recovered the underlying relationship reasonably well.
We then added an unnecessary quadratic term $x^2$. It did not meaningfully improve the model, reminding us that a more complicated model is not automatically a better one.

So, back to the title:

Can regression recover the truth?

In this simulated world, yes. When the true relationship is linear and the noise is not too large, linear regression can recover the underlying relationship reasonably well from the observed data.

More generally, what we did today is called a simulation study. We first decide how the world works, generate data from that known data-generating mechanism, apply statistical methods, and then check how closely they recover the truth. Because the truth is known, we can directly examine quantities such as

\[ |\hat\beta-\beta|. \]

But real-data analysis is different: There, nobody tells us the true data-generating mechanism. We therefore cannot directly check whether “the truth” has been recovered, because quantities such as $\beta$ are unknown.

Instead, when comparing models on real data, we have to rely on quantities that we can actually observe or estimate, such as out-of-sample $R^2$, MSE, MAE, and other predictive or diagnostic measures. This is often called empirical evaluation or real-data evaluation.

Both simulation studies and empirical evaluations are common ways to evaluate statistical methods in research:

Simulation studies: we know the truth and ask whether the method can recover it.
Empirical evaluations: the truth is unknown, so we compare models using observable evidence.

If you’re a TA teaching this tutorial, here are some you can use alongside these notes.

Tutorial 3 · Who Eats More Chicken Wings?

Fri, 11 Sep 2026 00:00:00 +0000

Adapted from NUS DSA3361 Tutorial 3.

Everyone loves chicken wings—including students at NUS! Imagine we asked a group of boys and girls how many chicken wings they could eat. Here are their answers:

Group	Number of chicken wings
Boys	7, 8, 8, 11, 13, 13, 14, 16, 16, 17, 17, 18, 18, 21, 21
Girls	4, 5, 5, 6, 7, 7, 8, 9, 11, 12, 12, 13, 13, 14, 14

Who eats more chicken wings—boys or girls? A difference in this sample is a starting point, but what can it tell us about the wider population?

We’ll use this example to explore two questions:

Is there evidence of a difference between the two groups?
How can we estimate the size of that difference with a confidence interval?

All you need to follow along is some basic Python. We’ll build the statistical ideas step by step 😊.

A difference of 5.2 in our sample

Using these survey responses, let’s calculate the difference between boys’ and girls’ average chicken wing counts in our sample.

import numpy as np
import matplotlib.pyplot as plt

male_eat = np.array([
 7, 8, 8, 11, 13, 13, 14, 16, 16, 17, 17, 18, 18, 21, 21
])
female_eat = np.array([
 4, 5, 5, 6, 7, 7, 8, 9, 11, 12, 12, 13, 13, 14, 14
])

obs_eat_diff = male_eat.mean() - female_eat.mean()
print(f"Male mean: {male_eat.mean():.2f}")
print(f"Female mean: {female_eat.mean():.2f}")
print(f"Observed difference: {obs_eat_diff:.2f}")

Male sample mean: 14.53
Female sample mean: 9.33
Observed sample difference: 5.20

In this sample, boys eat 5.2 more wings than girls on average. But we want to know whether this difference holds for boys and girls across NUS.

A real difference — or just sampling variation?

To find out, let’s introduce permutation test.

First, imagine a world with no systematic difference

Suppose boys and girls have the same distribution of hot wings consumption. In this imagined world, knowing how many wings someone eats gives us no information about their gender. The M and F labels are therefore exchangeable: we can randomly reassign them among the observed counts while keeping 15 boys and 15 girls.

With that assumption, here is what we can do

(1) Shuffle the labels. Keep all 30 hot wings counts unchanged, but randomly reassign the gender labels: 15 observations receive M, and the remaining 15 receive F.

(2) Recalculate the mean difference. Using these new groups, calculate avg(male_eat) - avg(female_eat).

(3) Repeat many times. Repeat steps (1) and (2) 10,000 times, then plot a histogram of the resulting mean differences. This shows what differences look like in our imagined no-difference world.

(4) Locate our observed difference. Where does our actual difference of 5.2 fall in this histogram? Would a difference this far from zero be unusual in this world?

Our observed difference of 5.2 is very extreme. Loosely speaking—but not quite correctly—we might say, “There is only about a 1-in-1,000 chance of this happening.”

What exactly does that probability refer to? We’ll unpack this shorthand after the Python code. For now, let’s return to the main idea and see how to implement the permutation test.

Let’s briefly retrace the logic. We assumed that H₀: NUS boys and girls eat about the same number of chicken wings. Under H₀, the quantity avg(male_eat) - avg(female_eat) should follow a distribution like the one above. Yet our observed difference of 5.2 is way out on the right—roughly speaking, a result this large or larger happens only about 1‰ of the time. So, using this shorthand for now: if there really were no systematic difference, we would have encountered a very rare result in just one sample. There are two possible explanations:

We got suuuuper lucky and happened to observe a very rare result.
Our original no-difference assumption is WRONG.

Sure, we could just be lucky! But a result this unusual gives us good reason to question our no-difference assumption. And that’s the idea behind a permutation test. Makes sense?

Then, the Python implementation

# Sorting preserves the Hotwings order used in the tutorial dataset.
all_eat = np.sort(np.concatenate([male_eat, female_eat]))
male_num = len(male_eat)
all_index = np.arange(len(all_eat))

np.random.seed(123)
N = 10000
permuted_eat_diff = np.zeros(N)

for i in range(N):
 # (1) Shuffle labels: choose the observations assigned to males.
 permuted_male_index = np.random.choice(
 len(all_eat), size=male_num, replace=False
 )
 # The remaining observations are assigned to females.
 permuted_female_index = np.setdiff1d(
 all_index, permuted_male_index
 )

 # (2) Recalculate the mean difference.
 permuted_male_eat = all_eat[permuted_male_index]
 permuted_female_eat = all_eat[permuted_female_index]
 permuted_eat_diff[i] = (
 permuted_male_eat.mean() - permuted_female_eat.mean()
 )

# (3) Inspect the distribution of repeated differences.
plt.hist(permuted_eat_diff, bins=30, edgecolor="white")
plt.axvline(-abs(obs_eat_diff), color="royalblue", linestyle="--")
plt.axvline(abs(obs_eat_diff), color="darkorange", linestyle="--")
plt.xlabel("Difference in mean hot wings (male − female)")
plt.ylabel("Frequency")
plt.title("Permutation distribution under no systematic difference")
plt.show()

Unpacking the code

(1) Choose indices, then take the complement. The code shuffles labels by randomly choosing which observations belong to the male group. It does not change anyone’s hot wings count.

Here is a smaller, illustrative example with six observations and two male labels:

Observation index	0	1	2	3	4	5
Hot wings count	4	7	5	6	11	9
New label	F	F	M	F	M	F

all_index = [0, 1, 2, 3, 4, 5]
permuted_male_index = [2, 4]
permuted_female_index = [0, 1, 3, 5]

replace=False makes every selected index distinct. np.setdiff1d finds the indices left over. Every observation therefore appears in exactly one group, once per permutation.

(2) Use those indices to retrieve the counts and compute the statistic.

New male counts: [5, 11] → mean = 8.0
New female counts: [4, 7, 6, 9] → mean = 6.5
New mean difference: 8.0 − 6.5 = 1.5

(3) Store one difference per repetition. In the full example, each repetition uses 15 observations per group. permuted_eat_diff[i] stores its result; the histogram collects all 10,000 results.

Finally, calculate the p-value

Earlier, we asked: how strange is our observed difference of 5.2? The number we use to answer that question is called the p-value. Now let’s make that idea a little more precise.

Under H₀: no difference, we expect avg(boys) - avg(girls) to be somewhere around 0. A large positive difference would suggest that boys eat more; a large negative difference would suggest that girls eat more. Either direction would challenge our no-difference assumption.

So we use our observed 5.2 to set two cutoffs: +5.2 and −5.2. Looking at the histogram, what proportion of permutation results are at least 5.2, or at most −5.2? These are the two extreme tails we care about.

# (4) Compute the two tail probabilities with an add-one correction.
abs_obs = abs(obs_eat_diff)
p_left = (np.sum(permuted_eat_diff <= -abs_obs) + 1) / (N + 1)
p_right = (np.sum(permuted_eat_diff >= abs_obs) + 1) / (N + 1)
p_value = min(2 * min(p_left, p_right), 1)

print(f"Two-sided permutation p-value: {p_value:.3f}")

Two-sided permutation p-value: 0.002

Since our question is whether boys and girls differ at all, so we need to consider both directions. So, we turn the two tail probabilities into a two-sided p-value:

$$ p_{\text{value}} = 2\min(p_{\text{left}},\,p_{\text{right}}). $$

For three optional follow-ups, see:

We now have strong evidence of a difference. But another sample might give us a number other than 5.2, so there’s some uncertainty in our estimate. Can we give a range instead?

Bootstrap: a range estimate

If we had enough time and money, we could head out every day and survey a fresh group of students. After 100 days, we’d have 100 estimates of the difference! We could line them up from smallest to largest and look at the middle 95% to see how much our estimates vary, as illustrated below. That’s the intuition we’ll build on to get a confidence interval (a range estimate) using bootstrap.

Let’s save ourselves all that sampling!

But see the catch? If we keep heading out to collect new samples every day, we’ll still be doing surveys next year! Instead, we treat our original sample as a miniature version of the NUS population and sample from it with replacement. This is the bootstrap.

Why sample with replacement?

We want to mimic sampling from the whole NUS population. NUS is large, so picking one student barely changes who’s available for the next draw. But our “mini population”—Sample 1—is tiny! If we keep removing people without putting them back, it quickly shrinks, and the next draw becomes quite different. Putting each observation back keeps our mini population unchanged for the next draw, helping us mimic repeated draws from a much larger population.

What does it mean to sample someone twice?

It does not mean that a person actually answered a questionnaire twice. We are drawing from a distribution represented by the observed data. Repeating a count can be loosely imagined as sampling another person with similar eating behaviour.

Why did the permutation test use no replacement?

Because in a permutation test, we are not drawing a new set of observations like Bootstrap does. We are just shuffling the labels to see if our observed avg(boys) - avg(girls) looks unusually extreme.

So if the with/without replacement distinction still bothers you, a keynote is:

Bootstrap: mimic taking new samples from the population.
Permutation test: just reassign (shuffle) the labels — like shuffling a deck of cards.

Then, the Python implementation

np.random.seed(123)
N = 10000
boots_eat_diff = np.zeros(N)

for i in range(N):
 boots_male_eat = np.random.choice(
 male_eat, size=len(male_eat), replace=True
 )
 boots_female_eat = np.random.choice(
 female_eat, size=len(female_eat), replace=True
 )
 boots_eat_diff[i] = (
 boots_male_eat.mean() - boots_female_eat.mean()
 )

# The 2.5th and 97.5th percentiles leave 95% in the middle.
eat_diff_ci = np.quantile(boots_eat_diff, [0.025, 0.975])
print(f"Observed mean difference: {obs_eat_diff:.2f}")
print(f"95% bootstrap percentile CI: "
 f"[{eat_diff_ci[0]:.2f}, {eat_diff_ci[1]:.2f}]")

Observed mean difference: 5.20
95% bootstrap percentile CI: [2.40, 8.00]

Unpacking the code

Code	What it does	Why it matters
`np.random.choice(...)`	Randomly draws from the supplied observations	Reuses the data we already have
`replace=True`	Allows a count to be drawn repeatedly	The composition, and therefore the mean, can vary
`boots_male_eat.mean() - boots_female_eat.mean()`	Recalculates the same statistic	The direction remains male minus female
`np.quantile(..., [0.025, 0.975])`	Finds the endpoints of the middle 95%	Produces the percentile confidence interval

The estimated difference is 5.2 hot wings, with a 95% bootstrap percentile confidence interval of [2.4, 8.0] hot wings for the population mean difference.

The interval expresses uncertainty in our estimate of the mean difference. It is not an interval containing 95% of individual people’s consumption differences. Its confidence level refers to the approximate long-run coverage of this interval-building procedure under suitable sampling conditions.