Who Eats More Chicken Wings? | Zhu Xuelin

Are Permutation Histograms Always Bell-Shaped?

Mon, 01 Jan 0001 00:00:00 +0000

A common misconception

Students sometimes connect permutation tests with the CLT, normal distributions, or bell-shaped histograms. Just one caution: a permutation distribution is not necessarily bell-shaped. Its shape depends on what we choose to recalculate.

In the main example, we use avg(boys) - avg(girls). This statistic can be positive or negative, and its permutation distribution is centred around 0. But we could instead use |avg(boys) - avg(girls)|. This quantity is never negative, so its permutation distribution has a very different shape:

The key question is not whether the histogram looks normal. The key question is: under the null assumption, which values would be considered extreme?

Once “extreme” has been defined appropriately, we can calculate the p-value in the same spirit.

Choosing a Quantity for a Permutation Test

Mon, 01 Jan 0001 00:00:00 +0000

What should we recalculate?

You may be wondering: in a permutation test, what quantity should we use? For example, if we want to test whether avg(boys) = avg(girls), we could use

avg(boys) - avg(girls), or
avg(boys) ÷ avg(girls).

That is a good question because it has two parts: what hypothesis are we trying to check, and which quantity helps us check it? Let’s use a small stock-return example.

Two stocks, five daily returns

Suppose the daily returns are:

	Daily returns
Stock 1	+1%, −2%, +3%, +2%, +5%
Stock 2	−3%, −2%, +4%, +1%, −7%

Our hypothesis is:

Hypothesis A: In the long run, avg(Stock 1) = avg(Stock 2).

Option 1: the signed difference

We can use avg(Stock 1) − avg(Stock 2). This quantity can be positive or negative. If Hypothesis A is true, we expect the permutation distribution to be centred around 0, with positive and negative values on both sides:

After this, the rest is just like chicken wing example: decide what counts as extreme, then calculate the proportion of permutation results at least that extreme.

Option 2: the absolute difference

We can also use |avg(Stock 1) − avg(Stock 2)|. This quantity is never negative. If Hypothesis A is true, we expect it to be small, so the permutation distribution looks different:

Here is a small challenge: which part of this histogram should count as extreme? Once we define “extreme,” we can again calculate the proportion of permutation results that are at least as extreme as the observed value. That proportion is our p-value.

Why not use the ratio?

The ratio avg(Stock 1) ÷ avg(Stock 2) is less convenient here:

The two averages can be positive or negative, so the ratio can jump between positive and negative values.
Some returns are close to 0%. If a permutation average gets close to 0 in the denominator, the ratio can become enormous or unstable.

In theory, if the two means were equal and safely away from zero, we might expect |avg(Stock 1) ÷ avg(Stock 2)| to be around 1. But the sign changes and near-zero denominators make this statistic awkward for this example, so we set it aside.

So, which quantity should we use?

There is no single fixed answer. It depends on the question we want to study: are we asking whether two groups have equal means, or whether one group’s mean equals a particular value such as 20 wings?

Under a given hypothesis, a quantity is useful when its permutation distribution has a pattern we can understand and use to define “extreme.” The two plots above show two reasonable choices for the same equality question; they simply require us to define extremeness in different ways.

Why Use the Smaller Tail?

Mon, 01 Jan 0001 00:00:00 +0000

Why define the two-sided p-value as `2 × min(p_left, p_right)` instead of using `max`?

First, don’t worry about this for exams or daily use. Usually, p_left and p_right are fairly similar, say 0.009 and 0.012. So min and max often lead to the same conclusion.

But let’s dig a little deeper. What if p_left = 0.001 and p_right = 0.026? Then:

2 × min = 0.002 → reject the assumption.
2 × max = 0.052 → just fail to reject.

Why do we use the min? The symbol ⇔ means “is equivalent to.”

2 × min < 0.05 ⇔ p_left < 0.025 OR p_right < 0.025 ⇔ at least one direction is sufficiently extreme.
2 × max < 0.05 ⇔ p_left < 0.025 AND p_right < 0.025 ⇔ both directions must be sufficiently extreme.

The second rule asks for more than we want: one sufficiently extreme direction is already enough evidence against our assumption. A silly analogy:

Assumption: this person is innocent.

Evidence 1: He is holding a knife covered with the victim’s blood.

Evidence 2: He owns an old rusty gun that stopped working years ago.

Even if Evidence 2 is only borderline suspicious, Evidence 1 alone may already give us a strong reason to question the assumption. We don’t need both pieces of evidence to be equally suspicious!

This analogy is just about OR versus AND. In our symmetric permutation distribution, the two tails are equal in theory; the unequal values above are only a thought experiment to make the logic easier to see.

Who Eats More Chicken Wings? | Zhu Xuelin

Are Permutation Histograms Always Bell-Shaped?

A common misconception

Choosing a Quantity for a Permutation Test

What should we recalculate?

Two stocks, five daily returns

Option 1: the signed difference

Option 2: the absolute difference

Why not use the ratio?

So, which quantity should we use?

Why Use the Smaller Tail?

Why define the two-sided p-value as 2 × min(p_left, p_right) instead of using max?

Why define the two-sided p-value as `2 × min(p_left, p_right)` instead of using `max`?