In my classes, I use one "simple" situation that might help you wonder and perhaps develop a gut feeling for what a degree of freedom may mean.
It is kind of a "Forrest Gump" approach to the subject, but it is worth the try.
Consider you have 10 independent observations X1,X2,…,X10∼N(μ,σ2) that came right from a normal population whose mean μ and variance σ2 are unknown.
Your observations bring to you collectively information both about μ and σ2. After all, your observations tend to be spread around one central value, which ought to be close to the actual and unknown value of μ and, likewise, if μ is very high or very low, then you can expect to see your observations gather around a very high or very low value respectively. One good "substitute" for μ (in the absence of knowledge of its actual value) is X¯, the average of your observation.
Also, if your observations are very close to one another, that is an indication that you can expect that σ2 must be small and, likewise, if σ2 is very large, then you can expect to see wildly different values for X1 to X10.
If you were to bet your week's wage on which should be the actual values of μ and σ2, you would need to choose a pair of values in which you would bet your money. Let's not think of anything as dramatic as losing your paycheck unless you guess μ correctly until its 200th decimal position. Nope. Let's think of some sort of prizing system that the closer you guess μ and σ2 the more you get rewarded.
In some sense, your better, more informed, and more polite guess for μ's value could be X¯. In that sense, you estimate that μ must be some value around X¯. Similarly, one good "substitute" for σ2 (not required for now) is S2, your sample variance, which makes a good estimate for σ.
If your were to believe that those substitutes are the actual values of μ and σ2, you would probably be wrong, because very slim are the chances that you were so lucky that your observations coordinated themselves to get you the gift of X¯ being equal to μ and S2 equal to σ2. Nah, probably it didn't happen.
But you could be at different levels of wrong, varying from a bit wrong to really, really, really miserably wrong (a.k.a., "Bye-bye, paycheck; see you next week!").
Ok, let's say that you took X¯ as your guess for μ. Consider just two scenarios: S2=2 and S2=20,000,000. In the first, your observations sit pretty and close to one another. In the latter, your observations vary wildly. In which scenario you should be more concerned with your potential losses? If you thought of the second one, you're right. Having a estimate about σ2 changes your confidence on your bet very reasonably, for the larger σ2 is, the wider you can expect X¯ to variate.
But, beyond information about μ and σ2, your observations also carry some amount of just pure random fluctuation that is not informative neither about μ nor about σ2.
How can you notice it?
Well, let's assume, for sake of argument, that there is a God and that He has spare time enough to give Himself the frivolity of telling you specifically the real (and so far unknown) values of both μ and σ.
And here is the annoying plot twist of this lysergic tale: He tells it to you after you placed your bet. Perhaps to enlighten you, perhaps to prepare you, perhaps to mock you. How could you know?
Well, that makes the information about μ and σ2 contained in your observations quite useless now. Your observations' central position X¯ and variance S2 are no longer of any help to get closer to the actual values of μ and σ2, for you already know them.
One of the benefits of your good acquaintance with God is that you actually know by how much you failed to guess correctly μ by using X¯, that is, (X¯−μ) your estimation error.
Xi∼N(μ,σ2)X¯∼N(μ,σ2/10)(X¯−μ)∼N(0,σ2/10)
X¯−μσ/10−−√∼N(0,1)
(guess what? trust me in that one as well), which carries absolutely no information about
μ or
σ2.
You know what? If you took any of your individual observations as a guess for μ, your estimation error (Xi−μ) would be distributed as N(0,σ2). Well, between estimating μ with X¯ and any Xi, choosing X¯ would be better business, because Var(X¯)=σ2/10<σ2=Var(Xi), so X¯ was less prone to be astray from μ than an individual Xi.
Anyway, (Xi−μ)/σ∼N(0,1) is also absolutely non informative about neither μ nor σ2.
"Will this tale ever end?" you may be thinking. You also may be thinking "Is there any more random fluctuation that is non informative about μ and σ2?".
[I prefer to think that you are thinking of the latter.]
Yes, there is!
The square of your estimation error for μ with Xi divided by σ,
(Xi−μ)2σ2=(Xi−μσ)2∼χ2
has a Chi-squared distribution, which is the distribution of the square
Z2 of a standard Normal
Z∼N(0,1), which I am sure you noticed has absolutely no information about either
μ nor
σ2, but conveys information about the variability you should expect to face.
That is a very well known distribution that arises naturally from the very scenario of you gambling problem for every single one of your ten observations and also from your mean:
(X¯−μ)2σ2/10=(X¯−μσ/10−−√)2=(N(0,1))2∼χ2
and also from the gathering of your ten observations' variation:
∑i=110(Xi−μ)2σ2/10=∑i=110(Xi−μσ/10−−√)2=∑i=110(N(0,1))2=∑i=110χ2.
Now that last guy doesn't have a Chi-squared distribution, because he is the sum of ten of those Chi-squared distributions, all of them independent from one another (because so are
X1,…,X10). Each one of those single Chi-squared distribution is one contribution to the amount of random variability you should expect to face, with roughly the same amount of contribution to the sum.
The value of each contribution is not mathematically equal to the other nine, but all of them have the same expected behavior in distribution. In that sense, they are somehow symmetric.
Each one of those Chi-square is one contribution to the amount of pure, random variability you should expect in that sum.
If you had 100 observations, the sum above would be expected to be bigger just because it have more sources of contibutions.
Each of those "sources of contributions" with the same behavior can be called degree of freedom.
Now take one or two steps back, re-read the previous paragraphs if needed to accommodate the sudden arrival of your quested-for degree of freedom.
Yep, each degree of freedom can be thought of as one unit of variability that is obligatorily expected to occur and that brings nothing to the improvement of guessing of μ or σ2.
The thing is, you start to count on the behavior of those 10 equivalent sources of variability. If you had 100 observations, you would have 100 independent equally-behaved sources of strictly random fluctuation to that sum.
That sum of 10 Chi-squares gets called a Chi-squared distributions with 10 degrees of freedom from now on, and written χ210. We can describe what to expect from it starting from its probability density function, that can be mathematically derived from the density from that single Chi-squared distribution (from now on called Chi-squared distribution with one degree of freedom and written χ21), that can be mathematically derived from the density of the normal distribution.
"So what?" --- you might be thinking --- "That is of any good only if God took the time to tell me the values of μ and σ2, of all the things He could tell me!"
Indeed, if God Almighty were too busy to tell you the values of μ and σ2, you would still have that 10 sources, that 10 degrees of freedom.
Things start to get weird (Hahahaha; only now!) when you rebel against God and try and get along all by yourself, without expecting Him to patronize you.
You have X¯ and S2, estimators for μ and σ2. You can find your way to a safer bet.
You could consider calculating the sum above with X¯ and S2 in the places of μ and σ2:
∑i=110(Xi−X¯)2S2/10=∑i=110(Xi−X¯S/10−−√)2,
but that is not the same as the original sum.
"Why not?" The term inside the square of both sums are very different. For instance, it is unlikely but possible that all your observations end up being larger than μ, in which case (Xi−μ)>0, which implies ∑10i=1(Xi−μ)>0, but, by its turn, ∑10i=1(Xi−X¯)=0, because ∑10i=1Xi−10X¯=10X¯−10X¯=0.
Worse, you can prove easily (Hahahaha; right!) that ∑10i=1(Xi−X¯)2≤∑10i=1(Xi−μ)2 with strict inequality when at least two observations are different (which is not unusual).
"But wait! There's more!"
Xi−X¯S/10−−√
doesn't have standard normal distribution,
(Xi−X¯)2S2/10
doesn't have Chi-squared distribution with one degree of freedom,
∑i=110(Xi−X¯)2S2/10
doesn't have Chi-squared distribution with 10 degrees of freedom
X¯−μS/10−−√
doesn't have standard normal distribution.
"Was it all for nothing?"
No way. Now comes the magic! Note that
∑i=110(Xi−X¯)2σ2=∑i=110[Xi−μ+μ−X¯]2σ2=∑i=110[(Xi−μ)−(X¯−μ)]2σ2=∑i=110(Xi−μ)2−2(Xi−μ)(X¯−μ)+(X¯−μ)2σ2=∑i=110(Xi−μ)2−(X¯−μ)2σ2=∑i=110(Xi−μ)2σ2−∑i=110(X¯−μ)2σ2=∑i=110(Xi−μ)2σ2−10(X¯−μ)2σ2=∑i=110(Xi−μ)2σ2−(X¯−μ)2σ2/10
or, equivalently,
∑i=110(Xi−μ)2σ2=∑i=110(Xi−X¯)2σ2+(X¯−μ)2σ2/10.
Now we get back to those known faces.
The first term has Chi-squared distribution with 10 degrees of freedom and the last term has Chi-squared distribution with one degree of freedom(!).
We simply split a Chi-square with 10 independent equally-behaved sources of variability in two parts, both positive: one part is a Chi-square with one source of variability and the other we can prove (leap of faith? win by W.O.?) to be also a Chi-square with 9 (= 10-1) independent equally-behaved sources of variability, with both parts independent from one another.
This is already a good news, since now we have its distribution.
Alas, it uses σ2, to which we have no access (recall that God is amusing Himself on watching our struggle).
Well,
S2=110−1∑i=110(Xi−X¯)2,
so
∑i=110(Xi−X¯)2σ2=∑10i=1(Xi−X¯)2σ2=(10−1)S2σ2∼χ2(10−1)
therefore
X¯−μS/10−−√=X¯−μσ/10√Sσ=X¯−μσ/10√S2σ2−−−√=X¯−μσ/10√(10−1)S2σ2(10−1)−−−−−−√=N(0,1)χ2(10−1)(10−1)−−−−−√,
which is a distribution that is not the standard normal, but whose density can be derived from the densities of the standard normal and the Chi-squared with
(10−1) degrees of freedom.
One very, very smart guy did that math[^1] in the beginning of 20th century and, as an unintended consequence, he made his boss the absolute world leader in the industry of Stout beer. I am talking about William Sealy Gosset (a.k.a. Student; yes, that Student, from the t distribution) and Saint James's Gate Brewery (a.k.a. Guinness Brewery), of which I am a devout.
[^1]: @whuber told in the comments below that Gosset did not do the math, but guessed instead! I really don't know which feat is more surprising for that time.
That, my dear friend, is the origin of the t distribution with (10−1) degrees of freedom. The ratio of a standard normal and the squared root of an independent Chi-square divided by its degrees of freedom, which, in an unpredictable turn of tides, wind up describing the expected behavior of the estimation error you undergo when using the sample average X¯ to estimate μ and using S2 to estimate the variability of X¯.
There you go. With an awful lot of technical details grossly swept behind the rug, but not depending solely on God's intervention to dangerously bet your whole paycheck.
