Large Sample Theory for MLEs

Introduction Consider the following:

• i.i.d. sample X_{1}, X_{2}, ..., X_{n}

• one-dimensional paramter θ with l(θ) = ∑_{i=1}^{n} log(f(x_{i}|θ)

• the true value of θ is θ_{0}

\underline{Thm A:} Under appropriate smoothness conditions on f, the MLE of θ from an i.i.d. sample is consistent, i.e.,

θ̂ ⟶^{p} θ_{0}, n → ∞.  (1)

\underline{Proof:} (The idea) First, think of the transformed variable logf(X|θ) and consider logf(X_{1}|θ), logf(X_{2}|θ), ..., logf(X_{n}|θ) as a random sample. So its mean is

\frac{1}{n} ∑_{i=1}^{n} logf(X_{i}|θ) ⟶^{p} E_{θ_{0}}[logf(X_{i}|θ)] n → ∞.  (2)

The subscript on θ, θ_{0}, is to identify the true unknown value of θ. With some imagination (or doing Calculus) we can guess the maximum of the righthand side should be at θ_{0} since the convergence says that the lefthand side and the righthand side will be close for large n. \underline{Lemma A:} Define I(θ), Fisher’s Information, by

I(θ) = E [\frac{d}{dθ} logf(X|θ)]^{2} .  (3)

Under the appropriate smoothness conditions on f, I(θ) may be represented as

I(θ) = −E [\frac{d^{2}}{dθ^{2}} logf(X|θ)] .  (4)

\underline{Proof:} First recall that for any density funtion

∫ f(x|θ)dx = 1  (5)

and so

\frac{d}{dθ} ∫ f(x|θ)dx = 0.  (6)

Note that since

\frac{d}{dx}log(g(x)) = \frac{\frac{d}{dx}g(x)}{g(x)},  (7)

then

\frac{d}{dθ} f(x|θ) = [\frac{d}{dθ} logf(x|θ)] f(x|θ).  (8)

So

0 = \frac{d}{dθ} ∫ f(x|θ)dx
= ∫ \frac{d}{dθ} f(x|θ)dx
= ∫ [\frac{d}{dθ} logf(x|θ)] f(x|θ)dx

The moving of the derivative inside the integral takes some math. Now take the second derivative.

0 = \frac{d}{dθ} ∫ [\frac{d}{dθ} logf(x|θ)] f(x|θ)dx
= ∫ [\frac{d^{2}}{dθ^{2}} logf(x|θ)] f(x|θ)dx + ∫ [\frac{d}{dθ} logf(x|θ)] \frac{d}{dθ} f(x|θ)dx

= ∫ [\frac{d^{2}}{dθ^{2}} logf(x|θ)] f(x|θ)dx + ∫ [\frac{d}{dθ} logf(x|θ)]^{2} f(x|θ)dx

= E [\frac{d^{2}}{dθ^{2}} logf(X|θ)] + E [\frac{d}{dθ} logf(X|θ)]^{2} .

Remarks about Fisher’s Information

For the one-parameter model f(x|θ) the Fisher Information I(θ) is the fundamental measure of the “information” an observation X carries about θ.

E [\frac{d^{2}}{dθ^{2}} logf(X|θ)] = −E [\frac{d}{dθ} logf(X|θ)]^{2} .  (9)

1. I_{X}(θ) is best thought of in relative terms, i.e., X is more informative about θ than Y if I_{X}(θ) > I_{Y} (θ).

2. The more information that X contains about θ, the larger I_{X}(θ) should be. If X = θ with probability one, then we would like to have I_{X}(θ) = ∞.

3. If X_{1}, X_{2}, ..., X_{n} i.i.d. F_{θ}, then

I_{X}(θ) = nI_{X}(θ)  (10)

where X = (X_{1}, X_{2}, ..., X_{n}).

The information of n X’s is n times the information from one X and as n → ∞, the information goes to ∞, and is eventually perfect. \underline{Proof:}

I_{X}(θ) = −E [\frac{d^{2}}{dθ^{2}} logf(X_{1}, X_{2}, ..., X_{n}|θ)]
= −E [\frac{d^{2}}{dθ^{2}} log ∏_{i=1}^{n} f(X_{i}|θ)]
= −E [\frac{d^{2}}{dθ^{2}} ∑_{i=1}^{n} logf(X_{i}|θ)]
= ∑_{i=1}^{n} [−E [\frac{d^{2}}{dθ^{2}} logf(X_{i}|θ)]]
= ∑_{i=1}^{n} I_{X}(θ)
= nI_{X}(θ)

The large sample distribution of an MLE is approximately normal with mean θ_{0} and variance \frac{1}{nI(θ_{0})}. This implies that the MLE is asymptotically unbiased. It also implies that the variance of the limiting normal distribution has the asymptotic variance of the MLE. \underline{Thm B:} Under the smoothness conditions of f, the probability distribution of

\sqrt{nI(θ_{0})}(θ̂ − θ_{0})  (11)

tends to a standard normal distribution. \underline{Proof:} Use the W.L.L.N., the C.L.T. and Lemma A. So we have

θ̂ ∼ N (θ_{0}, \frac{1}{nI(θ_{0})})  (12)

and approximately

θ̂ ∼ N (θ_{0}, \frac{1}{nI(θ̂)})  (13)

from which we can produce an approximate 100(1 − α) confidence interval for θ

θ̂ ± z_{α/2} \frac{1}{\sqrt{nI(θ̂)}}  (14)

Notes about the smoothness conditions The true parameter value, θ_{0}, is required to be an interior point of the set of all parameter values. It is also required that the support of the density f(x|θ) does not depend of θ. Final notes The asymptotic variance of the MLE θ̂ is

\frac{1}{nI(θ_{0})} = −\frac{1}{E [\frac{d^{2}}{dθ^{2}} l(θ)]}.  (15)