Estimations

Bias

$$\mathrm{Bias}(\hat{\theta },\theta )=E(\hat{\theta })-\theta$$

If $\mathrm{Bias}(\hat{\theta },\theta )=0$ for all $\theta$, then the estimator $\hat{\theta }$ is unbiased

Checking using expectation

Accuracy

Mean Square Error (MSE)

$$\mathrm{MSE}(\hat{\theta },\theta )=E[(\hat{\theta }-\theta {)}^2]=(\mathrm{Bias}(\hat{\theta },\theta ){)}^2+\mathrm{Var}(\hat{\theta })$$

When $n\to +\infty$, if MSE is smaller, the estimator is consistent

Check using variance

Precision

Sample Mean Estimator

$$\bar{X}=\frac{X_1+X_2+\cdots +X_n}{n}$$

Unbiased estimator of the population mean ${\mu }_X=E(X)$

$$Var(\bar{X})=\frac{{\sigma }_X^2}{n}$$

The precision of the sample mean estimator increases with the sample size n

Sample Variance Estimator

• Sample variance

$$S_{n-1}^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-\bar{X}{)}^2$$

• Sample variance is an unbiased estimator

$$E(S_{n-1}^2)={\sigma }_X^2$$

Note that

$$\mathrm{Var}(X)=E(X^2)-(E(X){)}^2$$

We take $X=X_i-\bar{X}$, and since it is unbiased so $E(X)=0$, $\mathrm{Var}(X)=E(X^2)$

$$\begin{array}{rl}E(S_{n-1}^2)& =E\left(\frac{1}{n-1}\sum _{i=1}^n(X_i-\bar{X}{)}^2\right)\\ & =\frac{1}{n-1}\sum _{i=1}^{n}E(X_i-\bar{X}{)}^2\\ & =\frac{1}{n-1}\sum _{i=1}^n\mathrm{Var}(X_i-\bar{X})\\ & =\frac{1}{n-1}\sum _{i=1}^n(\frac{n-1}{n}{\sigma }^2)\\ & ={\sigma }_X^2\end{array}$$

• It satisfy the following, where ${\sigma }^2=E[(X-\mu {)}^2]$, \mu_{4}=E[(X-\mu)^{4}]$,$\mu=E(X)$

$$\mathrm{Var}(S_{n-1}^2)=\frac{1}{n}\left({\mu }_4-\frac{n-3}{n-2}({\sigma }^2{)}^2\right)$$

Maximum Likelihood Estimator (MLE)

Value of $\theta$ that maximize $L(\theta \mid X_1,X_2,\dots X_n)$, i.e.

$$\hat{\theta }=\mathrm{argmax}p(X_1,X_2,\dots ,X_n)$$

It can also be written as the following by taking log

$$\hat{\theta }=\mathrm{argmax}\left(\prod _{i=1}^{n}p(X_i)\right)$$

Poison Distribution

$$\hat{\lambda }=\bar{X}$$

Normal Distribution

$$\hat{\mu }=\bar{X}$$

Central Limit Theorem

$$\underset{n\to \infty }{\lim }\frac{\bar{X}-\mu }{\sqrt{{\sigma }^2/n}}\to N(0,1)$$

cdf of $Z=\frac{\bar{X}-\mu }{\sqrt{{\sigma }^2/n}}$

$F_Z(x)\to \Phi (x)$

Confidence Interval

Alpha and critical value

• C is the x%
• Alpha is (1 - middle part) / 2, i.e. lower/upper tail

$$\alpha =\frac{1-C}{2}$$

$\bar{X}$ when ${\sigma }^2$ is known

• $z_{\alpha }={\Phi }^{-1}(1-\alpha )$, $P(X\le x)$
  • qnorm(1 - alpha)
• $z_{\alpha }=-{\Phi }^{-1}(\alpha )$, $P(X\le x)$
  • -qnorm(alpha)
• $P(X>x)$
  • qnorm(alpha, lower.tail=F)

$$P\left(\bar{X}-z_{\alpha }\frac{\sigma }{\sqrt{n}}\le \mu \le \bar{X}+z_{\alpha }\frac{\sigma }{\sqrt{n}}\right)=C$$ $$\left[\bar{X}-z_{\alpha }\frac{\sigma }{\sqrt{n}},\bar{X}+z_{\alpha }\frac{\sigma }{\sqrt{n}}\right]$$

$\bar{X}$ when ${\sigma }^2$ is unknown

• Default of cdf
  • qt(1 - alpha, df=n-1)
• $P(X>x)$ instead of $P(X\le x)$
  • qt(alpha, df=n-1, lower.tail=F)
• By symmetry
  • -qt(alpha, df=n-1)

$$\begin{array}{rl}& P\left(-t_{n-1,\alpha }\le \frac{\bar{X}-\mu }{S_{n-1}/\sqrt{n}}\le t_{n-1,\alpha }\right)=C\\ & P\left(\bar{X}-t_{n-1,\alpha }\frac{S_{n-1}}{\sqrt{n}}\le \mu \le \bar{X}+t_{n-1,\alpha }\frac{S_{n-1}}{\sqrt{n}}\right)=C\end{array}$$ $$\left[\bar{X}-t_{n-1,\alpha }\frac{S_{n-1}}{\sqrt{n}},\bar{X}+t_{n-1,\alpha }\frac{S_{n-1}}{\sqrt{n}}\right]$$

$S_{n-1}^2$

• ${\chi }_{n-1,1-\alpha }^2$
  • Default of cdf (lower tail = alpha)
    • qchisq(alpha, df=n-1, lower.tail=T)
  • Split at 1-a mark, but want RHS area
    • qchisq(1-alpha, df=n-1, lower.tail=F)
• ${\chi }_{n-1,\alpha }^2$
  • Default of cdf (lower tail = 1-alpha)
    • qchisq(1-alpha, df=n-1, lower.tail=T)
  • Split at a mark, but want RHS area
    • qchisq(alpha, df=n-1, lower.tail=F)

$$\begin{array}{rl}& P\left({\chi }_{n-1,1-\alpha }^2\le \frac{(n-1)S_{n-1}^2}{{\sigma }^2}\le {\chi }_{n-1,\alpha }^2\right)=C\\ & P\left(\frac{(n-1)S_{n-1}^2}{{\chi }_{n-1,\alpha }^2}\le {\sigma }^2\le \frac{(n-1)S_{n-1}^2}{{\chi }_{n-1,1-\alpha }^2}\right)=C\end{array}$$ $$[\frac{(n-1)S_{n-1}^2}{{\chi }_{n-1,\alpha }^2},\frac{(n-1)S_{n-1}^2}{{\chi }_{n-1,1-\alpha }^2}]$$

Warning

In the graph, left is $1-\alpha$, right is $\alpha$
In CI, left is $\alpha$, right is $1-\alpha$