Linear Regression

$$\begin{array}{rl}& Y\approx {\beta }_0+{\beta }_{1}x\\ & Y={\beta }_0+{\beta }_1{x}_i+{ϵ}_i\end{array}$$

• iid ${ϵ}_i\sim N(0,{\sigma }^2)$
• Dependent variable $Y$ (response, regressand)
• Independent variable $x$, (explanatory variable, regressor)
• ${\beta }_1$ is slope
• ${\beta }_0$ is the y-intercept
• $Y$ is an rv, for this course $x$ is not random

Least Square

• Find the straight line such that the total deviations from the sample points is the smallest

$$L({\beta }_0,{\beta }_1)=\sum _{i=1}^n\left[y_i-({\beta }_0+{\beta }_1{x}_i{)}^2\right]$$ $$\widehat{{\beta }_0},\widehat{{\beta }_1}=\arg \underset{{\beta }_0,{\beta }_1}{\min }L({\beta }_0,{\beta }_1)$$ $$\begin{array}{rl}& \widehat{{\beta }_0}=\bar{y}-\widehat{{\beta }_1}\bar{x}\\ & \widehat{{\beta }_1}=\frac{{\sum }_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}=\frac{S_{xy}}{S_{xx}}\end{array}$$

We use square because we want to use derivative to find the minimum

# By hand
dx = x-mean(x)
dy = y-mean(y)
 
sxy = sum(dx*dy)
sxx = sum(dx^2)
 
beta1 = sxy/sxx
beta0 = mean(y) - beta1*mean(x)
 
plot(x, y, xlab="Father's height", ylab="Son's height")
abline(a = beta0,b=beta1, col=“red")

# By function (y depends on x)
lm(y~x)

Estimators

$\widehat{{\beta }_0}$ and $\widehat{{\beta }_1}$ are unbiased estimator

$$\mathrm{Var}()$$

CI

• known sigma^2
  • for beta1

$$\left[\widehat{{\beta }_1}-z_{\alpha /2}\frac{\sigma }{\sqrt{S_{xx}}},\widehat{{\beta }_1}+z_{\alpha /2}\frac{\sigma }{\sqrt{S_{xx}}}\right]$$

• unknown sigma

Decomposition of Variance

$$\begin{array}{rlrl}& SST=\sum _{i=1}^n(y_i-\bar{y}{)}^2& & \text{df}=n-1\\ & RSS=\sum _{i=1}^n(\widehat{y_i}-\bar{y}{)}^2& & \text{df}=2-1\\ & SSE=\sum _{i=1}^n(y_i-\widehat{y_i}{)}^2& & \text{df}=n-2\end{array}$$ $$\begin{array}{rlrl}& SST=RSS+SSE& & \text{df}=n-1=(2-1)+(n-1)\end{array}$$ $$R^2=\frac{RSS}{SST}=\frac{{\left[{\sum }_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})\right]}^2}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2{\sum }_{i=1}^n(y_i-\bar{y}{)}^2}$$