當我們有 n n n k k k
Y 1 = β 0 + β 1 X 11 + β 2 X 12 + ⋯ + β k X 1 k + ε i Y 2 = β 0 + β 1 X 21 + β 2 X 22 + ⋯ + β k X 2 k + ε 2 ⋮ Y n = β 0 + β 1 X n 1 + β 2 X n 2 + ⋯ + β k X n k + ε n \begin{align*}
Y_1=&\beta_0+\beta_1X_{11}+\beta_2X_{12}+\cdots+\beta_kX_{1k}+\varepsilon_i\\
Y_2=&\beta_0+\beta_1X_{21}+\beta_2X_{22}+\cdots+\beta_kX_{2k}+\varepsilon_2\\
&\vdots\\
Y_n=&\beta_0+\beta_1X_{n1}+\beta_2X_{n2}+\cdots+\beta_kX_{nk}+\varepsilon_n
\end{align*} Y 1 = Y 2 = Y n = β 0 + β 1 X 11 + β 2 X 12 + ⋯ + β k X 1 k + ε i β 0 + β 1 X 21 + β 2 X 22 + ⋯ + β k X 2 k + ε 2 ⋮ β 0 + β 1 X n 1 + β 2 X n 2 + ⋯ + β k X nk + ε n 我們可以將這個模型寫成矩陣形式：
Y ~ = [ 1 X 11 X 12 ⋯ X 1 k 1 X 21 X 22 ⋯ X 2 k ⋮ ⋮ ⋮ ⋱ ⋮ 1 X n 1 X n 2 ⋯ X n k ] ⏟ Design Matrix β ~ + ε ~ \utilde{Y}=\underbrace{\begin{bmatrix}
1 & X_{11} & X_{12} & \cdots & X_{1k}\\
1 & X_{21} & X_{22} & \cdots & X_{2k}\\
\vdots & \vdots & \vdots & \ddots & \vdots\\
1 & X_{n1} & X_{n2} & \cdots & X_{nk}
\end{bmatrix}}_{\text{Design Matrix}}\utilde{\beta}+\utilde{\varepsilon} Y = Design Matrix 1 1 ⋮ 1 X 11 X 21 ⋮ X n 1 X 12 X 22 ⋮ X n 2 ⋯ ⋯ ⋱ ⋯ X 1 k X 2 k ⋮ X nk β + ε 因為 X ~ \utilde{X} X β ~ \utilde{\beta} β D D D
⟹ Y ~ n × 1 = D n × p β ~ p × 1 + ε ~ n × 1 \implies \utilde{Y}_{n\times 1}=D_{n\times p}\utilde{\beta}_{p\times 1}+\utilde{\varepsilon}_{n\times 1} ⟹ Y n × 1 = D n × p β p × 1 + ε n × 1 矩陣形式下的基礎定義和結論
W n × 1 = [ W 1 W 2 ⋮ W n ] : random vector U I × J = [ U 11 U 12 ⋯ U 1 J U 21 U 22 ⋯ U 2 J ⋮ ⋮ ⋱ ⋮ U I 1 U I 2 ⋯ U I J ] : random matrix W_{n\times 1} = \begin{bmatrix}
W_1\\
W_2\\
\vdots\\
W_n
\end{bmatrix}: \text{ random vector}\quad
U_{I\times J} = \begin{bmatrix}
U_{11} & U_{12} & \cdots & U_{1J}\\
U_{21} & U_{22} & \cdots & U_{2J}\\
\vdots & \vdots & \ddots & \vdots\\
U_{I1} & U_{I2} & \cdots & U_{IJ}
\end{bmatrix}: \text{ random matrix} W n × 1 = W 1 W 2 ⋮ W n : random vector U I × J = U 11 U 21 ⋮ U I 1 U 12 U 22 ⋮ U I 2 ⋯ ⋯ ⋱ ⋯ U 1 J U 2 J ⋮ U I J : random matrix ⟹ E [ W ] = [ E [ W 1 ] E [ W 2 ] ⋮ E [ W n ] ] E [ U ] = [ E [ U 11 ] E [ U 12 ] ⋯ E [ U 1 J ] E [ U 21 ] E [ U 22 ] ⋯ E [ U 2 J ] ⋮ ⋮ ⋱ ⋮ E [ U I 1 ] E [ U I 2 ] ⋯ E [ U I J ] ] \implies E[W] = \begin{bmatrix}
E[W_1]\\
E[W_2]\\
\vdots\\
E[W_n]
\end{bmatrix}\qquad
E[U] = \begin{bmatrix}
E[U_{11}] & E[U_{12}] & \cdots & E[U_{1J}]\\
E[U_{21}] & E[U_{22}] & \cdots & E[U_{2J}]\\
\vdots & \vdots & \ddots & \vdots\\
E[U_{I1}] & E[U_{I2}] & \cdots & E[U_{IJ}]
\end{bmatrix} ⟹ E [ W ] = E [ W 1 ] E [ W 2 ] ⋮ E [ W n ] E [ U ] = E [ U 11 ] E [ U 21 ] ⋮ E [ U I 1 ] E [ U 12 ] E [ U 22 ] ⋮ E [ U I 2 ] ⋯ ⋯ ⋱ ⋯ E [ U 1 J ] E [ U 2 J ] ⋮ E [ U I J ] ⟹ σ 2 { W } = E [ ( W − E [ W ] ) m × 1 ( W − E [ W ] ) 1 × m t ] = [ V a r [ W 1 ] C o v [ W 1 , W 2 ] ⋯ C o v [ W 1 , W n ] C o v [ W 2 , W 1 ] V a r [ W 2 ] ⋯ C o v [ W 2 , W n ] ⋮ ⋮ ⋱ ⋮ C o v [ W n , W 1 ] C o v [ W n , W 2 ] ⋯ V a r [ W n ] ] = [ V a r C o v C o v V a r ] is symmetric = Variance-Covariance Matrix of W \begin{align*}
\implies \sigma^2\set{W} &=E\left[\left(W-E[W]\right)_{m\times 1}\left(W-E[W]\right)_{1\times m} ^t\right]\\
&=\begin{bmatrix}
Var[W_1] & Cov[W_1,W_2] & \cdots & Cov[W_1,W_n]\\
Cov[W_2,W_1] & Var[W_2] & \cdots & Cov[W_2,W_n]\\
\vdots & \vdots & \ddots & \vdots\\
Cov[W_n,W_1] & Cov[W_n,W_2] & \cdots & Var[W_n]
\end{bmatrix}\\
&=\begin{bmatrix}
Var & Cov\\
Cov & Var
\end{bmatrix} \quad \text{is symmetric}\\
&=\text{ Variance-Covariance Matrix of } W
\end{align*} ⟹ σ 2 { W } = E [ ( W − E [ W ] ) m × 1 ( W − E [ W ] ) 1 × m t ] = Va r [ W 1 ] C o v [ W 2 , W 1 ] ⋮ C o v [ W n , W 1 ] C o v [ W 1 , W 2 ] Va r [ W 2 ] ⋮ C o v [ W n , W 2 ] ⋯ ⋯ ⋱ ⋯ C o v [ W 1 , W n ] C o v [ W 2 , W n ] ⋮ Va r [ W n ] = [ Va r C o v C o v Va r ] is symmetric = Variance-Covariance Matrix of W 性質 ：設 A , B , C A, B, C A , B , C W W W U U U
E[A] = A
E[AUB+C] = AE[U]B+C
σ 2 { W } = E ( w w t ) − E [ W ] ( E [ W ] ) t \sigma^2\set{W}=E(ww^t)-E[W](E[W])^t σ 2 { W } = E ( w w t ) − E [ W ] ( E [ W ] ) t σ 2 { A n × w W w × 1 } n × n = A n × w σ 2 { W } w × w A w × n t \sigma^2\set{A_{n\times w}W_{w\times 1}}_{n\times n} = A_{n\times w}\sigma^2\set{W}_{w\times w}A^t_{w\times n} σ 2 { A n × w W w × 1 } n × n = A n × w σ 2 { W } w × w A w × n t σ 2 { A W + B } = σ 2 { A W } \sigma^2\set{AW+B}=\sigma^2\set{AW} σ 2 { A W + B } = σ 2 { A W }
Note : 如果 W W W m × 1 m\times 1 m × 1 σ 2 { W } \sigma^2\set{W} σ 2 { W } m × m m\times m m × m ∀ a ~ ∈ R n \forall \utilde{a}\in\R^n ∀ a ∈ R n
a ~ t σ 2 { W } a ~ = σ 2 { a ~ t W } ≥ 0 \utilde{a}^t\sigma^2\set{W}\utilde{a}=\sigma^2\set{\utilde{a}^tW}\ge 0 a t σ 2 { W } a = σ 2 { a t W } ≥ 0 因此 a ~ t σ 2 { W } a ~ \utilde{a}^t\sigma^2\set{W}\utilde{a} a t σ 2 { W } a a ~ = 0 \utilde{a}=0 a = 0
令 f : R k → R f:\R^k\to\R f : R k → R f ( θ ~ ) ∈ R , ∀ θ = ( θ 1 , ⋯ , θ k ) t ∈ R k f(\utilde{\theta})\in\R, \forall\theta=(\theta_1, \cdots, \theta_k)^t\in\R^k f ( θ ) ∈ R , ∀ θ = ( θ 1 , ⋯ , θ k ) t ∈ R k
∂ ∂ θ ~ f ( θ ~ ) ≜ [ ∂ f ∂ θ 1 ∂ f ∂ θ 2 ⋮ ∂ f ∂ θ k ] \frac{\partial}{\partial\utilde{\theta}} f(\utilde{\theta}) \triangleq \begin{bmatrix}
\frac{\partial f}{\partial\theta_1}\\
\frac{\partial f}{\partial\theta_2}\\
\vdots\\
\frac{\partial f}{\partial\theta_k}
\end{bmatrix} ∂ θ ∂ f ( θ ) ≜ ∂ θ 1 ∂ f ∂ θ 2 ∂ f ⋮ ∂ θ k ∂ f Given c ~ ∈ R k \utilde{c}\in\R^k c ∈ R k f ( θ ~ ) = c ~ t θ ~ = θ ~ t c ~ f(\utilde{\theta})=\utilde{c}^t\utilde{\theta}=\utilde{\theta}^t\utilde{c} f ( θ ) = c t θ = θ t c ∀ θ ~ ∈ R k \forall \utilde{\theta}\in\R^k ∀ θ ∈ R k
∂ ∂ θ ~ f ( θ ~ ) = c ~ \frac{\partial}{\partial\utilde{\theta}}f(\utilde{\theta})=\utilde{c} ∂ θ ∂ f ( θ ) = c i.e. ∂ ∂ θ ~ ( c ~ t θ ~ ) = ∂ ∂ θ ~ ( θ ~ t c ~ ) = c ~ \frac{\partial}{\partial\utilde{\theta}}(\utilde{c}^t\utilde{\theta})=\frac{\partial}{\partial\utilde{\theta}}(\utilde{\theta}^t\utilde{c})=\utilde{c} ∂ θ ∂ ( c t θ ) = ∂ θ ∂ ( θ t c ) = c
如果 A A A k × k k\times k k × k
f ( θ ~ ) = θ ~ t A θ ~ = ∑ i , j θ i A i j θ j f(\utilde{\theta})=\utilde{\theta}^tA\utilde{\theta}=\sum_{i,j}\theta_iA_{ij}\theta_j f ( θ ) = θ t A θ = i , j ∑ θ i A ij θ j 並且
∂ ∂ θ ~ f ( θ ~ ) = 2 A θ ~ \frac{\partial}{\partial\utilde{\theta}}f(\utilde{\theta})=2A\utilde{\theta} ∂ θ ∂ f ( θ ) = 2 A θ 如果 A A A ∂ ∂ θ ~ f ( θ ~ ) = A θ ~ + A t θ ~ \frac{\partial}{\partial\utilde{\theta}}f(\utilde{\theta})=A\utilde{\theta}+A^t\utilde{\theta} ∂ θ ∂ f ( θ ) = A θ + A t θ
因此在矩陣形式下的一般線性回歸模型會有：
Y ~ n × 1 = D n × p β ~ p × 1 + ε ~ n × 1 with E [ ε ~ ] = 0 , σ 2 { ε ~ } = [ σ 2 ⋯ 0 ⋮ ⋱ ⋮ 0 ⋯ σ 2 ] = σ 2 I n × n \utilde{Y}_{n\times 1}=D_{n\times p}\utilde{\beta}_{p\times 1}+\utilde{\varepsilon}_{n\times 1}\text{ with } E[\utilde{\varepsilon}]=0, \sigma^2\set{\utilde{\varepsilon}}=\begin{bmatrix}
\sigma^2& \cdots &0\\
\vdots & \ddots & \vdots\\
0 & \cdots & \sigma^2
\end{bmatrix}
=\sigma^2I_{n\times n} Y n × 1 = D n × p β p × 1 + ε n × 1 with E [ ε ] = 0 , σ 2 { ε } = σ 2 ⋮ 0 ⋯ ⋱ ⋯ 0 ⋮ σ 2 = σ 2 I n × n ⟹ E [ Y ~ ] = E [ D β ~ + ε ~ ] = D β ~ = regression function σ 2 { Y ~ } = σ 2 { D β ~ + ε ~ } = σ 2 { ε ~ } = σ 2 I n × n \begin{align*}
\implies &E[\utilde{Y}]=E[D\utilde{\beta}+\utilde{\varepsilon}]=D\utilde{\beta}=\text{ regression function}\\
&\sigma^2\set{\utilde{Y}}=\sigma^2\set{D\utilde{\beta}+\utilde{\varepsilon}}=\sigma^2\set{\utilde{\varepsilon}}=\sigma^2I_{n\times n}
\end{align*} ⟹ E [ Y ] = E [ D β + ε ] = D β = regression function σ 2 { Y } = σ 2 { D β + ε } = σ 2 { ε } = σ 2 I n × n Q ( β ~ ) ≜ ∣ ∣ Y ~ − E [ Y ~ ] ∣ ∣ 2 = ∣ ∣ Y ~ − D β ~ ∣ ∣ 2 = ( Y ~ − D β ~ ) t ( Y ~ − D β ~ ) \begin{align*}
Q(\utilde{\beta})\triangleq& ||\utilde{Y}-E[\utilde{Y}]||^2\\
&=||\utilde{Y}-D\utilde{\beta}||^2\\
&=(\utilde{Y}-D\utilde{\beta})^t(\utilde{Y}-D\utilde{\beta})\\
\end{align*} Q ( β ) ≜ ∣∣ Y − E [ Y ] ∣ ∣ 2 = ∣∣ Y − D β ∣ ∣ 2 = ( Y − D β ) t ( Y − D β ) 如果 Q ( b ~ ) = min β ~ ∈ R n Q(\utilde{b})=\min_{\utilde{\beta}\in\R^n} Q ( b ) = min β ∈ R n b ~ \utilde{b} b β ~ \utilde{\beta} β
注意到
Q ( β ~ ) = Y ~ 1 × n t Y ~ n × 1 − Y ~ 1 × n t D n × p β ~ p × 1 − β ~ 1 × p t D p × n t Y ~ n × 1 + β ~ 1 × p t D p × n t D n × p β ~ p × 1 = Y ~ t Y ~ − 2 β ~ t D t Y ~ + β ~ t D t D β ~ \begin{align*}
Q(\utilde{\beta})&=\utilde{Y}^t_{1\times n}\utilde{Y}_{n\times 1}-\utilde{Y}^t_{1\times n}D_{n\times p}\utilde{\beta}_{p\times 1}-\utilde{\beta}^t_{1\times p}D^t_{p\times n}\utilde{Y}_{n\times 1}+\utilde{\beta}^t_{1\times p}D^t_{p\times n}D_{n\times p}\utilde{\beta}_{p\times 1}\\
&=\utilde{Y}^t\utilde{Y}-2\utilde{\beta}^tD^t\utilde{Y}+\utilde{\beta}^tD^tD\utilde{\beta}\\
\end{align*} Q ( β ) = Y 1 × n t Y n × 1 − Y 1 × n t D n × p β p × 1 − β 1 × p t D p × n t Y n × 1 + β 1 × p t D p × n t D n × p β p × 1 = Y t Y − 2 β t D t Y + β t D t D β 是 p × 1 p\times 1 p × 1
∂ ∂ β ~ Q ( β ~ ) = − 2 D t Y ~ + 2 D t D β ~ \frac{\partial}{\partial\utilde{\beta}}Q(\utilde{\beta})=-2D^t\utilde{Y}+2D^tD\utilde{\beta} ∂ β ∂ Q ( β ) = − 2 D t Y + 2 D t D β 如果 b ~ \utilde{b} b β ~ \utilde{\beta} β ⟹ \implies ⟹ ∂ ∂ β ~ Q ( β ~ ) ∣ b ~ = 0 ⟺ − 2 D t Y ~ + 2 D t D b ~ = 0 \frac{\partial}{\partial\utilde{\beta}}Q(\utilde{\beta})|_{\utilde{b}}=0 \iff -2D^t\utilde{Y}+2D^tD\utilde{b}=0 ∂ β ∂ Q ( β ) ∣ b = 0 ⟺ − 2 D t Y + 2 D t D b = 0
如果 b ~ \utilde{b} b D t D b ~ = D t Y ~ D^tD\utilde{b}=D^t\utilde{Y} D t D b = D t Y β ~ \utilde{\beta} β
Q ( β ~ ) = ∣ ∣ Y ~ − D β ~ ∣ ∣ 2 = ∣ ∣ Y ~ − D b ~ + D b ~ − D β ~ ∣ ∣ 2 = ∣ ∣ Y ~ − D b ~ ∣ ∣ 2 + ∣ ∣ D b ~ − D β ~ ∣ ∣ 2 + 2 ( Y ~ − D b ~ ) t ( D b ~ − D β ~ ) ⏟ = ( D t Y ~ − D t D b ~ ) ( b ~ − β ~ ) = 0 = ∣ ∣ Y ~ − D b ~ ∣ ∣ 2 + ∣ ∣ D b ~ − D β ~ ∣ ∣ 2 ⏟ ≥ 0 ≥ Q ( b ~ ) \begin{align*}
Q(\utilde{\beta})=||\utilde{Y}-D\utilde{\beta}||^2&=||\utilde{Y}-D\utilde{b}+D\utilde{b}-D\utilde{\beta}||^2\\
&=||\utilde{Y}-D\utilde{b}||^2+||D\utilde{b}-D\utilde{\beta}||^2+2\underbrace{(\utilde{Y}-D\utilde{b})^t(D\utilde{b}-D\utilde{\beta})}_{=(D^t\utilde{Y}-D^tD\utilde{b})(\utilde{b}-\utilde{\beta})=0}\\
&=||\utilde{Y}-D\utilde{b}||^2+\underbrace{||D\utilde{b}-D\utilde{\beta}||^2}_{\ge 0}\\
&\ge Q(\utilde{b})
\end{align*} Q ( β ) = ∣∣ Y − D β ∣ ∣ 2 = ∣∣ Y − D b + D b − D β ∣ ∣ 2 = ∣∣ Y − D b ∣ ∣ 2 + ∣∣ D b − D β ∣ ∣ 2 + 2 = ( D t Y − D t D b ) ( b − β ) = 0 ( Y − D b ) t ( D b − D β ) = ∣∣ Y − D b ∣ ∣ 2 + ≥ 0 ∣∣ D b − D β ∣ ∣ 2 ≥ Q ( b ) i.e. b ~ \utilde{b} b β ~ \utilde{\beta} β ⟺ D t D b ~ = D t Y ~ \iff D^tD\utilde{b}=D^t\utilde{Y} ⟺ D t D b = D t Y β ~ \utilde{\beta} β
因此我們可以得到的結論：b ~ \utilde{b} b β ~ \utilde{\beta} β ⟺ D t D b ~ = D t Y ~ \iff D^tD\utilde{b}=D^t\utilde{Y} ⟺ D t D b = D t Y
Normal Equation :
D t D b ~ = D t Y ~ D^tD\utilde{b}=D^t\utilde{Y} D t D b = D t Y b ~ is LSE of β ~ ⟺ D t D b ~ = D t Y ~ \utilde{b}\text{ is LSE of }\utilde{\beta}\iff D^tD\utilde{b}=D^t\utilde{Y} b is LSE of β ⟺ D t D b = D t Y ⟹ Y ^ ~ ≜ D b ~ called fitted value e ~ ≜ Y ~ − Y ^ ~ called residual \implies \utilde{\hat{Y}}\triangleq D\utilde{b} \text{ called fitted value}\qquad \utilde{e}\triangleq \utilde{Y}-\utilde{\hat{Y}}\text{ called residual} ⟹ Y ^ ≜ D b called fitted value e ≜ Y − Y ^ called residual 有的時候我們並不需要 b ~ \utilde{b} b Y ^ ~ \utilde{\hat{Y}} Y ^ n n n p p p
令 θ ~ = D β ~ ⟹ E [ Y ~ ] = D β ~ = θ ~ \utilde{\theta}=D\utilde{\beta} \implies E[\utilde{Y}]=D\utilde{\beta}=\utilde{\theta} θ = D β ⟹ E [ Y ] = D β = θ Y ~ = D β ~ + ε ~ = θ ~ + ε ~ \utilde{Y}=D\utilde{\beta}+\utilde{\varepsilon}=\utilde{\theta}+\utilde{\varepsilon} Y = D β + ε = θ + ε
with θ ~ ∈ Ω ≜ { D β ~ : β ~ ∈ R p } = s p a n { 1 , X 1 , X 2 , ⋯ , X k } \begin{align*}
\text{with } \utilde{\theta}\in\Omega&\triangleq\set{D\utilde{\beta}:\utilde{\beta}\in\R^p}\\
&=span\set{1, X_1, X_2, \cdots, X_k}
\end{align*} with θ ∈ Ω ≜ { D β : β ∈ R p } = s p an { 1 , X 1 , X 2 , ⋯ , X k } 令 r = dim ( Ω ) = r a n k ( D ) ≤ p r=\dim(\Omega)=rank(D)\le p r = dim ( Ω ) = r ank ( D ) ≤ p Ω \Omega Ω R n \R^n R n r r r Ω ≜ V r \Omega\triangleq V_r Ω ≜ V r R n ≜ V n \R^n\triangleq V_n R n ≜ V n
⟹ E [ Y ~ ] = θ ~ ∈ Ω \implies E[\utilde{Y}]=\utilde{\theta}\in\Omega ⟹ E [ Y ] = θ ∈ Ω Y ^ ~ = D b ~ ∈ Ω \utilde{\hat{Y}}=D\utilde{b}\in\Omega Y ^ = D b ∈ Ω
Q ( β ) = ∣ ∣ Y ~ − D β ~ ∣ ∣ 2 = ∣ ∣ Y ~ − θ ~ ∣ ∣ 2 ⟹ Q ( b ~ ) = min β ~ ∈ R p ⟺ ∣ ∣ Y ~ − D b ~ ∣ ∣ 2 = ∣ ∣ Y ~ − Y ^ ~ ∣ ∣ 2 = min β ~ ∈ R p ∣ ∣ Y ~ − D β ~ ∣ ∣ 2 = ∣ ∣ Y ~ − θ ~ ∣ ∣ 2 \begin{align*}
& Q(\beta)=||\utilde{Y}-D\utilde{\beta}||^2=||\utilde{Y}-\utilde{\theta}||^2\\
\implies & Q(\utilde{b})=\min_{\utilde{\beta}\in\R^p}\\
\iff & ||\utilde{Y}-D\utilde{b}||^2=||\utilde{Y}-\utilde{\hat{Y}}||^2=\min_{\utilde{\beta}\in\R^p}||\utilde{Y}-D\utilde{\beta}||^2=||\utilde{Y}-\utilde{\theta}||^2
\end{align*} ⟹ ⟺ Q ( β ) = ∣∣ Y − D β ∣ ∣ 2 = ∣∣ Y − θ ∣ ∣ 2 Q ( b ) = β ∈ R p min ∣∣ Y − D b ∣ ∣ 2 = ∣∣ Y − Y ^ ∣ ∣ 2 = β ∈ R p min ∣∣ Y − D β ∣ ∣ 2 = ∣∣ Y − θ ∣ ∣ 2 i.e. Y ^ ~ ∈ Ω \utilde{\hat{Y}}\in\Omega Y ^ ∈ Ω ∣ ∣ Y ~ − Y ^ ~ ∣ ∣ 2 = min θ ~ ∈ Ω ∣ ∣ Y ~ − θ ~ ∣ ∣ 2 ||\utilde{Y}-\utilde{\hat{Y}}||^2=\min_{\utilde{\theta}\in\Omega}||\utilde{Y}-\utilde{\theta}||^2 ∣∣ Y − Y ^ ∣ ∣ 2 = min θ ∈ Ω ∣∣ Y − θ ∣ ∣ 2
之前我們討論過 Y ~ \utilde{Y} Y Y ^ ~ \utilde{\hat{Y}} Y ^
這裡 Y ~ ∈ V n \utilde{Y}\in V_n Y ∈ V n Y ^ ~ ∈ Ω \utilde{\hat{Y}}\in\Omega Y ^ ∈ Ω Y ^ ~ \utilde{\hat{Y}} Y ^ Y ~ \utilde{Y} Y Ω \Omega Ω e ~ \utilde{e} e Ω \Omega Ω V r ⊥ = Ω ⊥ V_r^\perp=\Omega^\perp V r ⊥ = Ω ⊥
i.e. ∀ Y ~ ∈ V n = R n \forall \utilde{Y}\in V_n=\R^n ∀ Y ∈ V n = R n ∃ ! w ~ ∈ Ω , z ∈ Ω ⊥ \exist!\utilde{w}\in\Omega, z\in\Omega^\perp ∃ ! w ∈ Ω , z ∈ Ω ⊥ Y ~ = w ~ + z ~ \utilde{Y}=\utilde{w}+\utilde{z} Y = w + z w ~ \utilde{w} w Y ~ \utilde{Y} Y Ω \Omega Ω
⟹ Y ~ \implies \utilde{Y} ⟹ Y Ω \Omega Ω Y ^ ~ = θ ^ ~ \utilde{\hat{Y}}=\utilde{\hat{\theta}} Y ^ = θ ^ ∣ ∣ Y ~ − θ ~ ∣ ∣ 2 ||\utilde{Y}-\utilde{\theta}||^2 ∣∣ Y − θ ∣ ∣ 2
V r ⊂ V n V_r\subset V_n V r ⊂ V n
Y ~ ∈ V n , w ~ \utilde{Y}\in V_n, \utilde{w} Y ∈ V n , w Y ~ \utilde{Y} Y V r V_r V r
⟹ w ~ n × 1 = P n × n Y ~ n × 1 \implies \utilde{w}_{n\times 1}=P_{n\times n}\utilde{Y}_{n\times 1} ⟹ w n × 1 = P n × n Y n × 1 P P P
對稱（Symmetric）：P t = P P^t=P P t = P
幂等（Idempotent）：P 2 = P P^2=P P 2 = P
r a n k ( P ) = r rank(P)=r r ank ( P ) = r Proof : 令 α ~ 1 , α ~ 2 , ⋯ α ~ r \utilde{\alpha}_1,\utilde{\alpha}_2,\cdots\utilde{\alpha}_r α 1 , α 2 , ⋯ α r V r V_r V r
⟹ w ~ = ∑ i = 1 r ( Y ~ t α ~ i ) α ~ i e.g. [ 1 2 3 ] = 1 ⋅ [ 1 0 0 ] + 2 ⋅ [ 0 1 0 ] + 3 ⋅ [ 0 0 1 ] = ( ∑ i = 1 r α ~ i α ~ i t ) Y ~ note Y ~ t α ~ i = α ~ i t Y ~ is a scalar = T ⋅ T t Y ~ = P Y ~ where T n × r = [ α ~ 1 α ~ 2 ⋯ α ~ r ] , P n × n = T ⋅ T t \begin{align*}
\implies \utilde{w}&=\sum_{i=1}^r(\utilde{Y}^t\utilde{\alpha}_i)\utilde{\alpha}_i\quad \text{e.g. } \begin{bmatrix*}
1\\2\\3
\end{bmatrix*}=1\cdot\begin{bmatrix*}
1\\0\\0
\end{bmatrix*}+2\cdot\begin{bmatrix*}
0\\1\\0
\end{bmatrix*}+3\cdot\begin{bmatrix*}
0\\0\\1
\end{bmatrix*}\\
&=(\sum_{i=1}^r\utilde{\alpha}_i\utilde{\alpha}_i^t)\utilde{Y}\quad \text{note }\utilde{Y}^t\utilde{\alpha}_i=\utilde{\alpha}_i^t\utilde{Y} \text{ is a scalar}\\
&= T\cdot T^t\utilde{Y}=P\utilde{Y} \quad \text{where } T_{n\times r}=\begin{bmatrix}
\utilde{\alpha}_1 & \utilde{\alpha}_2 & \cdots & \utilde{\alpha}_r
\end{bmatrix}, P_{n\times n}=T\cdot T^t
\end{align*} ⟹ w = i = 1 ∑ r ( Y t α i ) α i e.g. 1 2 3 = 1 ⋅ 1 0 0 + 2 ⋅ 0 1 0 + 3 ⋅ 0 0 1 = ( i = 1 ∑ r α i α i t ) Y note Y t α i = α i t Y is a scalar = T ⋅ T t Y = P Y where T n × r = [ α 1 α 2 ⋯ α r ] , P n × n = T ⋅ T t ⟹ P t = ( T T t ) t = T T t = P Symmetric \implies P^t=(TT^t)^t=TT^t=P \quad \text{Symmetric} ⟹ P t = ( T T t ) t = T T t = P Symmetric 因為 T T T T t T = I T^tT=I T t T = I
P P = ( T T t ) ( T T t ) = T T t T T t = T T t = P Idempotent PP=(TT^t)(TT^t)=TT^tTT^t=TT^t=P \quad \text{Idempotent} PP = ( T T t ) ( T T t ) = T T t T T t = T T t = P Idempotent 並且
r a n k ( P ) = r a n k ( T T t ) = r a n k ( T ) = r a n k ( T t ) = r rank(P)=rank(TT_t)=rank(T)=rank(T^t)=r r ank ( P ) = r ank ( T T t ) = r ank ( T ) = r ank ( T t ) = r 因為
r a n k ( A B ) ≤ m i n ( r a n k ( A ) , r a n k ( B ) ) rank(AB)\le min(rank(A), rank(B)) r ank ( A B ) ≤ min ( r ank ( A ) , r ank ( B )) ⟹ r a n k ( P ) ≤ m i n ( r a n k ( T ) , r a n k ( T t ) ) = r \implies rank(P)\le min(rank(T), rank(T^t))=r ⟹ r ank ( P ) ≤ min ( r ank ( T ) , r ank ( T t )) = r r = r a n k ( T ) = r a n k ( T I ) = r a n k ( T T t T ) = r a n k ( P T ) ≤ m i n ( r a n k ( P ) , r a n k ( T ) ) ≤ r a n k ( P ) r=rank(T)=rank(TI)=rank(TT^tT)=rank(PT)\le min(rank(P), rank(T))\le rank(P) r = r ank ( T ) = r ank ( T I ) = r ank ( T T t T ) = r ank ( PT ) ≤ min ( r ank ( P ) , r ank ( T )) ≤ r ank ( P ) i.e. r a n k ( P ) = r rank(P)=r r ank ( P ) = r
P: n × n n\times n n × n r a n k ( P ) = r rank(P)=r r ank ( P ) = r
P P P r r r n − r n-r n − r tr ( P ) ≜ ∑ i = 1 n P i i = r \text{tr}(P)\triangleq\sum_{i=1}^n P_{ii}=r tr ( P ) ≜ ∑ i = 1 n P ii = r I − P I-P I − P r a n k ( I − P ) = n − r = tr ( I − P ) rank(I-P)=n-r=\text{tr}(I-P) r ank ( I − P ) = n − r = tr ( I − P ) ∀ a ~ ∈ R n , a ~ t P a ~ ≥ 0 \forall \utilde{a}\in\R^n, \utilde{a}^tP\utilde{a}\ge 0 ∀ a ∈ R n , a t P a ≥ 0 Proof :
因為 P P P A A A A t A = I A^tA=I A t A = I A t P A = d i a g ( λ 1 , λ 2 , ⋯ , λ n ) = B A^tPA=diag(\lambda_1, \lambda_2,\cdots, \lambda_n)=B A t P A = d ia g ( λ 1 , λ 2 , ⋯ , λ n ) = B λ i \lambda_i λ i P P P
⟹ B 2 = B B = A t P A ⋅ A t P A = A t P A = B \implies B^2=BB=A^tPA\cdot A^tPA=A^tPA=B ⟹ B 2 = BB = A t P A ⋅ A t P A = A t P A = B λ i 2 = λ i \lambda^2_i=\lambda_i λ i 2 = λ i i = 1 , 2 , ⋯ , n i=1,2,\cdots, n i = 1 , 2 , ⋯ , n
⟹ λ i = 0 \implies \lambda_i=0 ⟹ λ i = 0 1 , ∀ i 1, \forall i 1 , ∀ i
r = r a n k ( P ) = R a n k ( A t P A ) ∵ A is nonsingular = R a n k ( B ) \begin{align*}
r=rank(P)&=Rank(A^tPA) \quad \because A \text{ is nonsingular}\\
&=Rank(B)\\
\end{align*} r = r ank ( P ) = R ank ( A t P A ) ∵ A is nonsingular = R ank ( B ) ⟹ \implies ⟹ B B B r r r n − r n-r n − r
tr ( P ) = tr ( P A t A ) = tr ( A t P A ) = tr ( B ) = r \text{tr}(P)=\text{tr}(PA^tA)=\text{tr}(A^tPA)=\text{tr}(B)=r tr ( P ) = tr ( P A t A ) = tr ( A t P A ) = tr ( B ) = r 記得如果 b ~ \utilde{b} b β ~ \utilde{\beta} β ⟺ D t Y ~ = D t D b ~ \iff D^t\utilde{Y}=D^tD\utilde{b} ⟺ D t Y = D t D b
根據上面的結論，我們可以得到
Y ^ ~ = D b ~ = Lemma 3 P Y ~ with P : n × n symmetric idempotent matrix and r a n k ( P ) = r a n k ( D ) ≤ p = projection of Y ~ onto Ω with Ω ≜ s p a n ( D ) \begin{align*}
\utilde{\hat{Y}}&=D\utilde{b}\xlongequal{\text{Lemma 3}}P\utilde{Y} \quad \text{with } P: n\times n \text{ symmetric idempotent matrix} \text{ and } rank(P)=rank(D)\le p\\
&= \text{ projection of } \utilde{Y} \text{ onto } \Omega\quad \text{with } \Omega\triangleq span(D)
\end{align*} Y ^ = D b Lemma 3 P Y with P : n × n symmetric idempotent matrix and r ank ( P ) = r ank ( D ) ≤ p = projection of Y onto Ω with Ω ≜ s p an ( D ) 這裡我們假設未知的回歸係數數量 p = k + 1 ≤ p=k+1\le p = k + 1 ≤ n n n
r a n k ( D ) = p ⟺ D is full rank ⟺ dim ( Ω ) = p ⟺ the columns of D are linearly independent ⟺ D t D is nonsingular ⟺ ( D t D ) − 1 exists ⟹ normal equation has unique solution \begin{align*}
rank(D)=p&\iff D \text{ is full rank}\\
&\iff \dim(\Omega)=p\\
&\iff \text{ the columns of } D \text{ are linearly independent}\\
&\iff D^tD \text{ is nonsingular}\\
&\iff (D^tD)^{-1} \text{ exists}\\
&\implies \text{normal equation has unique solution}
\end{align*} r ank ( D ) = p ⟺ D is full rank ⟺ dim ( Ω ) = p ⟺ the columns of D are linearly independent ⟺ D t D is nonsingular ⟺ ( D t D ) − 1 exists ⟹ normal equation has unique solution i.e. b ~ p × 1 = ( D t D ) − 1 D t Y ~ \utilde{b}_{p\times 1}=(D^tD)^{-1}D^t\utilde{Y} b p × 1 = ( D t D ) − 1 D t Y
⟹ Y ^ ~ = D b ~ = D ( D t D ) − 1 D t Y ~ = H Y ~ where H ≜ D ( D t D ) − 1 D t hat matrix \begin{align*}
\implies \utilde{\hat{Y}}&=D\utilde{b}\\
&=D(D^tD)^{-1}D^t\utilde{Y}\\
&=H\utilde{Y}\quad \text{where } H\triangleq D(D^tD)^{-1}D^t\quad\text{ hat matrix}
\end{align*} ⟹ Y ^ = D b = D ( D t D ) − 1 D t Y = H Y where H ≜ D ( D t D ) − 1 D t hat matrix 並且 e ~ = Y ~ − Y ^ ~ = Y ~ − H Y ~ = ( I − H ) Y ~ ≜ M Y \utilde{e}=\utilde{Y}-\utilde{\hat{Y}}=\utilde{Y}-H\utilde{Y}=(I-H)\utilde{Y}\triangleq MY e = Y − Y ^ = Y − H Y = ( I − H ) Y ≜ M Y M ≜ I − H M\triangleq I-H M ≜ I − H
我們可以很容易的檢查 H H H M M M
H = D ( D t D ) − 1 D t n × n ⟹ H t = H Symmetric H H = H Idempotent with r a n k ( H ) = p = dim ( Ω ) \begin{align*}
& H=D(D^tD)^{-1}D^t\quad n\times n\\
\implies &H^t=H\quad \text{Symmetric}\\
&HH=H\quad \text{Idempotent}\\
&\text{with }rank(H)=p=\dim(\Omega)\\
\end{align*} ⟹ H = D ( D t D ) − 1 D t n × n H t = H Symmetric HH = H Idempotent with r ank ( H ) = p = dim ( Ω ) 並且 M = I − H M=I-H M = I − H r a n k ( M ) = n − p rank(M)=n-p r ank ( M ) = n − p
Note ：
H 是 Ω \Omega Ω Ω ⊥ \Omega^\perp Ω ⊥ R n = Ω + Ω ⊥ \R^n=\Omega+\Omega^\perp R n = Ω + Ω ⊥
如果我們把已經在 Ω \Omega Ω Ω \Omega Ω Ω \Omega Ω Ω \Omega Ω
H θ ~ = θ ~ M θ ~ = 0 ∀ θ ~ ∈ Ω H\utilde{\theta}=\utilde{\theta} \qquad M\utilde{\theta}=0 \quad \forall \utilde{\theta}\in\Omega H θ = θ M θ = 0 ∀ θ ∈ Ω 因為 D ∈ Ω , e ~ ∈ Ω ⊥ ⟹ D t e ~ = 0 D\in\Omega, \utilde{e}\in\Omega^\perp\implies D^t\utilde{e}=0 D ∈ Ω , e ∈ Ω ⊥ ⟹ D t e = 0 1 ~ t e ~ = 0 \utilde{1}^t\utilde{e}=0 1 t e = 0 X ~ j t e ~ = 0 \utilde{X}_j^t\utilde{e}=0 X j t e = 0 j = 1 , 2 , ⋯ , k j=1,2,\cdots,k j = 1 , 2 , ⋯ , k 1 ~ \utilde{1} 1 X ~ j \utilde{X}_j X j Ω \Omega Ω
上面說 Y ^ ~ \utilde{\hat{Y}} Y ^ Ω \Omega Ω Y ^ ~ t e ~ = 0 \utilde{\hat{Y}}^t\utilde{e}=0 Y ^ t e = 0
SSE 的性質：
∙ SSE ≜ e ~ t e ~ = ( M Y ~ ) t ( M Y ~ ) = Y ~ t M t M Y ~ = Y ~ t M Y ~ ∵ M is symmetric idempotent = ( D β ~ + ε ~ ) t M ( D β ~ + ε ~ ) = β ~ t D t M D β ~ + 2 β ~ t D t M ε ~ + ε ~ t M ε ~ = ε ~ t M ε ~ ∵ D t M = 0 ∙ E [ SSE ] = E [ ε ~ t M ε ~ ] = ∑ i = 1 n ∑ j = 1 n E [ ε i M i j ε j ] = ∑ i = 1 n M i j E [ ε i 2 ] ∵ E [ ε i ε j ] = 0 , i ≠ j = σ 2 tr ( M ) = σ 2 ( n − p ) ∙ E [ MSE ] = E [ SSE n − p ] = σ 2 \begin{align*}
& \begin{align*}
\bullet\quad\text{SSE} &\triangleq \utilde{e}^t\utilde{e}=(M\utilde{Y})^t(M\utilde{Y})=\utilde{Y}^tM^tM\utilde{Y}\\
&=\utilde{Y}^tM\utilde{Y} \quad \because M \text{ is symmetric idempotent}\\
&=(D\utilde{\beta}+\utilde{\varepsilon})^tM(D\utilde{\beta}+\utilde{\varepsilon})\\
&=\utilde{\beta}^tD^tMD\utilde{\beta}+2\utilde{\beta}^tD^tM\utilde{\varepsilon}+\utilde{\varepsilon}^tM\utilde{\varepsilon}\\
&=\utilde{\varepsilon}^tM\utilde{\varepsilon} \quad \because D^tM=0
\end{align*}\\
& \begin{align*}
\bullet\quad E[\text{SSE}]&=E[\utilde{\varepsilon}^tM\utilde{\varepsilon}]=\sum_{i=1}^n\sum_{j=1}^nE[\varepsilon_iM_{ij}\varepsilon_j]\\
&=\sum_{i=1}^nM_{ij}E[\varepsilon_i^2] \quad \because E[\varepsilon_i\varepsilon_j]=0, i\ne j\\
&=\sigma^2\text{tr}(M)=\sigma^2(n-p)
\end{align*}\\
& \bullet\quad E[\text{MSE}]=E[\frac{\text{SSE}}{n-p}]=\sigma^2
\end{align*} ∙ SSE ≜ e t e = ( M Y ) t ( M Y ) = Y t M t M Y = Y t M Y ∵ M is symmetric idempotent = ( D β + ε ) t M ( D β + ε ) = β t D t M D β + 2 β t D t M ε + ε t M ε = ε t M ε ∵ D t M = 0 ∙ E [ SSE ] = E [ ε t M ε ] = i = 1 ∑ n j = 1 ∑ n E [ ε i M ij ε j ] = i = 1 ∑ n M ij E [ ε i 2 ] ∵ E [ ε i ε j ] = 0 , i = j = σ 2 tr ( M ) = σ 2 ( n − p ) ∙ E [ MSE ] = E [ n − p SSE ] = σ 2
b ~ \utilde{b} b
∙ E [ b ~ ] = E [ ( D t D ) − 1 D t Y ~ ] = ( D t D ) − 1 D t ⋅ E [ Y ~ ] = ( D t D ) − 1 D t D β ~ = β ~ i.e. b ~ is unbiased for β ~ ∙ σ 2 { b ~ } = σ 2 { ( D t D ) − 1 D t Y ~ } = ( D t D ) − 1 D t σ 2 { Y ~ } D ( D t D ) − 1 = σ 2 ( D t D ) − 1 D t D ( D t D ) − 1 ⏟ = I σ 2 { Y ~ } = σ 2 I = σ 2 ( D t D ) − 1 ∙ S 2 { b ~ } = MSE ( D t D ) − 1 to est σ 2 { b ~ } \begin{align*}
&\begin{align*}
\bullet\quad E[\utilde{b}]&=E[(D^tD)^{-1}D^t\utilde{Y}]\\
&=(D^tD)^{-1}D^t\cdot E[\utilde{Y}]\\
&=(D^tD)^{-1}D^tD\utilde{\beta}\\
&=\utilde{\beta}\quad \text{i.e. } \utilde{b} \text{ is unbiased for } \utilde{\beta}
\end{align*}\\
&\begin{align*}
\bullet\quad \sigma^2\set{\utilde{b}}&=\sigma^2\set{(D^tD)^{-1}D^t\utilde{Y}}\\
&=(D^tD)^{-1}D^t\sigma^2\set{\utilde{Y}}D(D^tD)^{-1}\\
&=\sigma^2 (D^tD)^{-1}\underbrace{D^tD(D^tD)^{-1}}_{=I} \quad \sigma^2\set{\utilde{Y}}=\sigma^2I\\
&=\sigma^2(D^tD)^{-1}
\end{align*}\\
&\bullet\quad S^2\set{\utilde{b}}=\text{MSE}(D^tD)^{-1} \quad \text{to est } \sigma^2\set{\utilde{b}}
\end{align*} ∙ E [ b ] = E [( D t D ) − 1 D t Y ] = ( D t D ) − 1 D t ⋅ E [ Y ] = ( D t D ) − 1 D t D β = β i.e. b is unbiased for β ∙ σ 2 { b } = σ 2 { ( D t D ) − 1 D t Y } = ( D t D ) − 1 D t σ 2 { Y } D ( D t D ) − 1 = σ 2 ( D t D ) − 1 = I D t D ( D t D ) − 1 σ 2 { Y } = σ 2 I = σ 2 ( D t D ) − 1 ∙ S 2 { b } = MSE ( D t D ) − 1 to est σ 2 { b }
Y ^ ~ \utilde{\hat{Y}} Y ^
∙ E [ Y ^ ~ ] = E [ H Y ~ ] = H ⋅ E [ Y ~ ] = H D β ~ = D β ~ = E [ Y ~ ] ⟹ Y ^ ~ is unbiased for E [ Y ~ ] ∙ σ 2 { Y ^ ~ } = σ 2 { H Y ~ } = H σ 2 { Y ~ } H t = σ 2 H H t = σ 2 H \begin{align*}
&\begin{align*}
\bullet\quad E[\utilde{\hat{Y}}]&=E[H\utilde{Y}]=H\cdot E[\utilde{Y}]=HD\utilde{\beta}=D\utilde{\beta}=E[\utilde{Y}]\\
\implies& \utilde{\hat{Y}} \text{ is unbiased for } E[\utilde{Y}]
\end{align*}\\
&\begin{align*}
\bullet\quad \sigma^2\set{\utilde{\hat{Y}}}&=\sigma^2\set{H\utilde{Y}}=H\sigma^2\set{\utilde{Y}}H^t\\
&=\sigma^2HH^t=\sigma^2H
\end{align*}\\
\end{align*} ∙ E [ Y ^ ] ⟹ = E [ H Y ] = H ⋅ E [ Y ] = HD β = D β = E [ Y ] Y ^ is unbiased for E [ Y ] ∙ σ 2 { Y ^ } = σ 2 { H Y } = H σ 2 { Y } H t = σ 2 H H t = σ 2 H
e ~ \utilde{e} e
∙ E [ e ~ ] = E [ M Y ~ ] = M E [ Y ~ ] = M D ⏟ = 0 β ~ = 0 ∙ σ 2 { e ~ } = σ 2 { M Y ~ } = σ 2 M \begin{align*}
&\bullet\quad E[\utilde{e}]=E[M\utilde{Y}]=ME[\utilde{Y}]=\underbrace{MD}_{=0}\utilde{\beta}=0\\
&\bullet\quad \sigma^2\set{\utilde{e}}=\sigma^2\set{M\utilde{Y}}=\sigma^2M
\end{align*} ∙ E [ e ] = E [ M Y ] = ME [ Y ] = = 0 M D β = 0 ∙ σ 2 { e } = σ 2 { M Y } = σ 2 M
S 2 { ∗ } ≜ σ 2 { ∗ } ∣ σ 2 = MSE S^2\set{\ast}\triangleq\sigma^2\set{\ast}|_{\sigma^2=\text{MSE}} S 2 { ∗ } ≜ σ 2 { ∗ } ∣ σ 2 = MSE σ 2 { ∗ } \sigma^2\set{\ast} σ 2 { ∗ }
ANOVA
∙ Y ~ = Y ^ ~ + e ~ ⟹ ∣ ∣ Y ~ ∣ ∣ 2 = ∣ ∣ Y ^ ~ ∣ ∣ 2 + ∣ ∣ e ~ ∣ ∣ 2 + 2 Y ^ ~ t e ~ = ∣ ∣ Y ^ ~ ∣ ∣ 2 + ∣ ∣ e ~ ∣ ∣ 2 = ∣ ∣ Y ^ ~ ∣ ∣ 2 + SSE ∙ Y ~ = Y ~ − Y ˉ 1 ~ + Y ˉ 1 ~ Y ˉ 1 ~ ∈ Ω ⟹ ∣ ∣ Y ~ ∣ ∣ 2 = ∣ ∣ Y ~ − Y ˉ 1 ~ ∣ ∣ 2 + Y ˉ 2 ⋅ n = ∑ i = 1 n ( Y i − Y ˉ ) 2 + n Y ˉ 2 = SSTO + n Y ˉ 2 ∙ Y ^ ~ = Y ^ ~ − Y ˉ 1 ~ + Y ˉ ⟹ ∣ ∣ Y ^ ~ ∣ ∣ 2 = ∣ ∣ Y ^ ~ − Y ˉ 1 ~ ∣ ∣ 2 + Y ˉ 2 ⋅ n = ∑ i = 1 n ( Y ^ i − Y ˉ ) 2 + n Y ˉ 2 = SSR + n Y ˉ 2 ⟹ ∣ ∣ Y ~ ∣ ∣ 2 = ∣ ∣ Y ^ ~ ∣ ∣ 2 + ∣ ∣ e ~ ∣ ∣ 2 ⟹ SSTO + n Y ˉ 2 = SSR + n Y ˉ 2 + SSE \begin{align*}
&\begin{align*}
\bullet\quad &\utilde{Y}=\utilde{\hat{Y}}+\utilde{e}\\
&\begin{align*}
\implies ||\utilde{Y}||^2&=||\utilde{\hat{Y}}||^2+||\utilde{e}||^2+2\utilde{\hat{Y}}^t\utilde{e}=||\utilde{\hat{Y}}||^2+||\utilde{e}||^2\\
&=||\utilde{\hat{Y}}||^2 + \text{SSE}
\end{align*}
\end{align*}\\
&\begin{align*}
\bullet\quad &\utilde{Y}=\utilde{Y}-\bar{Y}\utilde{1}+\bar{Y}\utilde{1}\quad \bar{Y}\utilde{1}\in\Omega\\
&\begin{align*}
\implies ||\utilde{Y}||^2 &= ||\utilde{Y}-\bar{Y}\utilde{1}||^2 + \bar{Y}^2\cdot n\\
&=\sum_{i=1}^n(Y_i-\bar{Y})^2+n\bar{Y}^2\\
&=\text{SSTO}+n\bar{Y}^2
\end{align*}
\end{align*}\\
&\begin{align*}
\bullet\quad &\utilde{\hat{Y}}=\utilde{\hat{Y}}-\bar{Y}\utilde{1}+\bar{Y}\\
&\begin{align*}
\implies ||\utilde{\hat{Y}}||^2&=||\utilde{\hat{Y}}-\bar{Y}\utilde{1}||^2+\bar{Y}^2\cdot n\\
&=\sum_{i=1}^n(\hat{Y}_i-\bar{Y})^2+n\bar{Y}^2\\
&=\text{SSR}+n\bar{Y}^2
\end{align*}
\end{align*}\\
&\implies ||\utilde{Y}||^2=||\utilde{\hat{Y}}||^2+||\utilde{e}||^2\\
&\implies \text{SSTO}+\bcancel{n\bar{Y}^2}=\text{SSR}+\bcancel{n\bar{Y}^2}+\text{SSE}
\end{align*} ∙ Y = Y ^ + e ⟹ ∣∣ Y ∣ ∣ 2 = ∣∣ Y ^ ∣ ∣ 2 + ∣∣ e ∣ ∣ 2 + 2 Y ^ t e = ∣∣ Y ^ ∣ ∣ 2 + ∣∣ e ∣ ∣ 2 = ∣∣ Y ^ ∣ ∣ 2 + SSE ∙ Y = Y − Y ˉ 1 + Y ˉ 1 Y ˉ 1 ∈ Ω ⟹ ∣∣ Y ∣ ∣ 2 = ∣∣ Y − Y ˉ 1 ∣ ∣ 2 + Y ˉ 2 ⋅ n = i = 1 ∑ n ( Y i − Y ˉ ) 2 + n Y ˉ 2 = SSTO + n Y ˉ 2 ∙ Y ^ = Y ^ − Y ˉ 1 + Y ˉ ⟹ ∣∣ Y ^ ∣ ∣ 2 = ∣∣ Y ^ − Y ˉ 1 ∣ ∣ 2 + Y ˉ 2 ⋅ n = i = 1 ∑ n ( Y ^ i − Y ˉ ) 2 + n Y ˉ 2 = SSR + n Y ˉ 2 ⟹ ∣∣ Y ∣ ∣ 2 = ∣∣ Y ^ ∣ ∣ 2 + ∣∣ e ∣ ∣ 2 ⟹ SSTO + n Y ˉ 2 = SSR + n Y ˉ 2 + SSE Note:
∣ ∣ Y ˉ 1 ~ ∣ ∣ 2 = n Y ˉ 2 = Y ˉ 1 ~ t 1 ~ Y ˉ = 1 n Y ~ t 1 ~ 1 ~ t 1 ~ 1 n 1 ~ t Y ~ = Y ~ t 1 ~ 1 ~ t n Y ~ ∵ 1 ~ t 1 ~ = n = Y ~ t J n Y ~ where J = [ 1 ] n × n \begin{align*}
||\bar{Y}\utilde{1}||^2&=n\bar{Y}^2\\
&=\bar{Y}\utilde{1}^t\utilde{1}\bar{Y}\\
&=\frac{1}{n}\utilde{Y}^t\utilde{1}\utilde{1}^t\utilde{1}\frac{1}{n}\utilde{1}^t\utilde{Y}\\
&=\utilde{Y}^t\frac{\utilde{1}\utilde{1}^t}{n}\utilde{Y} \quad \because\utilde{1}^t\utilde{1}=n\\
&=\utilde{Y}^t\frac{J}{n}\utilde{Y}\quad \text{where } J=[1]_{n\times n}
\end{align*} ∣∣ Y ˉ 1 ∣ ∣ 2 = n Y ˉ 2 = Y ˉ 1 t 1 Y ˉ = n 1 Y t 1 1 t 1 n 1 1 t Y = Y t n 1 1 t Y ∵ 1 t 1 = n = Y t n J Y where J = [ 1 ] n × n
( J n ) t = 1 n J t = 1 n J (\frac{J}{n})^t=\frac{1}{n}J^t=\frac{1}{n}J ( n J ) t = n 1 J t = n 1 J J n ⋅ J n = J n \frac{J}{n}\cdot\frac{J}{n}=\frac{J}{n} n J ⋅ n J = n J
⟹ J n \implies \frac{J}{n} ⟹ n J n × n n\times n n × n s p a n { 1 ~ } span\set{\utilde{1}} s p an { 1 }
∙ Y ~ t I Y ~ = Y ~ t ( J n + I − J n ) Y ~ ⟺ ∣ ∣ Y ~ ∣ ∣ 2 = Y ~ t J n Y ~ + Y ~ t ( I − J n ) Y ~ = n Y ˉ 2 + SSTO i.e. SSTO ≜ ∑ i = 1 n ( Y i − Y ˉ ) 2 = Y ~ t ( I − J n ) Y ~ ∙ SSE = ∣ ∣ e ~ ∣ ∣ 2 = e ~ t e ~ = ( M Y ~ ) t M Y ~ = Y ~ t M t M Y ~ = Y ~ t M Y ~ = Y ~ t ( I − H ) Y ~ ∙ SSR = ∑ i = 1 n ( Y ^ i − Y ˉ ) 2 = SSTO − SSE = Y ~ t ( I − J n ) Y ~ − Y ~ t ( I − H ) Y ~ = Y ~ t ( H − J n ) Y ~ \begin{align*}
&\begin{align*}
\bullet\quad &\utilde{Y}^tI\utilde{Y}=\utilde{Y}^t\left(\frac{J}{n}+I-\frac{J}{n}\right)\utilde{Y}\\
&\begin{align*}
\iff ||\utilde{Y}||^2&=\utilde{Y}^t\frac{J}{n}\utilde{Y}+\utilde{Y}^t\left(I-\frac{J}{n}\right)\utilde{Y}\\
&=n\bar{Y}^2+\text{SSTO}
\end{align*}\\
&\text{i.e. SSTO }\triangleq \sum_{i=1}^n(Y_i-\bar{Y})^2=\utilde{Y}^t\left(I-\frac{J}{n}\right)\utilde{Y}
\end{align*}\\
&\begin{align*}
\bullet\quad \text{SSE } &=||\utilde{e}||^2=\utilde{e}^t\utilde{e}=(M\utilde{Y})^tM\utilde{Y}=\utilde{Y}^tM^tM\utilde{Y}\\
&=\utilde{Y}^tM\utilde{Y}=\utilde{Y}^t(I-H)\utilde{Y}
\end{align*}\\
&\begin{align*}
\bullet\quad \text{SSR } &= \sum_{i=1}^n(\hat{Y}_i-\bar{Y})^2=\text{SSTO}-\text{SSE}\\
&=\utilde{Y}^t\left(I-\frac{J}{n}\right)\utilde{Y}-\utilde{Y}^t(I-H)\utilde{Y}\\
&=\utilde{Y}^t(H-\frac{J}{n})\utilde{Y}
\end{align*}
\end{align*} ∙ Y t I Y = Y t ( n J + I − n J ) Y ⟺ ∣∣ Y ∣ ∣ 2 = Y t n J Y + Y t ( I − n J ) Y = n Y ˉ 2 + SSTO i.e. SSTO ≜ i = 1 ∑ n ( Y i − Y ˉ ) 2 = Y t ( I − n J ) Y ∙ SSE = ∣∣ e ∣ ∣ 2 = e t e = ( M Y ) t M Y = Y t M t M Y = Y t M Y = Y t ( I − H ) Y ∙ SSR = i = 1 ∑ n ( Y ^ i − Y ˉ ) 2 = SSTO − SSE = Y t ( I − n J ) Y − Y t ( I − H ) Y = Y t ( H − n J ) Y Note:
I = J n ⏟ ∈ Ω + ( I − J n ) ⏟ ∈ Ω ⊥ = H ⏟ ∈ Ω + ( I − H ) ⏟ ∈ Ω ⊥ = ( H − J n ) ⏟ ∈ Ω + I − H + J n ⏟ ∈ Ω ⊥ and SSTO = Y ~ t ( I − J n ) Y ~ SSE = Y ~ t ( I − H ) Y ~ SSR = Y ~ t ( H − J n ) Y ~ \begin{align*}
I&=\underbrace{\frac{J}{n}}_{\in\Omega}+\underbrace{(I-\frac{J}{n})}_{\in\Omega^\perp}\\
&=\underbrace{H}_{\in\Omega}+\underbrace{(I-H)}_{\in\Omega^\perp}\\
&=\underbrace{(H-\frac{J}{n})}_{\in\Omega}+\underbrace{I-H+\frac{J}{n}}_{\in\Omega^\perp}
\end{align*}\qquad\text{and}\qquad
\begin{align*}
&\text{SSTO}=\utilde{Y}^t\left(I-\frac{J}{n}\right)\utilde{Y}\\
&\text{SSE}=\utilde{Y}^t(I-H)\utilde{Y}\\
&\text{SSR}=\utilde{Y}^t(H-\frac{J}{n})\utilde{Y}
\end{align*} I = ∈ Ω n J + ∈ Ω ⊥ ( I − n J ) = ∈ Ω H + ∈ Ω ⊥ ( I − H ) = ∈ Ω ( H − n J ) + ∈ Ω ⊥ I − H + n J and SSTO = Y t ( I − n J ) Y SSE = Y t ( I − H ) Y SSR = Y t ( H − n J ) Y I = P 1 + P 2 + ⋯ + P m I=P_1+P_2+\cdots+P_m I = P 1 + P 2 + ⋯ + P m P i P_i P i ∣ ∣ Y ~ ∣ ∣ 2 = ∑ i = 1 m Y ~ t P i Y ~ ||\utilde{Y}||^2=\sum_{i=1}^m\utilde{Y}^tP_i\utilde{Y} ∣∣ Y ∣ ∣ 2 = ∑ i = 1 m Y t P i Y
從上面的推理我們可以知道 SSTO, SSE, SSR 都是 quadratic form。Y ~ t P Y ~ \utilde{Y}^tP\utilde{Y} Y t P Y P P P n × n n\times n n × n
Gauss-Markov Theorem
Let ξ = c ~ t β ~ \xi=\utilde{c}^t\utilde{\beta} ξ = c t β c ~ ∈ R p \utilde{c}\in\R^p c ∈ R p
ξ \xi ξ ∃ a ~ ∈ R n \exist \utilde{a}\in\R^n ∃ a ∈ R n ∀ β ~ , E [ a ~ t Y ~ ] = ξ = c ~ t β ~ \forall\utilde{\beta}, E[\utilde{a}^t\utilde{Y}]=\xi=\utilde{c}^t\utilde{\beta} ∀ β , E [ a t Y ] = ξ = c t β a ~ t Y ~ \utilde{a}^t\utilde{Y} a t Y ξ \xi ξ
EX :
[ z 1 ⋮ z m w 1 ⋮ w n ] = [ 1 , 0 ⋮ , ⋮ 1 , 0 0 , 1 ⋮ , ⋮ 0 , 1 ] [ μ 1 μ 2 ] + ε \begin{bmatrix}
z_1\\\vdots\\z_m\\w_1\\\vdots\\w_n
\end{bmatrix}=
\begin{bmatrix}
1, 0\\
\vdots, \vdots\\
1, 0\\
0, 1\\
\vdots, \vdots\\
0, 1
\end{bmatrix}
\begin{bmatrix}
\mu_1\\\mu_2
\end{bmatrix}+\varepsilon z 1 ⋮ z m w 1 ⋮ w n = 1 , 0 ⋮ , ⋮ 1 , 0 0 , 1 ⋮ , ⋮ 0 , 1 [ μ 1 μ 2 ] + ε If interested in ξ = μ 1 − μ 2 = [ 1 , − 1 ] β = c t β \xi=\mu_1-\mu_2=[1, -1]\beta = c^t\beta ξ = μ 1 − μ 2 = [ 1 , − 1 ] β = c t β
a t = [ 1 , 0 , ⋯ , 0 , − 1 , 0 , ⋯ , 0 ] ⟹ E ( a t Y ) = E [ Z 1 − W 1 ] = μ 1 − μ 2 , ∀ β ∈ R 2 a^t=[1,0,\cdots,0,-1,0,\cdots,0]\implies E(a^tY)=E[Z_1-W_1]=\mu_1-\mu_2, \forall\beta\in\R^2 a t = [ 1 , 0 , ⋯ , 0 , − 1 , 0 , ⋯ , 0 ] ⟹ E ( a t Y ) = E [ Z 1 − W 1 ] = μ 1 − μ 2 , ∀ β ∈ R 2 a t = [ 1 / m , ⋯ , 1 / m , 1 / n , ⋯ , 1 / n ] ⟹ E ( a t Y ) = E [ Z ˉ − W ˉ ] = μ 1 − μ 2 , ∀ β ∈ R 2 a^t=[1/m,\cdots,1/m,1/n,\cdots,1/n]\implies E(a^tY)=E[\bar{Z}-\bar{W}]=\mu_1-\mu_2, \forall\beta\in\R^2 a t = [ 1/ m , ⋯ , 1/ m , 1/ n , ⋯ , 1/ n ] ⟹ E ( a t Y ) = E [ Z ˉ − W ˉ ] = μ 1 − μ 2 , ∀ β ∈ R 2
Gauss-Markov Theorem :
Suppose # of observations n ≥ p n\ge p n ≥ p
If Y ~ = D β ~ + ε ~ with E [ ε ~ ] = 0 , σ 2 { ε ~ } = σ 2 I n × n \text{If }\utilde{Y}=D\utilde{\beta}+\utilde{\varepsilon}\text{ with } E[\utilde{\varepsilon}]=0, \sigma^2\set{\utilde{\varepsilon}}=\sigma^2I_{n\times n} If Y = D β + ε with E [ ε ] = 0 , σ 2 { ε } = σ 2 I n × n Let c ~ ∈ R p \utilde{c}\in\R^p c ∈ R p ξ = c ~ t β ~ \xi =\utilde{c}^t\utilde{\beta} ξ = c t β ξ ^ = c ~ t b ~ \hat{\xi}=\utilde{c}^t\utilde{b} ξ ^ = c t b b ~ \utilde{b} b β ~ \utilde{\beta} β
Then
ξ ^ \hat{\xi} ξ ^ ξ \xi ξ σ 2 { ξ ^ } ≤ σ 2 { ξ ~ } , ∀ ξ ~ \sigma^2\set{\hat{\xi}}\le\sigma^2\set{\tilde{\xi}}, \forall \tilde{\xi} σ 2 { ξ ^ } ≤ σ 2 { ξ ~ } , ∀ ξ ~ ξ \xi ξ i.e. ξ ^ \hat{\xi} ξ ^ ξ \xi ξ
Proof :
因為我們對於所有 β ~ \utilde{\beta} β ξ = c ~ t β ~ \xi=\utilde{c}^t\utilde{\beta} ξ = c t β c ~ ∈ R p \utilde{c}\in\R^p c ∈ R p
∃ a ~ ∈ R n ∀ β ∈ R p , s.t. E [ a ~ t Y ~ ] = ξ ⟺ a ~ t D β ~ = c ~ t β ~ , ∀ β ~ ∈ R p ⟺ ( a ~ t D − c ~ t ) β ~ = 0 , ∀ β ~ ∈ R p ⟺ a ~ t D − c ~ t = 0 ∵ ∀ β ~ ∈ R p ⟺ a ~ t D = c ~ t \begin{align*}
&\exist \utilde{a}\in\R^n \forall\beta\in\R^p, \text{ s.t. } E[\utilde{a}^t\utilde{Y}]=\xi\\
\iff& \utilde{a}^tD\utilde{\beta}=\utilde{c}^t\utilde{\beta},\forall\utilde{\beta}\in\R^p\\
\iff& \left(\utilde{a}^tD-\utilde{c}^t\right)\utilde{\beta}=0,\forall\utilde{\beta}\in\R^p\\
\iff& \utilde{a}^tD-\utilde{c}^t=0\quad\because \forall\utilde{\beta}\in\R^p\\
\iff& \utilde{a}^tD=\utilde{c}^t
\end{align*} ⟺ ⟺ ⟺ ⟺ ∃ a ∈ R n ∀ β ∈ R p , s.t. E [ a t Y ] = ξ a t D β = c t β , ∀ β ∈ R p ( a t D − c t ) β = 0 , ∀ β ∈ R p a t D − c t = 0 ∵ ∀ β ∈ R p a t D = c t Let Ω = \Omega= Ω = D D D R n \R^n R n
⟹ ∀ a ~ ∈ R n , ∃ ! d ~ ∈ Ω , ω ~ ∈ Ω ⊥ \implies \forall\utilde{a}\in\R^n,\exist!\utilde{d}\in\Omega,\utilde{\omega}\in\Omega^\perp ⟹ ∀ a ∈ R n , ∃ ! d ∈ Ω , ω ∈ Ω ⊥ a ~ = d ~ + ω ~ \utilde{a}=\utilde{d}+\utilde{\omega} a = d + ω a ~ \utilde{a} a
⟹ E [ a ~ t Y ~ ] = E [ ( d ~ + ω ~ ) t Y ~ ] = E [ d ~ t Y ~ ] + E [ ω ~ t Y ~ ] = d ~ t D β ~ + ω ~ t D β ~ ⏟ = 0 ∵ ω ~ ∈ Ω ⊥ and D ∈ Ω = d ~ t D β ~ \begin{align*}
\implies E[\utilde{a}^t\utilde{Y}]&=E[(\utilde{d}+\utilde{\omega})^t\utilde{Y}]=E[\utilde{d}^t\utilde{Y}] + E[\utilde{\omega}^t\utilde{Y}]\\
&=\utilde{d}^tD\utilde{\beta}+\underbrace{\utilde{\omega}^tD\utilde{\beta}}_{=0}\quad \because \utilde{\omega}\in\Omega^\perp\text{ and } D\in\Omega\\
&=\utilde{d}^tD\utilde{\beta}
\end{align*} ⟹ E [ a t Y ] = E [( d + ω ) t Y ] = E [ d t Y ] + E [ ω t Y ] = d t D β + = 0 ω t D β ∵ ω ∈ Ω ⊥ and D ∈ Ω = d t D β 而 E [ a ~ t Y ~ ] = ξ = c ~ t β ~ , ∀ β ~ ∈ R p ⟹ d ~ t Y ~ E[\utilde{a}^t\utilde{Y}]=\xi=\utilde{c}^t\utilde{\beta},\forall\utilde{\beta}\in\R^p\implies\utilde{d}^t\utilde{Y} E [ a t Y ] = ξ = c t β , ∀ β ∈ R p ⟹ d t Y ξ \xi ξ d ~ t D = c ~ t \utilde{d}^tD=\utilde{c}^t d t D = c t
Claim : ∀ β ~ ∈ R p , d ~ \forall\utilde{\beta}\in\R^p,\utilde{d} ∀ β ∈ R p , d Ω \Omega Ω E [ d ~ t Y ~ ] = ξ E[\utilde{d}^t\utilde{Y}]=\xi E [ d t Y ] = ξ
Proof : Let α ~ ∈ Ω \utilde{\alpha}\in\Omega α ∈ Ω E [ α ~ t Y ~ ] = ξ = c ~ t β ~ , ∀ β ~ ∈ R p E[\utilde{\alpha}^t\utilde{Y}]=\xi=\utilde{c}^t\utilde{\beta},\forall\utilde{\beta}\in\R^p E [ α t Y ] = ξ = c t β , ∀ β ∈ R p
⟹ E [ ( α ~ − d ~ ) t Y ~ ] = 0 ⟹ E [ ( α ~ − d ~ ) t D β ~ ] = 0 ∀ β ~ ∈ R p ⟹ ( α ~ − d ~ ) t D = 0 ∀ β ~ ∈ R p ⟹ α ~ − d ~ ∈ Ω ⊥ , but α ~ and d ~ are both in Ω ⟹ α ~ − d ~ ∈ Ω ∩ Ω ⊥ = ∅ ⟹ α ~ = d ~ \begin{align*}
&\implies E[(\utilde{\alpha}-\utilde{d})^t\utilde{Y}]=0\\
&\implies E[(\utilde{\alpha}-\utilde{d})^tD\utilde{\beta}]=0\quad \forall\utilde{\beta}\in\R^p\\
&\implies (\utilde{\alpha}-\utilde{d})^tD=0\quad \forall\utilde{\beta}\in\R^p\\
&\implies \utilde{\alpha}-\utilde{d}\in\Omega^\perp\text{, but }\utilde{\alpha} \text{ and } \utilde{d} \text{ are both in } \Omega\\
&\implies \utilde{\alpha}-\utilde{d}\in \Omega\cap\Omega^\perp=\empty\\
&\implies \utilde{\alpha}=\utilde{d}
\end{align*} ⟹ E [( α − d ) t Y ] = 0 ⟹ E [( α − d ) t D β ] = 0 ∀ β ∈ R p ⟹ ( α − d ) t D = 0 ∀ β ∈ R p ⟹ α − d ∈ Ω ⊥ , but α and d are both in Ω ⟹ α − d ∈ Ω ∩ Ω ⊥ = ∅ ⟹ α = d
i.e. 即使能為 ξ \xi ξ Ω \Omega Ω
令他們共同的投影為 d ~ \utilde{d} d a ~ t D = c ~ t , d ~ t D = c ~ t \utilde{a}^tD=\utilde{c}^t, \utilde{d}^tD=\utilde{c}^t a t D = c t , d t D = c t E [ d ~ t Y ~ ] = ξ = c ~ t β ~ , ∀ β ~ ∈ R p E[\utilde{d}^t\utilde{Y}]=\xi=\utilde{c}^t\utilde{\beta}, \forall\utilde{\beta}\in\R^p E [ d t Y ] = ξ = c t β , ∀ β ∈ R p
Note d ~ ∈ Ω \utilde{d} \in \Omega d ∈ Ω Y ~ − D b ~ = Y ~ − Y ^ ~ = e ~ ∈ Ω ⊥ ⟹ d ~ t e ~ = 0 \utilde{Y}-D\utilde{b}=\utilde{Y}-\utilde{\hat{Y}}=\utilde{e} \in\Omega^\perp\implies \utilde{d}^t\utilde{e}=0 Y − D b = Y − Y ^ = e ∈ Ω ⊥ ⟹ d t e = 0
⟹ d ~ t ( Y ~ − D b ~ ) = d ~ t Y ~ − d ~ t D b ~ = d ~ t Y ~ − c ~ t b ~ = 0 \implies \utilde{d}^t(\utilde{Y}-D\utilde{b})=\utilde{d}^t\utilde{Y}-\utilde{d}^tD\utilde{b}=\utilde{d}^t\utilde{Y}-\utilde{c}^t\utilde{b}=0 ⟹ d t ( Y − D b ) = d t Y − d t D b = d t Y − c t b = 0
i.e. ξ ^ ≜ c ~ t b ~ = d ~ t Y ~ \hat{\xi}\triangleq\utilde{c}^t\utilde{b}=\utilde{d}^t\utilde{Y} ξ ^ ≜ c t b = d t Y
i.e. ξ ^ \hat{\xi} ξ ^ ξ \xi ξ
Now let ξ ~ \tilde{\xi} ξ ~ ξ \xi ξ
i.e. ξ ~ = a ~ t Y ~ \tilde{\xi}=\utilde{a}^t\utilde{Y} ξ ~ = a t Y a ~ ∈ R n \utilde{a}\in\R^n a ∈ R n E [ a ~ t Y ~ ] = ξ = c ~ t β ~ , ∀ β ∈ R p E[\utilde{a}^t\utilde{Y}]=\xi=\utilde{c}^t\utilde{\beta}, \forall\beta\in\R^p E [ a t Y ] = ξ = c t β , ∀ β ∈ R p
Let d ~ \utilde{d} d a ~ \utilde{a} a Ω \Omega Ω
σ 2 { ξ ~ } = σ 2 { a ~ t Y ~ } = a ~ t σ 2 { Y ~ } a ~ = σ 2 ∣ ∣ a ~ ∣ ∣ 2 = σ 2 ∣ ∣ d ~ + ω ~ ∣ ∣ 2 ≥ σ ∣ ∣ d ~ ∣ ∣ 2 = d ~ t σ 2 I d ~ = σ 2 { d ~ t Y ~ } = σ 2 { ξ ^ } \begin{align*}
\sigma^2\set{\tilde{\xi}}&=\sigma^2\set{\utilde{a}^t\utilde{Y}}=\utilde{a}^t\sigma^2\set{\utilde{Y}}\utilde{a}=\sigma^2||\utilde{a}||^2=\sigma^2||\utilde{d}+\utilde{\omega}||^2\\
&\ge \sigma||\utilde{d}||^2=\utilde{d}^t\sigma^2I\utilde{d}=\sigma^2\set{\utilde{d}^t\utilde{Y}}=\sigma^2\set{\hat{\xi}}
\end{align*} σ 2 { ξ ~ } = σ 2 { a t Y } = a t σ 2 { Y } a = σ 2 ∣∣ a ∣ ∣ 2 = σ 2 ∣∣ d + ω ∣ ∣ 2 ≥ σ ∣∣ d ∣ ∣ 2 = d t σ 2 I d = σ 2 { d t Y } = σ 2 { ξ ^ } i.e. σ 2 { ξ ~ ≥ } σ 2 { ξ ^ } , ∀ ξ ~ \sigma^2\set{\tilde{\xi}\ge}\sigma^2\set{\hat{\xi}},\forall \tilde{\xi} σ 2 { ξ ~ ≥ } σ 2 { ξ ^ } , ∀ ξ ~
Remark :
Gauss-Markov Theorem 不需要分配的假設，並且以上證明對於 D D D
對於 ξ = c ~ t β ~ \xi=\utilde{c}^t\utilde{\beta} ξ = c t β
i.e. ∃ a ~ ∈ R n \exist\utilde{a}\in\R^n ∃ a ∈ R n E [ a ~ t Y ~ ] = ξ = c ~ t β ~ , ∀ β ~ ∈ R p E[\utilde{a}^t\utilde{Y}]=\xi=\utilde{c}^t\utilde{\beta},\forall\utilde{\beta}\in\R^p E [ a t Y ] = ξ = c t β , ∀ β ∈ R p
⟹ a ~ t D = c ~ t \implies\utilde{a}^tD=\utilde{c}^t ⟹ a t D = c t
i.e. c ~ t \utilde{c}^t c t
i.e. c ~ t ∈ \utilde{c}^t\in c t ∈
i.e. c ~ t β ~ \utilde{c}^t\utilde{\beta} c t β ⟺ c ~ t ∈ \iff\utilde{c}^t\in ⟺ c t ∈
對於某些設計矩陣 D，c ~ t β ~ \utilde{c}^t\utilde{\beta} c t β
e.g. D = [ 1 0 ⋮ ⋮ 1 0 ] row space = { ( c , 0 ) ; c ∈ R } \text{e.g. }D=\begin{bmatrix*}
1&0\\
\vdots&\vdots\\
1&0
\end{bmatrix*}\quad \text{row space}=\set{(c,0);c\in\R} e.g. D = 1 ⋮ 1 0 ⋮ 0 row space = { ( c , 0 ) ; c ∈ R } ⟺ \iff ⟺ c β 0 , ∀ c ∈ R c\beta_0,\forall c\in\R c β 0 , ∀ c ∈ R β 1 \beta_1 β 1
If row space of D = R p D=\R^p D = R p ( D t D ) − 1 (D^tD)^{-1} ( D t D ) − 1 ⟺ ∀ c ~ ∈ R p , c ~ t β ~ \iff\forall\utilde{c}\in\R^p, \utilde{c}^t\utilde{\beta} ⟺ ∀ c ∈ R p , c t β
如果 rank ( D ) = p \text{rank}(D)=p rank ( D ) = p ( D t D ) − 1 (D^tD)^{-1} ( D t D ) − 1 Gauss-Markov Theorem 有一個更簡單的證明：
σ 2 { ξ ~ } = σ 2 a ~ t I a ~ σ 2 { ξ ^ } = σ 2 { c ~ t b ~ } = σ 2 { c ~ t ( D t D ) − 1 D t Y ~ } = c ~ t ( D t D ) − 1 D t σ 2 { Y ~ } D ( D t D ) − 1 c ~ = σ 2 c ~ t ( D t D ) − 1 D t D ( D t D ) − 1 c ~ = σ 2 c ~ t ( D t D ) − 1 c ~ = σ 2 a ~ t D ( D t D ) − 1 D t a ~ ∵ c ~ t = a ~ t D = σ 2 a ~ t H a ~ σ 2 { ξ ~ } − σ 2 { ξ ^ } = σ 2 a ~ t ( I − H ) a ~ = σ 2 a ~ t M a ~ ≥ 0 ∵ M is Positive definite \begin{align*}
&\sigma^2\set{\tilde{\xi}}=\sigma^2\utilde{a}^tI\utilde{a}\\
&\begin{align*}
\sigma^2\set{\hat{\xi}}&=\sigma^2\set{\utilde{c}^t\utilde{b}}=\sigma^2\set{\utilde{c}^t(D^tD)^{-1}D^t\utilde{Y}}\\
&=\utilde{c}^t(D^tD)^{-1}D^t\sigma^2\set{\utilde{Y}}D(D^tD)^{-1}\utilde{c}\\
&=\sigma^2\utilde{c}^t(D^tD)^{-1}D^tD(D^tD)^{-1}\utilde{c}\\
&=\sigma^2\utilde{c}^t(D^tD)^{-1}\utilde{c}\\
&=\sigma^2\utilde{a}^tD(D^tD)^{-1}D^t\utilde{a}\quad \because \utilde{c}^t=\utilde{a}^tD\\
&=\sigma^2\utilde{a}^tH\utilde{a}
\end{align*}\\
&\sigma^2\set{\tilde{\xi}}-\sigma^2\set{\hat{\xi}}=\sigma^2\utilde{a}^t(I-H)\utilde{a}=\sigma^2\utilde{a}^tM\utilde{a}\ge 0\quad\because M \text{ is Positive definite}
\end{align*} σ 2 { ξ ~ } = σ 2 a t I a σ 2 { ξ ^ } = σ 2 { c t b } = σ 2 { c t ( D t D ) − 1 D t Y } = c t ( D t D ) − 1 D t σ 2 { Y } D ( D t D ) − 1 c = σ 2 c t ( D t D ) − 1 D t D ( D t D ) − 1 c = σ 2 c t ( D t D ) − 1 c = σ 2 a t D ( D t D ) − 1 D t a ∵ c t = a t D = σ 2 a t H a σ 2 { ξ ~ } − σ 2 { ξ ^ } = σ 2 a t ( I − H ) a = σ 2 a t M a ≥ 0 ∵ M is Positive definite
Multivariate Normal
Z 1 , Z 2 , ⋯ , Z n ∼ iid N ( 0 , 1 ) Z_1,Z_2,\cdots, Z_n\overset{\text{iid}}{\sim}N(0,1) Z 1 , Z 2 , ⋯ , Z n ∼ iid N ( 0 , 1 )
Joint pdf f ( z ~ ) = ( 1 2 π ) n e − ∑ i = 1 n z i 2 / 2 = ( 1 2 π ) n e − ∣ ∣ z ∣ ∣ 2 / 2 , z ~ ∈ R n f(\utilde{z})=\left(\frac{1}{\sqrt{2\pi}}\right)^ne^{-\sum_{i=1}^n z^2_i /2}=\left(\frac{1}{\sqrt{2\pi}}\right)^ne^{-||z||^2/2},\utilde{z}\in\R^n f ( z ) = ( 2 π 1 ) n e − ∑ i = 1 n z i 2 /2 = ( 2 π 1 ) n e − ∣∣ z ∣ ∣ 2 /2 , z ∈ R n ∣ ∣ z ~ ∣ ∣ 2 = ∑ i = 1 n z i 2 ∼ χ n 2 ||\utilde{z}||^2=\sum_{i=1}^nz^2_i\sim\chi^2_n ∣∣ z ∣ ∣ 2 = ∑ i = 1 n z i 2 ∼ χ n 2
E [ Z ~ ] = 0 , σ 2 { Z ~ } = I E[\utilde{Z}]=0, \sigma^2\set{\utilde{Z}}=I E [ Z ] = 0 , σ 2 { Z } = I Z ~ ∼ N n ( 0 , I ) \utilde{Z}\sim N_n(0, I) Z ∼ N n ( 0 , I )
Let A m × n , η ~ m × 1 A_{m\times n}, \utilde{\eta}_{m\times 1} A m × n , η m × 1 W ~ m × 1 = A Z ~ n × 1 \utilde{W}_{m\times 1}=A\utilde{Z}_{n\times 1} W m × 1 = A Z n × 1 η ~ \utilde{\eta} η
⟹ E [ W ~ ] = A E [ Z ~ ] + η ~ = η ~ , σ 2 { W ~ } = A A t \implies E[\utilde{W}]=AE[\utilde{Z}]+\utilde{\eta}=\utilde{\eta}, \sigma^2\set{\utilde{W}}=AA^t ⟹ E [ W ] = A E [ Z ] + η = η , σ 2 { W } = A A t W ~ \utilde{W} W
W ~ ∼ N m ( E [ W ~ ] , σ 2 { W ~ } ) = N m ( η ~ , Σ w ~ ) where Σ w ~ = A A t \utilde{W}\sim N_m(E[\utilde{W}], \sigma^2\set{\utilde{W}})=N_m(\utilde{\eta}, \bcancel\Sigma_{\utilde{w}}) \quad \text{where } \bcancel\Sigma_{\utilde{w}}=AA^t W ∼ N m ( E [ W ] , σ 2 { W }) = N m ( η , Σ w ) where Σ w = A A t If Σ W ~ − 1 \bcancel{\Sigma}_{\utilde{W}}^{-1} Σ W − 1 Σ W ~ \bcancel{\Sigma}_{\utilde{W}} Σ W W ~ \utilde{W} W
f W ~ ( w ~ ) = ( 1 2 π ) m ∣ Σ W ~ ∣ − 1 2 e − Q 2 where ∣ Σ W ~ ∣ = det ( Σ W ~ ) , Q = ( w ~ − η ~ ) t Σ W ~ − 1 ( w ~ − η ~ ) f_{\utilde{W}}(\utilde{w})=\left(\frac{1}{\sqrt{2\pi}}\right)^m|\bcancel{\Sigma}_{\utilde{W}}|^{\frac{-1}{2}}e^{-\frac{Q}{2}}\quad \text{where } |\bcancel{\Sigma}_{\utilde{W}}|=\det(\bcancel{\Sigma}_{\utilde{W}}), Q=(\utilde{w}-\utilde{\eta})^t\bcancel{\Sigma}_{\utilde{W}}^{-1}(\utilde{w}-\utilde{\eta}) f W ( w ) = ( 2 π 1 ) m ∣ Σ W ∣ 2 − 1 e − 2 Q where ∣ Σ W ∣ = det ( Σ W ) , Q = ( w − η ) t Σ W − 1 ( w − η ) f W ~ ( w ~ ) = ∏ i = 1 m 1 2 π 1 σ i exp { − ( w i − η i ) 2 2 σ i 2 } f_{\utilde{W}}(\utilde{w})=\prod_{i=1}^m\frac{1}{\sqrt{2\pi}}\frac{1}{\sigma_i}\exp\set{\frac{-(w_i-\eta_i)^2}{2\sigma_i^2}} f W ( w ) = i = 1 ∏ m 2 π 1 σ i 1 exp { 2 σ i 2 − ( w i − η i ) 2 } ⟺ Σ w ~ = diag ( σ 1 2 , σ 2 2 , ⋯ , σ m 2 ) ⟺ σ { w i , w j } = cov ( w i , w j ) = 0 ∀ i ≠ j ⟺ w 1 , w 2 , ⋯ , w m are independent \begin{align*}
&\iff \bcancel{\Sigma}_{\utilde{w}}=\text{diag}(\sigma_1^2,\sigma_2^2,\cdots,\sigma_m^2)\\
&\iff \sigma\set{w_i,w_j}=\text{cov}(w_i,w_j)=0\quad \forall i\ne j\\
&\iff w_1,w_2,\cdots,w_m \text{ are independent}
\end{align*} ⟺ Σ w = diag ( σ 1 2 , σ 2 2 , ⋯ , σ m 2 ) ⟺ σ { w i , w j } = cov ( w i , w j ) = 0 ∀ i = j ⟺ w 1 , w 2 , ⋯ , w m are independent W ~ ∼ N m ( η ~ , Σ W ~ ) \utilde{W}\sim N_m(\utilde{\eta}, \bcancel{\Sigma}_{\utilde{W}}) W ∼ N m ( η , Σ W ) Σ W ~ \bcancel{\Sigma}_{\utilde{W}} Σ W
⟹ Q = ( W ~ − η ~ ) t Σ W ~ − 1 ( W ~ − η ~ ) ∼ χ m 2 \implies Q=(\utilde{W}-\utilde{\eta})^t\bcancel{\Sigma}_{\utilde{W}}^{-1}(\utilde{W}-\utilde{\eta})\sim \chi^2_m ⟹ Q = ( W − η ) t Σ W − 1 ( W − η ) ∼ χ m 2
Proof :
Way 1: Σ W ~ \bcancel{\Sigma}_{\utilde{W}} Σ W ∃ B \exist B ∃ B B Σ W ~ B t = I B\bcancel{\Sigma}_{\utilde{W}}B^t=I B Σ W B t = I
⟹ Σ W ~ = ( B t B ) − 1 \implies \bcancel{\Sigma}_{\utilde{W}}=(B^tB)^{-1} ⟹ Σ W = ( B t B ) − 1 Σ W ~ − 1 = B t B \bcancel{\Sigma}_{\utilde{W}}^{-1}=B^tB Σ W − 1 = B t B
B ( W ~ − η ~ ) ∼ N m ( 0 ~ , B σ 2 W ~ B t ) = N m ( 0 ~ , I ) B(\utilde{W}-\utilde{\eta})\sim N_m(\utilde{0}, B\sigma^2{\utilde{W}}B^t)=N_m(\utilde{0}, I) B ( W − η ) ∼ N m ( 0 , B σ 2 W B t ) = N m ( 0 , I )
⟹ ( B ( W ~ − η ~ ) ) t ( B ( W ~ − η ~ ) ) ∼ χ m 2 \implies (B(\utilde{W}-\utilde{\eta}))^t(B(\utilde{W}-\utilde{\eta}))\sim\chi^2_m ⟹ ( B ( W − η ) ) t ( B ( W − η )) ∼ χ m 2
⟹ ( W ~ − η ~ ) t Σ W ~ − 1 ( W ~ − η ~ ) ∼ χ m 2 \implies(\utilde{W}-\utilde{\eta})^t\bcancel{\Sigma}_{\utilde{W}}^{-1}(\utilde{W}-\utilde{\eta})\sim\chi^2_m ⟹ ( W − η ) t Σ W − 1 ( W − η ) ∼ χ m 2
Way 2:
M Q ( t ) = E [ e t Q ] = ∫ R m f W ~ ( w ~ ) e t Q d w ~ = ∫ R m e t Q ( 1 2 π ) m ∣ Σ W ~ ∣ − 1 2 e − Q 2 d w ~ = ∫ R m ( 1 2 π ) m ∣ Σ W ~ ∣ − 1 2 exp { − 1 2 ( 1 − 2 t ) ( w ~ − η ~ ) t Σ W ~ − 1 ( w ~ − η ~ ) } d w ~ = ∫ R m ( 1 2 π ) m ∣ Σ W ~ ∣ − 1 2 exp { − 1 2 ( w ~ − η ~ ) t ( Σ W ~ 1 − 2 t ) − 1 ( w ~ − η ~ ) t } d w ~ Σ ∗ ≜ Σ W ~ 1 − 2 t = ( 1 − 2 t ) − m 2 ∫ R m ( 1 2 π ) m ∣ Σ ∗ ∣ − 1 2 exp { − 1 2 ( w ~ − η ~ ) t Σ ∗ ( w ~ − η ~ ) } d w ~ = ( 1 − 2 t ) − m 2 = mgf of χ m 2 ⟹ Q ∼ χ m 2 \begin{align*}
M_Q(t)=E[e^{tQ}]&=\int_{\R^m}f_{\utilde{W}}(\utilde{w})e^{tQ}d\utilde{w}\\
&=\int_{\R^m}e^{tQ}\left(\frac{1}{\sqrt{2\pi}}\right)^m|\bcancel{\Sigma}_{\utilde{W}}|^{\frac{-1}{2}}e^{-\frac{Q}{2}}d\utilde{w}\\
&=\int_{\R^m}\left(\frac{1}{\sqrt{2\pi}}\right)^m|\bcancel{\Sigma}_{\utilde{W}}|^{\frac{-1}{2}}\exp\set{\frac{-1}{2}(1-2t)(\utilde{w}-\utilde{\eta})^t\bcancel{\Sigma}_{\utilde{W}}^{-1}(\utilde{w}-\utilde{\eta})}d\utilde{w}\\
&=\int_{\R^m}\left(\frac{1}{\sqrt{2\pi}}\right)^m|\bcancel{\Sigma}_{\utilde{W}}|^{\frac{-1}{2}}\exp\set{\frac{-1}{2}(\utilde{w}-\utilde{\eta})^t(\frac{\bcancel{\Sigma}_{\utilde{W}}}{1-2t})^{-1}(\utilde{w}-\utilde{\eta})^t}d\utilde{w}\\
\bcancel{\Sigma}^*\triangleq\frac{\bcancel{\Sigma}_{\utilde{W}}}{1-2t}\quad &=(1-2t)^{-\frac{m}{2}}\int_{\R^m}\left(\frac{1}{\sqrt{2\pi}}\right)^m|\bcancel{\Sigma}^*|^{\frac{-1}{2}}\exp\set{\frac{-1}{2}(\utilde{w}-\utilde{\eta})^t\bcancel{\Sigma}^*(\utilde{w}-\utilde{\eta})}d\utilde{w}\\
&=(1-2t)^{-\frac{m}{2}}\\
&=\text{mgf of }\chi^2_m\\
&\implies Q\sim\chi^2_m
\end{align*} M Q ( t ) = E [ e tQ ] Σ ∗ ≜ 1 − 2 t Σ W = ∫ R m f W ( w ) e tQ d w = ∫ R m e tQ ( 2 π 1 ) m ∣ Σ W ∣ 2 − 1 e − 2 Q d w = ∫ R m ( 2 π 1 ) m ∣ Σ W ∣ 2 − 1 exp { 2 − 1 ( 1 − 2 t ) ( w − η ) t Σ W − 1 ( w − η ) } d w = ∫ R m ( 2 π 1 ) m ∣ Σ W ∣ 2 − 1 exp { 2 − 1 ( w − η ) t ( 1 − 2 t Σ W ) − 1 ( w − η ) t } d w = ( 1 − 2 t ) − 2 m ∫ R m ( 2 π 1 ) m ∣ Σ ∗ ∣ 2 − 1 exp { 2 − 1 ( w − η ) t Σ ∗ ( w − η ) } d w = ( 1 − 2 t ) − 2 m = mgf of χ m 2 ⟹ Q ∼ χ m 2
non-central chi-square :
W i ∼ N ( θ i , 1 ) , i = 1 , ⋯ , n W_i\sim N(\theta_i, 1), i=1,\cdots, n W i ∼ N ( θ i , 1 ) , i = 1 , ⋯ , n W ~ ∼ N n ( θ ~ , I ) \utilde{W}\sim N_n(\utilde{\theta}, I) W ∼ N n ( θ , I )
∣ ∣ W ~ ∣ ∣ 2 = ∑ i = 1 n W i 2 ∼ χ n , δ 2 where δ = ∣ ∣ θ ~ ∣ ∣ 2 = ∑ i = 1 n θ i 2 ||\utilde{W}||^2=\sum_{i=1}^nW_i^2\sim\chi^2_{n,\delta}\text{ where }\delta=||\utilde{\theta}||^2=\sum_{i=1}^n\theta_i^2 ∣∣ W ∣ ∣ 2 = i = 1 ∑ n W i 2 ∼ χ n , δ 2 where δ = ∣∣ θ ∣ ∣ 2 = i = 1 ∑ n θ i 2
n n n degree of freedom δ \delta δ non-centrality Note: χ n , 0 2 = χ n 2 \chi^2_{n,0}=\chi^2_n χ n , 0 2 = χ n 2
Note:
u = ∣ ∣ W ~ ∣ ∣ 2 ∼ χ n , δ 2 where δ = ∣ ∣ θ ~ ∣ ∣ 2 = d ∑ i = 1 n N ( θ i , 1 ) 2 δ = ∑ i = 1 n θ i 2 = ( ∑ i = 1 n θ i 2 ) 2 = d N ( ∣ ∣ θ ~ ∣ ∣ , 1 ) + ∑ i = 1 n − 1 N ( 0 , 1 ) 2 = d χ 1 , δ 2 + χ n − 1 2 \begin{align*}
u&=||\utilde{W}||^2\sim\chi^2_{n,\delta}\text{ where }\delta=||\utilde{\theta}||^2\\
&\overset{\text{d}}{=}\sum_{i=1}^nN(\theta_i, 1)^2\quad\delta=\sum_{i=1}^n\theta_i^2=\left(\sqrt{\sum_{i=1}^n\theta_i^2}\right)^2\\
&\overset{\text{d}}{=}N(||\utilde{\theta}||, 1) + \sum_{i=1}^{n-1}N(0,1)^2\\
&\overset{\text{d}}{=}\chi^2_{1, \delta}+\chi^2_{n-1}
\end{align*} u = ∣∣ W ∣ ∣ 2 ∼ χ n , δ 2 where δ = ∣∣ θ ∣ ∣ 2 = d i = 1 ∑ n N ( θ i , 1 ) 2 δ = i = 1 ∑ n θ i 2 = ( i = 1 ∑ n θ i 2 ) 2 = d N ( ∣∣ θ ∣∣ , 1 ) + i = 1 ∑ n − 1 N ( 0 , 1 ) 2 = d χ 1 , δ 2 + χ n − 1 2 E [ u ] = E [ ( Z + ∣ ∣ θ ∣ ∣ ) 2 ] + n − 1 = E [ Z 2 ] + 2 ∣ ∣ θ ∣ ∣ E [ Z ] + ∣ ∣ θ ∣ ∣ 2 + n − 1 = n + ∣ ∣ θ ∣ ∣ 2 i.e. E [ χ n , δ 2 ] = n + δ \begin{align*}
E[u]&=E[(Z+||\theta||)^2]+n-1\\
&=E[Z^2]+2||\theta||E[Z]+||\theta||^2+n-1\\
&=n+||\theta||^2\\
\text{i.e. }& E[\chi^2_{n,\delta}]=n+\delta
\end{align*} E [ u ] i.e. = E [( Z + ∣∣ θ ∣∣ ) 2 ] + n − 1 = E [ Z 2 ] + 2∣∣ θ ∣∣ E [ Z ] + ∣∣ θ ∣ ∣ 2 + n − 1 = n + ∣∣ θ ∣ ∣ 2 E [ χ n , δ 2 ] = n + δ V a r [ u ] = V a r [ ( Z + ∣ ∣ θ ∣ ∣ ) 2 ] + 2 ( n − 1 ) = E [ ( Z + ∣ ∣ θ ∣ ∣ ) 4 ] − E [ ( Z + ∣ ∣ θ ∣ ∣ ) 2 ] 2 + 2 ( n − 1 ) = 2 n + 4 δ \begin{align*}
Var[u]&=Var[(Z+||\theta||)^2]+2(n-1)\\
&=E[(Z+||\theta||)^4]-E[(Z+||\theta||)^2]^2+2(n-1)\\
&=2n+4\delta
\end{align*} Va r [ u ] = Va r [( Z + ∣∣ θ ∣∣ ) 2 ] + 2 ( n − 1 ) = E [( Z + ∣∣ θ ∣∣ ) 4 ] − E [( Z + ∣∣ θ ∣∣ ) 2 ] 2 + 2 ( n − 1 ) = 2 n + 4 δ U 1 ∼ χ n 1 , δ 1 2 , U 2 ∼ χ n 2 , δ 2 2 independent ⟹ U 1 + U 2 ∼ χ n 1 + n 2 , δ 1 + δ 2 2 U_1\sim\chi^2_{n_1,\delta_1}, U_2\sim\chi^2_{n_2,\delta_2}\text{ independent}\implies U_1+U_2\sim\chi^2_{n_1+n_2, \delta_1+\delta_2} U 1 ∼ χ n 1 , δ 1 2 , U 2 ∼ χ n 2 , δ 2 2 independent ⟹ U 1 + U 2 ∼ χ n 1 + n 2 , δ 1 + δ 2 2 non-central F, t :
F m , n , δ = d χ m , δ 2 / m χ n 2 / n t m , δ = d N ( δ , 1 ) χ m 2 / m 分子分母均獨立 F_{m,n,\delta}\overset{\text{d}}{=}\frac{\chi^2_{m,\delta}/m}{\chi^2_n/n} \qquad t_{m,\delta}\overset{\text{d}}{=}\frac{N(\sqrt{\delta},1)}{\sqrt{\chi^2_m/m}}\quad\text{分子分母均獨立} F m , n , δ = d χ n 2 / n χ m , δ 2 / m t m , δ = d χ m 2 / m N ( δ , 1 ) 分子分母均獨立 ⟹ ( t m , δ ) 2 = d F 1 , m , δ \implies (t_{m,\delta})^2\overset{\text{d}}{=}F_{1,m,\delta} ⟹ ( t m , δ ) 2 = d F 1 , m , δ
Y ~ = D β ~ + ε ~ = θ ~ + ε ~ where ε ~ ∼ N n ( 0 ~ , I ) \utilde{Y}=D\utilde{\beta}+\utilde{\varepsilon}=\utilde{\theta}+\utilde{\varepsilon}\quad\text{where }\utilde{\varepsilon}\sim N_n(\utilde{0}, I) Y = D β + ε = θ + ε where ε ∼ N n ( 0 , I ) ⟹ Y ~ ∼ N n ( θ ~ , σ 2 I ) \implies \utilde{Y}\sim N_n(\utilde{\theta}, \sigma^2I) ⟹ Y ∼ N n ( θ , σ 2 I )
Let Y ~ ∼ N n ( θ ~ , σ 2 I ) \utilde{Y}\sim N_n(\utilde{\theta}, \sigma^2I) Y ∼ N n ( θ , σ 2 I ) P P P n × n n\times n n × n r r r
⟹ Y ~ t P Y ~ σ 2 ∼ χ r , δ 2 where δ = θ ~ t P θ ~ σ 2 \implies \frac{\utilde{Y}^tP\utilde{Y}}{\sigma^2}\sim\chi^2_{r, \delta}\text{ where }\delta=\frac{\utilde{\theta}^tP\utilde{\theta}}{\sigma^2} ⟹ σ 2 Y t P Y ∼ χ r , δ 2 where δ = σ 2 θ t P θ Proof : ∃ A n × n = ( a 1 ~ , ⋯ , a n ~ ) \exist A_{n\times n}=(\utilde{a_1}, \cdots, \utilde{a_n}) ∃ A n × n = ( a 1 , ⋯ , a n )
A t P A = diag ( 1 , 1 , ⋯ , 1 ⏟ r , 0 , 0 , ⋯ , 0 ⏟ n − r ) = [ I r 0 0 0 ] \begin{align*}
A^tPA&=\text{diag}(\underbrace{1,1,\cdots,1}_{r},\underbrace{0,0,\cdots,0}_{n-r})\\
&=\begin{bmatrix*}
I_r&0\\
0&0
\end{bmatrix*}
\end{align*} A t P A = diag ( r 1 , 1 , ⋯ , 1 , n − r 0 , 0 , ⋯ , 0 ) = [ I r 0 0 0 ] ⟹ P = A [ I r 0 0 0 ] A t = ( a 1 ~ , ⋯ , a r ~ ) ( a 1 ~ t ⋮ a r ~ t ) = A r A r t ( △ 1 ) \implies P=A\begin{bmatrix*}
I_r&0\\
0&0
\end{bmatrix*}A^t=(\utilde{a_1}, \cdots, \utilde{a_r})\begin{pmatrix}
\utilde{a_1}^t\\
\vdots\\
\utilde{a_r}^t\\
\end{pmatrix}=A_rA_r^t\tag{$\vartriangle_1$} ⟹ P = A [ I r 0 0 0 ] A t = ( a 1 , ⋯ , a r ) a 1 t ⋮ a r t = A r A r t ( △ 1 ) Let W ~ = A t Y ~ ⟺ A W ~ = Y ~ \utilde{W}=A^t\utilde{Y}\iff A\utilde{W}=\utilde{Y} W = A t Y ⟺ A W = Y
⟹ Y ~ t P Y ~ = ( A W ~ ) t P ( A W ~ ) = W ~ t A t P A W ~ = W ~ t [ I r 0 0 0 ] W ~ = ∑ i = 1 r W i 2 ( △ 2 ) \implies \utilde{Y}^tP\utilde{Y}=(A\utilde{W})^tP(A\utilde{W})=\utilde{W}^tA^tPA\utilde{W}=\utilde{W}^t\begin{bmatrix*}
I_r&0\\
0&0
\end{bmatrix*}\utilde{W}=\sum_{i=1}^rW_i^2\tag{$\vartriangle_2$} ⟹ Y t P Y = ( A W ) t P ( A W ) = W t A t P A W = W t [ I r 0 0 0 ] W = i = 1 ∑ r W i 2 ( △ 2 ) ∵ Y ~ ∼ N n ( θ ~ , σ 2 I ) \because\utilde{Y}\sim N_n(\utilde{\theta}, \sigma^2I) ∵ Y ∼ N n ( θ , σ 2 I )
∴ W ~ = A t Y ~ ∼ N n ( A t θ ~ , σ 2 I ) \therefore\utilde{W}=A^t\utilde{Y}\sim N_n(A^t\utilde{\theta}, \sigma^2I) ∴ W = A t Y ∼ N n ( A t θ , σ 2 I ) σ 2 { A t Y ~ } = σ 2 A I A t = σ 2 I \sigma^2\set{A^t\utilde{Y}}=\sigma^2AIA^t=\sigma^2I σ 2 { A t Y } = σ 2 A I A t = σ 2 I
W i , i = 1 , ⋯ , n W_i, i=1,\cdots, n W i , i = 1 , ⋯ , n
⟹ ∑ i = 1 r W i 2 σ 2 ∼ χ r , δ 2 with δ = 1 σ 2 ∑ i = 1 r ( a i ~ 1 × n t θ ~ n × 1 ) 2 = θ ~ t A r A r t θ ~ σ 2 = △ 1 by θ ~ t P θ ~ σ 2 \implies \sum_{i=1}^r\frac{W_i^2}{\sigma^2}\sim\chi^2_{r, \delta} \text{ with }\delta =\frac{1}{\sigma^2}\sum_{i=1}^r(\utilde{a_i}_{1\times n}^t\utilde{\theta}_{n\times 1})^2=\frac{\utilde{\theta}^tA_rA_r^t\utilde{\theta}}{\sigma^2}\xlongequal[\vartriangle_1]{\text{by}}\frac{\utilde{\theta}^tP\utilde{\theta}}{\sigma^2} ⟹ i = 1 ∑ r σ 2 W i 2 ∼ χ r , δ 2 with δ = σ 2 1 i = 1 ∑ r ( a i 1 × n t θ n × 1 ) 2 = σ 2 θ t A r A r t θ by △ 1 σ 2 θ t P θ ⇒ by △ 2 1 σ 2 ∑ i = 1 r W i 2 = Y ~ t P Y ~ σ 2 ∼ χ r , δ 2 where δ = θ ~ t P θ ~ σ 2 = E [ Y ~ ] t P E [ Y ~ ] σ 2 \xRightarrow{\text{by }\vartriangle_2}\frac{1}{\sigma^2}\sum_{i=1}^rW_i^2=\frac{\utilde{Y}^tP\utilde{Y}}{\sigma^2}\sim\chi^2_{r, \delta}\text{ where }\delta=\frac{\utilde{\theta}^tP\utilde{\theta}}{\sigma^2}=\frac{E[\utilde{Y}]^tPE[\utilde{Y}]}{\sigma^2} by △ 2 σ 2 1 i = 1 ∑ r W i 2 = σ 2 Y t P Y ∼ χ r , δ 2 where δ = σ 2 θ t P θ = σ 2 E [ Y ] t PE [ Y ] ⟹ SSTO σ 2 = Y ~ t ( I − J n ) Y ~ σ 2 ∼ χ n , δ 2 with δ = ∑ ( θ i − θ ˉ ) t σ 2 SSR σ 2 = Y ~ t ( H − J n ) Y ~ σ 2 ∼ χ p − 1 , δ 2 with δ = θ ~ t ( H − J n ) θ ~ σ 2 SSE σ 2 = Y ~ t ( I − H ) Y ~ σ 2 ∼ χ n − r , δ 2 with δ = θ ~ t ( I − H ) θ ~ σ 2 \begin{align*}
\implies& \frac{\text{SSTO}}{\sigma^2}=\frac{\utilde{Y}^t(I-\frac{J}{n})\utilde{Y}}{\sigma^2}\sim\chi^2_{n, \delta}\text{ with } \delta=\frac{\sum(\theta_i-\bar{\theta})^t}{\sigma^2}\\
&\frac{\text{SSR}}{\sigma^2}=\frac{\utilde{Y}^t(H-\frac{J}{n})\utilde{Y}}{\sigma^2}\sim\chi^2_{p-1, \delta}\text{ with }\delta=\frac{\utilde{\theta}^t(H-\frac{J}{n})\utilde{\theta}}{\sigma^2}\\
&\frac{\text{SSE}}{\sigma^2}=\frac{\utilde{Y}^t(I-H)\utilde{Y}}{\sigma^2}\sim\chi^2_{n-r, \delta} \text{ with }\delta = \frac{\utilde{\theta}^t(I-H)\utilde{\theta}}{\sigma^2}
\end{align*} ⟹ σ 2 SSTO = σ 2 Y t ( I − n J ) Y ∼ χ n , δ 2 with δ = σ 2 ∑ ( θ i − θ ˉ ) t σ 2 SSR = σ 2 Y t ( H − n J ) Y ∼ χ p − 1 , δ 2 with δ = σ 2 θ t ( H − n J ) θ σ 2 SSE = σ 2 Y t ( I − H ) Y ∼ χ n − r , δ 2 with δ = σ 2 θ t ( I − H ) θ 我們知道互相獨立的 χ 2 \chi^2 χ 2 χ 2 \chi^2 χ 2 χ 2 \chi^2 χ 2 χ 2 \chi^2 χ 2
這裡我們發現 SSTO, SSR, SSE 都是 χ 2 \chi^2 χ 2
Cochran's Theorem :
Y ~ ∼ N n ( θ ~ , σ 2 I ) \utilde{Y}\sim N_n(\utilde{\theta},\sigma^2 I) Y ∼ N n ( θ , σ 2 I )
Let P j , j = 1 , 2 , ⋯ , m : n × n P_j, j=1,2,\cdots, m: n\times n P j , j = 1 , 2 , ⋯ , m : n × n r j r_j r j I = ∑ j = 1 m P j I=\sum_{j=1^m}P_j I = ∑ j = 1 m P j
Hence Y ~ t I Y ~ = ∑ j = 1 m Y ~ t P j Y ~ \utilde{Y}^tI\utilde{Y}=\sum_{j=1}^m\utilde{Y}^tP_j\utilde{Y} Y t I Y = ∑ j = 1 m Y t P j Y
⟹ j = 1 , 2 , ⋯ , m Y ~ t P Y ~ σ 2 ∼ χ r j , δ j 2 with δ j = θ ~ t P j θ ~ σ 2 are independent ⟺ ∑ j = 1 m r j = n \implies j=1,2,\cdots,m \quad\frac{\utilde{Y}^tP\utilde{Y}}{\sigma^2}\sim\chi^2_{r_j,\delta_j}\text{ with }\delta_j=\frac{\utilde{\theta}^tP_j\utilde{\theta}}{\sigma^2} \text{ are independent}\iff \sum_{j=1}^m r_j=n ⟹ j = 1 , 2 , ⋯ , m σ 2 Y t P Y ∼ χ r j , δ j 2 with δ j = σ 2 θ t P j θ are independent ⟺ j = 1 ∑ m r j = n Note :
If ( D t D ) − 1 (D^tD)^{-1} ( D t D ) − 1 rank ( D ) = p = dim ( Ω ) = tr ( H ) \text{rank}(D)=p=\dim(\Omega)=\text{tr}(H) rank ( D ) = p = dim ( Ω ) = tr ( H )
rank:
n p n − p I = H + ( I − H ) \begin{alignat*}{3}
&n &p& &n-p\\
&I=&H&+&(I-H)
\end{alignat*} n I = p H + n − p ( I − H ) ⟹ Y ~ t H Y ~ σ 2 = ∣ ∣ Y ^ ∣ ∣ 2 σ 2 ∼ χ p , θ ~ t H θ ~ σ 2 2 = χ p , ∣ ∣ θ ∣ ∣ 2 σ 2 2 Y ~ t M Y ~ σ 2 = SSE σ 2 ∼ χ n − p , θ ~ t M θ ~ σ 2 = χ n − p 2 \implies \begin{align*}
&\frac{\utilde{Y}^tH\utilde{Y}}{\sigma^2}=\frac{||\hat{Y}||^2}{\sigma^2}\sim\chi^2_{p,\frac{\utilde{\theta}^tH\utilde{\theta}}{\sigma^2}}=\chi^2_{p,\frac{||\theta||^2}{\sigma^2}}\\
&\frac{\utilde{Y}^tM\utilde{Y}}{\sigma^2}=\frac{\text{SSE}}{\sigma^2}\sim\chi_{n-p,\frac{\utilde{\theta}^tM\utilde{\theta}}{\sigma^2}}=\chi^2_{n-p}
\end{align*} ⟹ σ 2 Y t H Y = σ 2 ∣∣ Y ^ ∣ ∣ 2 ∼ χ p , σ 2 θ t H θ 2 = χ p , σ 2 ∣∣ θ ∣ ∣ 2 2 σ 2 Y t M Y = σ 2 SSE ∼ χ n − p , σ 2 θ t M θ = χ n − p 2
rank:
n = n − p p − 1 1 I = ( I − H ) + ( H − J n ) + J n \begin{alignat*}{4}
&n&=&n-p&&p-1&&1\\
&I&=&(I-H)&+&(H-\frac{J}{n})&+&\frac{J}{n}
\end{alignat*} n I = = n − p ( I − H ) + p − 1 ( H − n J ) + 1 n J ⟹ Y ~ t ( I − H ) Y ~ σ 2 = SSE σ 2 ∼ χ n − p 2 ∵ θ ~ t ( I − H ) θ ~ = 0 ∵ θ ~ ∈ Ω Y ~ t ( H − J n ) Y ~ σ 2 = SSR σ 2 ∼ χ p − 1 , θ ~ t ( H − J n ) θ ~ σ 2 = ∑ ( θ i − θ ˉ ) 2 σ 2 = ∣ ∣ θ ∣ ∣ 2 − n θ ˉ 2 σ 2 2 Y ~ t J n Y ~ σ 2 = n Y ˉ 2 σ 2 ∼ χ 1 , θ ~ t J n θ ~ σ 2 = n θ ˉ 2 σ 2 2 \implies\begin{alignat*}{2}
&\frac{\utilde{Y}^t(I-H)\utilde{Y}}{\sigma^2}&=&\frac{\text{SSE}}{\sigma^2}\sim\chi^2_{n-p}\quad\because\utilde{\theta}^t(I-H)\utilde{\theta}=0 \because \utilde{\theta}\in\Omega\\
&\frac{\utilde{Y}^t(H-\frac{J}{n})\utilde{Y}}{\sigma^2}&=&\frac{\text{SSR}}{\sigma^2}\sim\chi^2_{p-1,\frac{\utilde{\theta}^t(H-\frac{J}{n})\utilde{\theta}}{\sigma^2}=\frac{\sum(\theta_i-\bar{\theta})^2}{\sigma^2}=\frac{||\theta||^2-n\bar{\theta}^2}{\sigma^2}}\\
&\frac{\utilde{Y}^t\frac{J}{n}\utilde{Y}}{\sigma^2}&=&\frac{n\bar{Y}^2}{\sigma^2}\sim\chi^2_{1, \frac{\utilde{\theta}^t\frac{J}{n}\utilde{\theta}}{\sigma^2}=\frac{n\bar{\theta}^2}{\sigma^2}}
\end{alignat*} ⟹ σ 2 Y t ( I − H ) Y σ 2 Y t ( H − n J ) Y σ 2 Y t n J Y = = = σ 2 SSE ∼ χ n − p 2 ∵ θ t ( I − H ) θ = 0 ∵ θ ∈ Ω σ 2 SSR ∼ χ p − 1 , σ 2 θ t ( H − n J ) θ = σ 2 ∑ ( θ i − θ ˉ ) 2 = σ 2 ∣∣ θ ∣ ∣ 2 − n θ ˉ 2 2 σ 2 n Y ˉ 2 ∼ χ 1 , σ 2 θ t n J θ = σ 2 n θ ˉ 2 2
Remark :
Y ~ n × 1 = D n × p β ~ p × 1 + ε ~ n × 1 where β ~ = ( β 0 , β 1 , ⋯ , β k ) t , p = k + 1 < n , ε ~ ∼ N n ( 0 ~ , σ 2 I ) \utilde{Y}_{n\times 1}=D_{n\times p}\utilde{\beta}_{p\times 1}+\utilde{\varepsilon}_{n\times 1}\quad\text{where }\utilde{\beta}=(\beta_0, \beta_1, \cdots, \beta_k)^t,p=k+1<n,\utilde{\varepsilon}\sim N_n(\utilde{0}, \sigma^2I) Y n × 1 = D n × p β p × 1 + ε n × 1 where β = ( β 0 , β 1 , ⋯ , β k ) t , p = k + 1 < n , ε ∼ N n ( 0 , σ 2 I )
E [ Y ~ ] = D β ~ ≜ θ ~ ∈ Ω E[\utilde{Y}]=D\utilde{\beta}\triangleq\utilde{\theta}\in\Omega E [ Y ] = D β ≜ θ ∈ Ω Ω = \Omega= Ω = D D D
If D D D ⟺ rank ( D ) = p = dim ( Ω ) ⟺ ( D t D ) p × p − 1 \iff \text{rank}(D)=p=\dim(\Omega)\iff (D^tD)^{-1}_{p\times p} ⟺ rank ( D ) = p = dim ( Ω ) ⟺ ( D t D ) p × p − 1
b ~ = ( D t D ) − 1 D t Y \utilde{b}=(D^tD)^{-1}D^tY b = ( D t D ) − 1 D t Y Y ^ ~ = D b ~ = D ( D t D ) − 1 D t Y ~ = H Y ~ \utilde{\hat{Y}}=D\utilde{b}=D(D^tD)^{-1}D^t\utilde{Y}=H\utilde{Y} Y ^ = D b = D ( D t D ) − 1 D t Y = H Y
SSR σ 2 = Y ~ t ( H − J n Y ~ ) σ 2 ∼ χ p − 1 , θ ~ t ( H − J n ) θ ~ σ 2 = ∑ ( θ i − θ ˉ ) 2 σ 2 2 SSE σ 2 = Y ~ t ( I − H ) Y ~ σ 2 ∼ χ n − p , θ ~ t ( I − H ) θ ~ σ 2 = 0 2 \begin{align*}
&\frac{\text{SSR}}{\sigma^2}=\frac{\utilde{Y}^t(H-\frac{J}{n}\utilde{Y})}{\sigma^2}\sim\chi^2_{p-1, \frac{\utilde{\theta}^t(H-\frac{J}{n})\utilde{\theta}}{\sigma^2}=\frac{\sum(\theta_i-\bar{\theta})^2}{\sigma^2}}\\
&\frac{\text{SSE}}{\sigma^2}=\frac{\utilde{Y}^t(I-H)\utilde{Y}}{\sigma^2}\sim\chi^2_{n-p, \frac{\utilde{\theta}^t(I-H)\utilde{\theta}}{\sigma^2}=0}\\
\end{align*} σ 2 SSR = σ 2 Y t ( H − n J Y ) ∼ χ p − 1 , σ 2 θ t ( H − n J ) θ = σ 2 ∑ ( θ i − θ ˉ ) 2 2 σ 2 SSE = σ 2 Y t ( I − H ) Y ∼ χ n − p , σ 2 θ t ( I − H ) θ = 0 2 ⟹ SSTO σ 2 = SSR σ 2 + SSE σ 2 ∼ χ n − 1 , ( θ i − θ ˉ ) 2 σ 2 = ∣ ∣ θ ~ ∣ ∣ 2 − n θ ˉ 2 σ 2 2 \implies \frac{\text{SSTO}}{\sigma^2}=\frac{\text{SSR}}{\sigma^2}+\frac{\text{SSE}}{\sigma^2}\sim\chi^2_{n-1,\frac{(\theta_i-\bar{\theta})^2}{\sigma^2}=\frac{||\utilde{\theta}||^2-n\bar{\theta}^2}{\sigma^2}} ⟹ σ 2 SSTO = σ 2 SSR + σ 2 SSE ∼ χ n − 1 , σ 2 ( θ i − θ ˉ ) 2 = σ 2 ∣∣ θ ∣ ∣ 2 − n θ ˉ 2 2 Let MS ≜ SS df \triangleq\frac{\text{SS}}{\text{df}} ≜ df SS SSE n − p \frac{\text{SSE}}{n-p} n − p SSE SSR p − 1 \frac{\text{SSR}}{p-1} p − 1 SSR
E [ MSE ] = E [ SSE n − p ] = σ 2 n − p E [ SSE σ 2 ⏟ ∼ χ n − p 2 ] = σ 2 n − p ( n − p ) = σ 2 E[\text{MSE}]=E[\frac{\text{SSE}}{n-p}]=\frac{\sigma^2}{n-p}E[\underbrace{\frac{\text{SSE}}{\sigma^2}}_{\sim\chi^2_{n-p}}]=\frac{\sigma^2}{n-p}(n-p)=\sigma^2 E [ MSE ] = E [ n − p SSE ] = n − p σ 2 E [ ∼ χ n − p 2 σ 2 SSE ] = n − p σ 2 ( n − p ) = σ 2
E [ MSR ] = σ 2 p − 1 E [ SSR σ 2 ] SSR σ 2 ∼ χ p − 1 , ∑ ( θ i − θ ˉ ) 2 σ 2 2 = σ 2 p − 1 ( p − 1 + ∑ ( θ i − θ ˉ ) 2 σ 2 ) = σ 2 + 1 p − 1 ∑ ( θ i − θ ˉ ) 2 ≥ 0 \begin{align*}
E[\text{MSR}]&=\frac{\sigma^2}{p-1}E[\frac{\text{SSR}}{\sigma^2}]\quad \frac{\text{SSR}}{\sigma^2}\sim\chi^2_{p-1,\frac{\sum(\theta_i-\bar{\theta})^2}{\sigma^2}}\\
&=\frac{\sigma^2}{p-1}(p-1+\frac{\sum(\theta_i-\bar{\theta})^2}{\sigma^2})\\
&=\sigma^2+\frac{1}{p-1}\sum(\theta_i-\bar{\theta})^2\ge 0
\end{align*} E [ MSR ] = p − 1 σ 2 E [ σ 2 SSR ] σ 2 SSR ∼ χ p − 1 , σ 2 ∑ ( θ i − θ ˉ ) 2 2 = p − 1 σ 2 ( p − 1 + σ 2 ∑ ( θ i − θ ˉ ) 2 ) = σ 2 + p − 1 1 ∑ ( θ i − θ ˉ ) 2 ≥ 0
Note :
E [ MSR ] = σ 2 + 1 p − 1 ∑ ( θ i − θ ˉ ) 2 ≥ 0 " = " holds ⟺ θ i = θ ˉ ∀ i E[\text{MSR}]=\sigma^2+\frac{1}{p-1}\sum(\theta_i-\bar{\theta})^2\ge 0\quad "="\text{ holds }\iff\theta_i=\bar{\theta}\forall i E [ MSR ] = σ 2 + p − 1 1 ∑ ( θ i − θ ˉ ) 2 ≥ 0 " = " holds ⟺ θ i = θ ˉ ∀ i Hence E [ MSE ] = σ 2 ≤ σ 2 + 1 p − 1 ∑ ( θ i − θ ˉ ) 2 = E [ MSR ] \text{Hence } E[\text{MSE}]=\sigma^2\le\sigma^2+\frac{1}{p-1}\sum(\theta_i-\bar{\theta})^2=E[\text{MSR}] Hence E [ MSE ] = σ 2 ≤ σ 2 + p − 1 1 ∑ ( θ i − θ ˉ ) 2 = E [ MSR ] with "=" hold ⟺ β 1 = β 2 = ⋯ = β k = 0 \iff \beta_1=\beta_2=\cdots=\beta_k=0 ⟺ β 1 = β 2 = ⋯ = β k = 0 Y ~ \utilde{Y} Y X ~ \utilde{X} X ⟺ Y i = β 0 + ε i \iff Y_i=\beta_0+\varepsilon_i ⟺ Y i = β 0 + ε i
因為我們通常會關注解釋變數和反應變數之間是否真的有線性關係，所以我們會做假設檢定：
H 0 : β 1 = β 2 = ⋯ = β k = 0 v.s. H 1 : β j ≠ 0 for some j = 1 , 2 , ⋯ , k H_0:\beta_1=\beta_2=\cdots=\beta_k=0\quad\text{ v.s. }\quad H_1:\beta_j\ne 0\text{ for some }j=1,2,\cdots,k H 0 : β 1 = β 2 = ⋯ = β k = 0 v.s. H 1 : β j = 0 for some j = 1 , 2 , ⋯ , k 注：β 0 \beta_0 β 0
Under H 0 : β 1 = β 2 = ⋯ = β k = 0 H_0:\beta_1=\beta_2=\cdots=\beta_k=0 H 0 : β 1 = β 2 = ⋯ = β k = 0
E [ MSE ] = σ 2 = E [ MSR ] ⟹ MSR MSE 傾向接近 1 E[\text{MSE}]=\sigma^2=E[\text{MSR}]\implies \frac{\text{MSR}}{\text{MSE}}\text{ 傾向接近 }1 E [ MSE ] = σ 2 = E [ MSR ] ⟹ MSE MSR 傾向接近 1
Under H 1 : β j ≠ 0 for some j = 1 , 2 , ⋯ , k H_1:\beta_j\ne 0\text{ for some }j=1,2,\cdots,k H 1 : β j = 0 for some j = 1 , 2 , ⋯ , k
E [ MSE ] = σ 2 < E [ MSR ] ⟹ MSR MSE 傾向大於 1 E[\text{MSE}]=\sigma^2<E[\text{MSR}]\implies \frac{\text{MSR}}{\text{MSE}}\text{ 傾向大於 }1 E [ MSE ] = σ 2 < E [ MSR ] ⟹ MSE MSR 傾向大於 1
Test Statistic :
F ∗ ≜ MSR MSE F^*\triangleq\frac{\text{MSR}}{\text{MSE}} F ∗ ≜ MSE MSR reject H 0 ⟺ F ∗ > c H_0\iff F^*>c H 0 ⟺ F ∗ > c c c c P ( F ∗ > c ∣ H 0 ) = α P(F^*>c|H_0)=\alpha P ( F ∗ > c ∣ H 0 ) = α
Note : Under H 0 : β 1 = β 2 = ⋯ = β k = 0 H_0:\beta_1=\beta_2=\cdots=\beta_k=0 H 0 : β 1 = β 2 = ⋯ = β k = 0
⊥ < SSR σ 2 ∼ χ p − 1 , ∑ ( θ i − θ ˉ ) 2 σ 2 = H 0 0 2 SSE σ 2 ∼ χ n − p 2 \perp\left<\begin{align*}
&\frac{\text{SSR}}{\sigma^2}\sim\chi^2_{p-1,\frac{\sum(\theta_i-\bar{\theta})^2}{\sigma^2}\xlongequal{H_0}0}\\
&\frac{\text{SSE}}{\sigma^2}\sim\chi^2_{n-p}
\end{align*}
\right. ⊥ ⟨ σ 2 SSR ∼ χ p − 1 , σ 2 ∑ ( θ i − θ ˉ ) 2 H 0 0 2 σ 2 SSE ∼ χ n − p 2 ⟹ F ∗ = MSR MSE = SSR / p − 1 SSER / n − p = 1 p − 1 SSR σ 2 ⏞ χ p − 1 2 1 n − p SSE σ 2 ⏟ χ n − p 2 > ⊥ = d χ p − 1 2 / p − 1 χ n − p 2 / n − p > ⊥ = d F p − 1 , n − p \implies F^*=\frac{\text{MSR}}{\text{MSE}}=\frac{\text{SSR}/p-1}{\text{SSER}/n-p}=\frac{\frac{1}{p-1}\overbrace{\frac{\text{SSR}}{\sigma^2}}^{\chi^2_{p-1}}}{\frac{1}{n-p}\underbrace{\frac{\text{SSE}}{\sigma^2}}_{\chi^2_{n-p}}}\Bigg>\perp \overset{\text{d}}{=}\frac{\chi^2_{p-1}/p-1}{\chi^2_{n-p}/n-p}\Big>\perp\overset{\text{d}}{=}F_{p-1,n-p} ⟹ F ∗ = MSE MSR = SSER / n − p SSR / p − 1 = n − p 1 χ n − p 2 σ 2 SSE p − 1 1 σ 2 SSR χ p − 1 2 ⟩ ⊥ = d χ n − p 2 / n − p χ p − 1 2 / p − 1 ⟩ ⊥ = d F p − 1 , n − p If D D D if there is a linear relationship between X ~ \utilde{X} X Y ~ \utilde{Y} Y , i.e.
H 0 : β 1 = β 2 = ⋯ = β k = 0 v.s. H 1 : β j ≠ 0 for some j = 1 , 2 , ⋯ , k H_0:\beta_1=\beta_2=\cdots=\beta_k=0\quad\text{ v.s. }\quad H_1:\beta_j\ne 0\text{ for some }j=1,2,\cdots,k H 0 : β 1 = β 2 = ⋯ = β k = 0 v.s. H 1 : β j = 0 for some j = 1 , 2 , ⋯ , k We reject H 0 H_0 H 0 α \alpha α F ∗ > F p − 1 , n − p , α F^*>F_{p-1,n-p,\alpha} F ∗ > F p − 1 , n − p , α
而我們得到數據後，計算出 f ∗ = f^*= f ∗ = = P H 0 ( F p − 1 , n − p > f ∗ ) =P_{H_0}(F_{p-1,n-p}>f^*) = P H 0 ( F p − 1 , n − p > f ∗ ) α \alpha α
ANOVA Table
Source SS df MS F P-value Regression SSR p-1 MSR MSR/MSE=f ∗ f^* f ∗ P ( F p − 1 , n − p > f ∗ ) P(F_{p-1,n-p}>f^*) P ( F p − 1 , n − p > f ∗ ) Error SSE n-p MSE Total SSTO n-1
其中的假設檢定為：Y ~ \utilde{Y} Y X ~ \utilde{X} X
H 0 : β 1 = β 2 = ⋯ = β k = 0 v.s. H 1 : β j ≠ 0 for some j = 1 , 2 , ⋯ , k H_0:\beta_1=\beta_2=\cdots=\beta_k=0\quad\text{ v.s. }\quad H_1:\beta_j\ne 0\text{ for some }j=1,2,\cdots,k H 0 : β 1 = β 2 = ⋯ = β k = 0 v.s. H 1 : β j = 0 for some j = 1 , 2 , ⋯ , k Remark :在 R 語言中，如果 fm 是我們的迴歸模型，我們可以使用 anova(fm) 來得到 ANOVA 表。
Source SS df perdict1 SSR1 1 perdict2 SSR2 1 ⋮ \vdots ⋮ ⋮ \vdots ⋮ ⋮ \vdots ⋮ perdict(p-1) SSR(p-1) 1 Error SSE n-p
也就是說，R 會先對一個解釋變數做投影，扣除這個變數的影響後，再對下一個變數做投影，以此類推得到所有的 SSR。但因為解釋變數之間不一定是垂直的，所以投影的順序可能會影響 SSR 的大小。
因此在構建模型時，會把我們認為比較重要的變數放在前面，這樣可以更好的解釋模型。比如 lm(Y~X1+X2+X3)，我們認為 X1 對 Y 的影響最大，X2 次之，X3 最小，所以我們把 X1 放在前面。
