跳至主要内容

最小方差无偏估计(UMVUE)

Recall

If δ\delta is unbiased est's, MSE(δ,η(θ))=Var(δ(X~))MSE(\delta, \eta(\theta))=Var(\delta(\utilde{X}))

如果我們限定在無偏的估計方法中找最優的,那麼就是找方差最小的估計方法。也就是 Uniformly Minimum Variance Unbiased Estimator ,最小方差無偏估計。

危險

無偏並不一定是最好的。

Definition

δ(X~)\delta^*(\utilde{X}) is called UMVUE for η(θ)\eta(\theta) iff

  1. Eθδ(X~)=η(θ),θE_\theta\delta^*(\utilde{X})=\eta(\theta), \forall\theta i.e. unbiased.

  2. \forall unbiased est. δ(X~)\delta(\utilde{X}) for η(θ)\eta(\theta),

    Varθ(δ(X~))Varθ(δ(X~))Var_\theta(\delta^*(\utilde{X}))\le Var_\theta(\delta(\utilde{X})) i.e. 方差最小。

Q: 充分統計量在估計上有什麼用?

Recall
  1. 隨機變量 T,WT, W

    E[E(TW)]=E[T]Var(T)=E[Var(TW)]+Var[E(TW)]Var[E(TW)] \begin{align*} E[E(T|W)] &= E[T]\\ Var(T) &= E[Var(T|W)]+Var[E(T|W)]\\ &\ge Var[E(T|W)] \end{align*}
  2. If T=T(X~)T=T(\utilde{X}) is suff for θ\theta     fX~T(x~t)θ\iff f_{\utilde{X}|T}(\utilde{x}|t) \perp \theta

        δ(X~),E[δ(X~)T]=δ(x~)fX~T(x~t)dx~θ\implies \forall \delta(\utilde{X}), E[\delta(\utilde{X})|T] = \int \delta(\utilde{x})*f_{\utilde{X}|T}(\utilde{x}|t)d\utilde{x} \perp\theta

    since both δ(x~)\delta(\utilde{x}) and fX~T(x~t)f_{\utilde{X}|T}(\utilde{x}|t) are independent of θ\theta.

Theorem

Rao-Blackwellized

Suppse X~iidf(x~;θ),θΩ\utilde{X}\stackrel{\text{iid}}{\sim} f(\utilde{x};\theta), \theta\in\Omega

S(X~)S(\utilde{X}) be any est. for η(θ)\eta(\theta) and T=T(X~)T=T(\utilde{X}) is suff for θ\theta, with EθS(X~)E_\theta|S(\utilde{X})|\le\infty

Let δ(T)=E[S(X~)T]\delta(T)=E[S(\utilde{X})|T], then

  1. δ(T)\delta(T) is a statistic with Eθδ(T)=Eθ(S(X~)),θE_\theta\delta(T)=E_\theta(S(\utilde{X})), \forall\theta

  2. For any function η(θ)\eta(\theta), MSE(δ(T),η(θ))MSE(S(X~),η(θ)),θMSE(\delta(T), \eta(\theta))\le MSE(S(\utilde{X}), \eta(\theta)), \forall\theta

  3. If Var(S(X~)),θVar(S(\utilde{X}))\le\infty, \forall\theta, then

    MSE(δ(T),η(θ))<MSE(S(X~),η(θ)),θMSE(\delta(T), \eta(\theta)) < MSE(S(\utilde{X}), \eta(\theta)), \forall\theta

EX: X1,,XniidB(1,p),pΩ=(0,1)X_1,\cdots,X_n\stackrel{\text{iid}}{\sim} B(1, p), p\in\Omega=(0,1), let η(p)=p\eta(p)=p

    T=i=1nXi\implies T=\sum_{i=1}^nX_i is suff for pp.

隨便找一個 η(p)\eta(p) 的無偏估計 S(X~)=X1S(\utilde{X})=X_1, then Ep(S(X~))=Ep(X1)=pE_p(S(\utilde{X}))=E_p(X_1)=p and Varp(S(X~))=p(1p)Var_p(S(\utilde{X}))=p(1-p)

Let δ(T)=E[S(X~)T]=E[X1T]=Tn=Xˉ\delta(T)=E[S(\utilde{X})|T]=E[X_1|T]=\frac{T}{n}=\bar{X}, and Var(Xˉ)=p(1p)n<p(1p)Var(\bar{X})=\frac{p(1-p)}{n}<p(1-p) is better.

備註

nE(X1T)=E(XiT)=E(XiT)=E(TT)=1nE(X_1|T)=\sum E(X_i|T)=E(\sum X_i|T)=E(T|T)=1

我們可以利用充分統計量,通過 Rao-Blackwellized 來改善估計方法,使得偏差不變的情況下,方差更小。

但如果我們有兩個充分統計量 T1,T2T_1, T_2 ,以及任意一個對於 θ\theta 的估計量 S(X~)S(\utilde{X})

BlackwellRaoδ1(T1)=E[S(X~)T1]δ2(T2)=E[S(X~)T2]\begin{align*} \xRightarrow[Blackwell]{Rao} \delta_1(T_1)&=E[S(\utilde{X})|T_1]\\ \delta_2(T_2)&=E[S(\utilde{X})|T_2] \end{align*}

δ1(T1)\delta_1(T_1)δ2(T2)\delta_2(T_2) 都比 S(X~)S(\utilde{X}) 好,但是我們無法比較 δ1(T1)\delta_1(T_1)δ2(T2)\delta_2(T_2) 誰更好。

為了找到最好的充分統計量,我們需要引入完備(complete)的概念。

Definition

X~iidf(x~;θ),θΩRr\utilde{X}\stackrel{\text{iid}}{\sim} f(\utilde{x};\theta), \theta\in\Omega\subseteq\R^r

F={f(;θ);θΩ}\mathscr{F}=\set{f(;\theta);\theta\in\Omega} is complete     \iff

h\forall h s.t. Eθ[h(X~)]=0    Pθ(h(X~)=0)=1,θE_\theta[h(\utilde{X})]=0\implies P_\theta(h(\utilde{X})=0)=1, \forall \theta i.e. h(X~)=0h(\utilde{X})=0 almost surely.

Now, X~iidf(x~;θ),θΩ\utilde{X}\stackrel{\text{iid}}{\sim} f(\utilde{x};\theta), \theta\in\Omega, let T=T(X~)T=T(\utilde{X}) with pdf fT(t;θ)f_T(t;\theta) and FT={fT(;θ);θΩ}\mathscr{F}_T=\set{f_T(;\theta);\theta\in\Omega}

Definition

TT is complete

    FT\iff \mathscr{F}_T is complete

    h\iff \forall h with Eθh(T)=0    h(T)=0E_\theta h(T)=0 \implies h(T)=0 almost surely θ\forall\theta

Theorem

Lehmann-Scheffe

Let X~iidf(x~;θ),θΩRr\utilde{X}\stackrel{\text{iid}}{\sim} f(\utilde{x};\theta), \theta\in\Omega\subseteq\R^r

T=T(X~)T=T(\utilde{X}) is suff for θ\theta and is complete

EX: XiidB(m,p),p(0,1),m=1,2,X\stackrel{\text{iid}}{\sim} B(m,p), p\in(0,1), m=1,2,\cdots

Let hh s.t. Eph(X)=0,p(0,1)E_ph(X)=0, \forall p\in(0,1)

    x=0mh(x)fX(x)=x=0mh(x)(mx)px(1p)mx=0\iff \sum_{x=0}^m h(x)f_X(x)=\sum_{x=0}^m h(x)\binom{m}{x}p^x(1-p)^{m-x}=0, p(0,1)\forall p\in(0,1)

Since p(0,1)    px(1p)mx0p\in(0,1)\implies p^x(1-p)^{m-x}\neq 0

    h(x)(mx)\iff h(x)\binom{m}{x}, x=0,1,,m\forall x=0,1,\cdots,m

    h(x)=0\iff h(x)=0, x=0,1,,m\forall x=0,1,\cdots,m

i.e. Pp(h(X)=0)=1P_p(h(X)=0)=1

    B(m,p)\implies B(m,p) is complete.


EX: X1,,XniidB(1,p),p(0,1)X_1,\cdots, X_n \stackrel{\text{iid}}{\sim} B(1,p), p\in(0,1)

  1. T0=T0(X~)=X~T_0=T_0(\utilde{X})=\utilde{X} is suff for pp, but not complete.

    let h(T0)=X1X2h(T_0)=X_1-X_2, then Ep(h(T0))=0,p(0,1)E_p(h(T_0))=0, \forall p\in(0,1) but h(T0)0h(T_0)\neq 0 almost surely.

  2. T_1=X_1 is not suff for pp, but T1=X1iidB(1,p)T_1=X_1\stackrel{\text{iid}}{\sim} B(1,p) is complete but the previous EX.

  3. T=T(X~)=i=1nXiiidB(n,p)T=T(\utilde{X})=\sum_{i=1}^n X_i \stackrel{\text{iid}}{\sim} B(n,p) is suff for pp and is complete.


Let T=T(X~)T=T(\utilde{X}) be suff for θ\theta and u(X~)u(\utilde{X}) be any unbiased est.

By Rao-Blackwellized, δ(T)=E[u0(X~)T]\delta^*(T)=E[u_0(\utilde{X})|T] dominates u(X~)u(\utilde{X}).

For any u(X~)u(\utilde{X}) unbiased est, and δ(T)=E[u(X~)T]\delta(T)=E[u(\utilde{X})|T] is also dominates u(X~)u(\utilde{X}).

Let h(T)=δ(T)δ(T)h(T)=\delta^*(T)-\delta(T)

Eθ(δ(T)δ(T))=Eθδ(T)Eθδ(T)=Eθu0(X~)Eθu(X~)=η(θ)η(θ)=0\begin{align*} E_\theta(\delta^*(T)-\delta(T)) &= E_\theta\delta^*(T)-E_\theta\delta(T)\\ &=E_\theta u_0(\utilde{X}) - E_\theta u(\utilde{X})\\ &=\eta(\theta)-\eta(\theta)\\ &=0 \end{align*}

Since TT is complete     h(T)=0\implies h(T)=0 almost surely θ\forall\theta     δ(T)=δ(T)\implies \delta(T)=\delta^*(T)

i.e. For any unbiased est. u(X~)u(\utilde{X}) is dmoniated by δ(T)\delta^*(T),     δ(T)\implies\delta^*(T) is the UMVUE for η(θ)\eta(\theta).

任何無偏估計都可以用充分完備統計量來改善。改善後的結果是唯一的,並且改善得到的估計量一定比原來的更好(或一樣好)。所以我們只需要找到一個充分完備統計量,就可以得到 UMVUE。

Theorem

Lehmann-Scheffe

Let X~iidf(x~;θ),θΩRr\utilde{X}\stackrel{\text{iid}}{\sim} f(\utilde{x};\theta), \theta\in\Omega\subset \R^r

T=T(X~)T=T(\utilde{X}) is suff for θ\theta and is complete

Let u(X~)u(\utilde{X}) be any unbiased est. for η(θ)\eta(\theta) with finite variance

and δ(T)=E[u(X~)T]\delta(T)=E[u(\utilde{X})|T], then δ(T)\delta(T) is the unique UMVUE for η(θ)\eta(\theta)

因此,如果我們有一個對於 θ\theta 充分且完備的統計量,那麼我們可以有兩種方法計算 η(θ)\eta(\theta) 的UMVUE:

  1. 隨便找一個無偏估計,然後通過 Rao-Blackwellized 計算 δ(T)=E[δ(X~)T]\delta(T) = E[\delta(\utilde{X})|T],則 δ(T)\delta(T) 是 UMVUE。但計算過程可能很複雜。

  2. 根據 η(θ)\eta(\theta) 猜一個 TT 的函數 g(T)g(T),計算 E[g(T)]E[g(T)],然後根據結果調整 g(T)g(T),直到 E[g(T)]=η(θ)E[g(T)]=\eta(\theta),則 g(T)g(T) 是 UMVUE。

EX: X1,,XniidB(1,p),p(0,1)X_1, \cdots, X_n\stackrel{\text{iid}}{\sim} B(1,p),p\in(0,1)

    T=i=1nXi\implies T=\sum_{i=1}^nX_i is suff for pp and is complete.

p,E(T)=np    E(Tn)=p\forall p ,E(T)=np\implies E(\frac{T}{n})=p, so Tn\frac{T}{n} is the UMVUE for pp.


EX: X1,,XniidU(0,θ),θ>0X_1,\cdots,X_n\stackrel{\text{iid}}{\sim} U(0,\theta), \theta>0

know: T=X(n)T=X_{(n)} is suff for θ\theta

Q: Is X(n)X_{(n)} complete?

h\forall h s.t. Eθh(X(n))=0,θ>0E_\theta h(X_{(n)})=0, \forall\theta>0

0=h(t)fT(t)dt=h(t)n(F(t))n1f(t)dt=n0h(t)(tθ)n1dt=nθn0h(t)tn1dt    0h(t)tn1dt=0,θ>0\begin{align*} 0&=\int_{-\infty}^\infty h(t)f_T(t) dt\\ &=\int_{-\infty}^\infty h(t)n(F(t))^{n-1}f(t) dt\\ &= n \int_{0}^\infty h(t)(\frac{t}{\theta})^{n-1} dt\\ &= \frac{n}{\theta^n}\int_{0}^\infty h(t)t^{n-1} dt\\ \iff &\int_{0}^\infty h(t)t^{n-1} dt=0, \forall\theta>0 \end{align*}

假設 h(t)h(t) 是連續的,則 h(t)tn1h(t)t^{n-1} 也是連續的,那麼 h(t)tn1=0,t(0,)h(t)t^{n-1}=0, \forall t\in(0,\infty)

    h(t)=0,t(0,)\implies h(t)=0, \forall t\in(0,\infty)

    X(n)\implies X_{(n)} is complete.

Hence, X(n)X_{(n)} is suff for θ\theta and is complete.

  1. η(θ)=θ=Eθ(X12),θ>0\eta(\theta)=\theta=E_\theta(\frac{X_1}{2}),\forall \theta >0

    1. X12\frac{X_1}{2}:unbaised for η\eta     \implies The UMVUE for θ\theta is E(X12T)E(\frac{X_1}{2}|T), 但其中的計算過程很複雜。

    2. know, fT(t)=ntn1θn,0<t<θf_T(t)=n\frac{t^{n-1}}{\theta^n}, 0<t<\theta

      Eθ[X(n)]=0θtntn1θndt=nθn0θtndt=nθntn+1n+10θ=nθnθn+1n+1=nn+1θ\begin{align*} E_\theta[X_{(n)}] &= \int_0^\theta t\cdot n\frac{t^{n-1}}{\theta^n} dt\\ &=\frac{n}{\theta^n}\int_0^\theta t^n dt\\ &=\frac{n}{\theta^n}\frac{t^{n+1}}{n+1}\bigg|_0^\theta\\ &=\frac{n}{\theta^n}\frac{\theta^{n+1}}{n+1}\\ &=\frac{n}{n+1}\theta \end{align*}

          E(X(n)n+1)=θ\implies E(\frac{X_{(n)}}{n+1})=\theta, i.e. X(n)n+1\frac{X_{(n)}}{n+1} is the UMVUE for θ\theta.

      直觀來說,Xi,i=1,,nX_i,i=1,\dots,n(0,θ)(0,\theta) 分成了 n+1n+1 個區間。因為是均勻分佈,所有每個區間的平均長度都會是 θn+1\frac{\theta}{n+1},而 X(n)X_{(n)} 就是從左到右第 nn 個分隔點,所以 E(X(n))=nn+1θE(X_{(n)})=\frac{n}{n+1}\theta

  2. η(θ)=θ2\eta(\theta)=\theta^2

    Try Eθ(X(n)2)=nθn0θtn+1dt=nn+2θ2E_\theta(X_{(n)}^2)=\frac{n}{\theta^n}\int_0^\theta t^{n+1} dt=\frac{n}{n+2}\theta^2

        E(n+2nX(n)2)=θ2\implies E(\frac{n+2}{n}X_{(n)}^2)=\theta^2

Definition

Exp Family Let X~iidf(x~;θ),θΩRr\utilde{X}\stackrel{\text{iid}}{\sim} f(\utilde{x};\theta), \theta\in\Omega\subseteq\R^r Suppose that x~,θ\forall \utilde{x},\theta

f(x~;θ)=c(θ)exp{i=1kQi(θ)Ti(x~)}h(x~)f(\utilde{x};\theta)=c(\theta)\exp\set{\sum_{i=1}^k Q_i(\theta)T_i(\utilde{x})}h(\utilde{x})

and

X={x~:f(x~;θ)>0}θ,c(θ)>0,h(x~)>0X=\set{\utilde{x}:f(\utilde{x};\theta)>0}\perp\theta, c(\theta)>0, h(\utilde{x})>0

then f(x~;θ)f(\utilde{x};\theta\in) k-parameter exp family.

指數部分需要有 K 個部分組成,那麼 f(x~;θ)f(\utilde{x};\theta) 就是一個 K-parameter exp family。

Theorem

Let X~iidf(x~;θ),θΩRr\utilde{X}\stackrel{\text{iid}}{\sim} f(\utilde{x};\theta), \theta\in\Omega\subseteq\R^r and f(x~;θ)f(\utilde{x};\theta)\in k-parameter exp family with

B{(Q1(θ),,Qk(θ));θΩ}RkB\triangleq\set{(Q_1(\theta),\cdots,Q_k(\theta));\theta\in\Omega}\in\R^k

Suppose B˚\mathring{B}\neq\empty (內點集非空), i.e. dim(B)=k\dim(B)=k

then T(X~)=(T1(X~),,Tk(X~))T(\utilde{X})=(T_1(\utilde{X}),\cdots,T_k(\utilde{X})) is suff for θ\theta and is complete.

EX: X1,,XniidB(1,p),p(0,1)X_1,\cdots,X_n\overset{\text{iid}}{\sim}B(1,p), p\in(0,1)

f(x~;p)=pxi(1p)nxi,xi=0,1=(1p)n(p1p)xi=(1p)nexp[xiln(p1p)]=c(p)exp(Q(p)T(x~))h(x~)\begin{align*} f(\utilde{x};p)&=p^{\sum x_i}(1-p)^{n-\sum x_i},\quad x_i=0,1\\ &=(1-p)^n\cdot\left(\frac{p}{1-p}\right)^{\sum x_i}\\ &=(1-p)^n\exp\left[\sum x_i\ln\left(\frac{p}{1-p}\right)\right]\\ &=c(p)\exp\left(Q(p)T(\utilde{x})\right)h(\utilde{x}) \end{align*}

and X={x~:f(x~;p)>0}={x:xi=0 or 1,i}pX=\set{\utilde{x}:f(\utilde{x};p)>0}=\set{x:x_i=0 \text{ or }1, \forall i}\perp p

    f(x~;p)\implies f(\utilde{x};p)\in 1-par exp family with

B={Q(p):pΩ}R1={ln(p1p):p(0,1)}=(,)=R1i.e. B˚\begin{align*} B&=\set{Q(p):p\in\Omega}\subseteq R^1\\ &=\set{\ln\left(\frac{p}{1-p}\right):p\in(0,1)}\\ &=(-\infty,\infty)=\R^1\\ \text{i.e. }& \mathring{B}\neq\empty \end{align*}

    T(X~)=Xi\implies T(\utilde{X})=\sum X_i is suff for pp and is complete.

e.g. η(P)=P(1P)=P(X1=1,X2=0)=Ep(I(X1=1,X2=0))\eta(P)=P(1-P)=P(X_1=1,X_2=0)=E_p(I(X_1=1,X_2=0))

    \implies The UMVUE for P(1P)P(1-P) is

E(I(X1=1,X2=0)T)=P(X1=1,X2=0T)=P(X1=1,X=0.T=t)P(T=t)=P(X1=1)P(X2=0)P(i=3nXi=t1)P(T=t)=P(1P)(n2t1)Pt1(1P)n1t(nt)Pt(1P)nt=(n2)!(t1)!(n1t)!n!t!(nt)!=t(nt)n(n1)\begin{align*} E(I(X_1=1,X_2=0)|T) &= P(X_1=1,X_2=0|T)\\ &=\frac{P(X_1=1,X_=0.T=t)}{P(T=t)}\\ &=\frac{P(X_1=1)P(X_2=0)P(\sum_{i=3}^nX_i=t-1)}{P(T=t)}\\ &=\frac{P(1-P)\binom{n-2}{t-1}P^{t-1}(1-P)^{n-1-t}}{\binom{n}{t}P^t(1-P)^{n-t}}\\ &=\frac{\frac{(n-2)!}{(t-1)!(n-1-t)!}}{\frac{n!}{t!(n-t)!}}\\ &=\frac{t(n-t)}{n(n-1)}\\ \end{align*}

EX: X1,,XniidP(λ)X_1,\cdots,X_n\overset{\text{iid}}{\sim}P(\lambda)

    f(x~;λ)=eλλxixi!=enλλxi1xi!=c(λ)exp(xilnλ)h(x~)\implies f(\utilde{x};\lambda)=\prod\frac{e^{-\lambda}\lambda^{x_i}}{x_i!}=e^{-n\lambda}\lambda^{\sum x_i}\frac{1}{\prod x_i!}=c(\lambda)\exp\left(\sum x_i \ln\lambda\right)h(\utilde{x})\in 1-par exp family and X={x~:xN}λX=\set{\utilde{x}:x\in \N}\perp \lambda

with B={lnλ:λ>0}=(,)    B˚B=\set{\ln\lambda:\lambda>0}=(-\infty,\infty)\implies \mathring{B}\neq\empty

    T=Xi\implies T=\sum X_i is suff for λ\lambda and is complete.

  1. η(λ)=λ\eta(\lambda)=\lambda

    Note TP(nλ)T\sim P(n\lambda) with E(T)=nλ    E(Tn)=λE(T)=n\lambda\implies E(\frac{T}{n})=\lambda i.e. Tn\frac{T}{n} is the UMVUE for λ\lambda.

  2. η(λ)=e2λλ3=3!P(X1=3,X2=0)=E(6I(X1=3,X2=0))\eta(\lambda)=e^{-2\lambda}\lambda^3=3!P(X_1=3,X_2=0)=E\left(6I(X_1=3,X_2=0) \right)

        \implies The UMVUE for λ3e2λ\lambda^3e^{-2\lambda} is E(6I(X1=3,X2=0)T)=P(X1=3,X2=0T)E(6I(X_1=3,X_2=0)|T)=P(X_1=3,X_2=0|T)

Theorem

Basu's Theorem Let T=T(X~)T=T(\utilde{X}) be suff for θ\theta and is complete

Let V=V(X~)V=V(\utilde{X}) be a statistic whose distribution does not depend on θ\theta

Then TT and VV are independent.

EX: X1,,XniidN(μ,σ2)X_1,\cdots,X_n\stackrel{\text{iid}}{\sim} N(\mu,\sigma^2) with σ2=σ02\sigma^2=\sigma^2_0 known

    \implies Xˉ\bar{X} is suff for μ\mu and is complete.

note that

(n1)S2σ02iidχn12μ\frac{(n-1)S^2}{\sigma_0^2}\stackrel{\text{iid}}{\sim}\chi^2_{n-1}\perp\mu

by Basu's Theorem, Xˉ\bar{X} and S2S^2 are independent.


EX: X1,,XniidN(μ,σ2)X_1,\cdots,X_n\overset{\text{iid}}{\sim}N(\mu,\sigma^2)

f(x~;μ,σ2)=1(2πσ2)n/2exp(12σ2(xiμ)2)=(12πσ2)nexp(12σ2xi2+μσ2xinμ22σ2)\begin{align*} f(\utilde{x};\mu,\sigma^2)&=\frac{1}{(2\pi\sigma^2)^{n/2}}\exp\left(-\frac{1}{2\sigma^2}\sum(x_i-\mu)^2\right)\\ &=\left(\frac{1}{\sqrt{2\pi\sigma^2}}\right)^n\exp\left(-\frac{1}{2\sigma^2}\sum x_i^2 +\frac{\mu}{\sigma^2}\sum x_i-\frac{n\mu^2}{2\sigma^2}\right)\\ \end{align*}
  1. σ2=σ02\sigma^2=\sigma_0^2 known

    f(x~;μ)=(12πσ02)nenμ22σ2exp(μσ2xi)e12σ2xi2f(\utilde{x};\mu)=\left(\frac{1}{\sqrt{2\pi\sigma_0^2}}\right)^ne^{-\frac{n\mu^2}{2\sigma^2}}\exp\left(\frac{\mu}{\sigma^2}\sum x_i\right)e^{-\frac{1}{2\sigma^2}\sum x_i^2}

    with X=Rnμ    f(x~;μ)X=\R^n\perp\mu\implies f(\utilde{x};\mu)\in 1-par exp family and B={Q(μ):μΩ}={muσ02:μR}=RB=\set{Q(\mu):\mu\in\Omega}=\set{\frac{mu}{\sigma_0^2}:\mu\in\R}=\R

        B˚\implies \mathring{B}\neq\empty

        T=Xi\implies T=\sum X_i is suff for μ\mu and is complete 11Xˉ\xRightarrow{1-1} \bar{X} is suff for μ\mu and is complete.

    1. η(μ)=μ,XˉN(μ,σ02n)\eta(\mu)=\mu, \bar{X}\sim N(\mu,\frac{\sigma_0^2}{n}) with E(Xˉ)=μE(\bar{X})=\mu i.e. Xˉ\bar{X} is the UMVUE for μ\mu.

    2. η(μ)=μ2,XˉN(μ,σ02n)\eta(\mu)=\mu^2, \bar{X}\sim N(\mu,\frac{\sigma_0^2}{n}) with Var(Xˉ)=σ02n    E(Xˉ2)=μ2+σ02nVar(\bar{X})=\frac{\sigma_0^2}{n}\implies E(\bar{X}^2)=\mu^2+\frac{\sigma_0^2}{n}

          Xˉ2σ02n\implies \bar{X}^2-\frac{\sigma_0^2}{n} is the UMVUE for μ2\mu^2.

  2. μ=μ0\mu=\mu_0: known, θ=σ2Ω=(0,)\theta=\sigma^2\in\Omega=(0,\infty)

    f(x~;σ2)=(12πσ2)nexp(12σ2(xiμ0)2)1-par exp familyf(\utilde{x};\sigma^2)=\left(\frac{1}{\sqrt{2\pi\sigma^2}}\right)^n\exp\left(-\frac{1}{2\sigma^2}\sum(x_i-\mu_0)^2\right) \in \text{1-par exp family}

    with B={12σ2:σ2>0}=(,0)R1    B˚B=\set{-\frac{1}{2\sigma^2}:\sigma^2>0}=(-\infty,0)\subset R^1\implies \mathring{B}\neq\empty

        T=T(X~)=(Xiμ0)2\implies T=T(\utilde{X})=\sum(X_i-\mu_0)^2 is suff for σ2\sigma^2 and is complete.

    Note: (Xiμ0σ)2χn2\sum\left(\frac{X_i-\mu_0}{\sigma}\right)^2\sim\chi^2_n with E(χn2)=nE(\chi^2_n)=n, i.e. Tσ2χn2\frac{T}{\sigma^2}\sim\chi^2_n

        E(T)=σ2E(Tσ2)=σ2n    E(Tn)=σ2\implies E(T)=\sigma^2E(\frac{T}{\sigma^2})=\sigma^2n\implies E(\frac{T}{n})=\sigma^2

    i.e. 1n(Xiμ0)2\frac{1}{n}\sum(X_i-\mu_0)^2 is the UMVUE for σ2\sigma^2.

    for η(σ2)=σ=σ2\eta(\sigma^2)=\sigma=\sqrt{\sigma^2}, try E(T)=σE(Tσ2)=σE(Y)=σCnE(\sqrt{T})=\sigma E\left(\sqrt{\frac{T}{\sigma^2}}\right)=\sigma E(\sqrt{Y})=\sigma C_n with Yχn2Y\sim\chi^2_n

    E(Y)=0y1Γ(n2)2n/2yn/21ey/2dy=Γ(n+12)2(n+1)/2Γ(n2)2n/201Γ(n+12)2(n+1)/2y(n+1)/21ey/2dy=1=2Γ(n+12)Γ(n2)Cn\begin{align*} E(\sqrt{Y})&=\int_0^\infty \sqrt{y}\frac{1}{\Gamma(\frac{n}{2})2^{n/2}}y^{n/2-1}e^{-y/2}dy\\ &=\frac{\Gamma(\frac{n+1}{2})2^{(n+1)/2}}{\Gamma(\frac{n}{2})2^{n/2}}\underbrace{\int_0^\infty \frac{1}{\Gamma(\frac{n+1}{2})2^{(n+1)/2}}y^{(n+1)/2-1}e^{-y/2}dy}_{=1}\\ &=\sqrt2\frac{\Gamma(\frac{n+1}{2})}{\Gamma(\frac{n}{2})}\triangleq C_n \end{align*}

        TΓ(n2)2Γ(n+12)\implies\frac{\sqrt{T}\Gamma(\frac{n}{2})}{\sqrt{2}\Gamma(\frac{n+1}{2})} is the UMVUE for σ\sigma.

  3. θ=(μ,σ2)Ω=R×(0,)\theta=(\mu,\sigma^2)\in\Omega=\R\times(0,\infty)

    f(x~;μ,σ2)=(12πσ2)nenμ2σ2exp(12σ2Xi2+μσ2Xi)2-par exp familyf(\utilde{x};\mu,\sigma^2)=\left(\frac{1}{\sqrt{2\pi\sigma^2}}\right)^ne^{-\frac{n\mu^2}{\sigma^2}} \exp\left(-\frac{1}{2\sigma^2}\sum X_i^2+\frac{\mu}{\sigma^2}\sum X_i\right)\in\text{2-par exp family}

    with B={μσ2,12σ2}=R×(,0)    B˚B=\set{\frac{\mu}{\sigma^2},-\frac{1}{2\sigma^2}}=\R\times(-\infty,0)\implies \mathring{B}\neq\empty

        T=(Xi,Xi2)\implies T^*=(\sum X_i,\sum X_i^2) is suff for (μ,σ2)(\mu,\sigma^2) and is complete.

    11T=(Xˉ,S2)\xRightarrow{1-1}T=(\bar{X},S^2) is suff for (μ,σ2)(\mu,\sigma^2) and is complete.

    1. η(θ)=μ\eta(\theta)=\mu, δ(T)=δ(S2,Xˉ)=Xˉ\delta(T)=\delta(S^2,\bar{X})=\bar{X}, Eδ(T)=EXˉ=μE\delta(T)=E\bar{X}=\mu

          Xˉ\implies \bar{X} is the UMVUE for μ\mu (same as when σ2\sigma^2 is known)

    2. η(θ)=μ2\eta(\theta)=\mu^2

      E(Xˉ2)=Var(Xˉ)+(EXˉ)2=σ2n+μ2E(\bar{X}^2)=Var(\bar{X})+(E\bar{X})^2=\frac{\sigma^2}{n}+\mu^2, E(S2)=σ2n1E((n1)S2σ2χn12)=σ2E(S^2)=\frac{\sigma^2}{n-1}E\left(\frac{(n-1)S^2}{\sigma^2}\sim\chi^2_{n-1}\right)=\sigma^2

          E(Xˉ2S2n)=μ2\implies E(\bar{X}^2-\frac{S^2}{n})=\mu^2

          Xˉ2S2n\implies \bar{X}^2-\frac{S^2}{n} is the UMVUE for μ2\mu^2 (same as when σ2\sigma^2 is known)

    3. η(θ)=σ2\eta(\theta)=\sigma^2

      (n1)S2σ2χn12    E(S2)=σ2\frac{(n-1)S^2}{\sigma^2}\sim\chi^2_{n-1}\implies E(S^2)=\sigma^2

          S2\implies S^2 is the UMVUE for σ2\sigma^2 (\neq when μ\mu is known)

    4. η(θ)=μσ\eta(\theta)=\frac{\mu}{\sigma}

      Try E(XˉS)=E(Xˉ)E(1S)=μE(1(n1)S2σ2χn12)n1σ=μσn1an\text{Try }E\left(\frac{\bar{X}}{S}\right)=E(\bar{X})E\left(\frac{1}{S}\right)=\mu E\left(\frac{1}{\sqrt{\frac{(n-1)S^2}{\sigma^2}\sim \chi^2_{n-1}}}\right)\frac{\sqrt{n-1}}{\sigma}=\frac{\mu}{\sigma}\sqrt{n-1}a_n

      for Yχm2Y\sim \chi^2_m

      E(1Y)=01y1Γ(m2)2m/2ym/21ey/2dy=Γ(m12)2(m1)/2Γ(m2)2m/201Γ(m12)2(m1)/2y(m1)/21ey/2dy=1=Γ(n22)Γ(n12)2an\begin{align*} E\left(\frac{1}{Y}\right)&=\int_0^\infty \frac{1}{y}\frac{1}{\Gamma(\frac{m}{2})2^{m/2}}y^{m/2-1}e^{-y/2}dy\\ &=\frac{\Gamma(\frac{m-1}{2})2^{(m-1)/2}}{\Gamma(\frac{m}{2})2^{m/2}}\underbrace{\int_0^\infty \frac{1}{\Gamma(\frac{m-1}{2})2^{(m-1)/2}}y^{(m-1)/2-1}e^{-y/2}dy}_{=1}\\ &=\frac{\Gamma(\frac{n-2}{2})}{\Gamma(\frac{n-1}{2})\sqrt2}\triangleq a_n \end{align*}

          XˉS2Γ(n12)n1Γ(n22)\implies \frac{\bar{X}}{S}\frac{\sqrt{2}\Gamma(\frac{n-1}{2})}{\sqrt{n-1}\Gamma(\frac{n-2}{2})} is the UMVUE for μσ\frac{\mu}{\sigma}


Xf(x;θ)X\sim f(x;\theta), T=T(X~)T=T(\utilde{X}) is suff for θ\theta and is complete

如果對給定 x0x_0 時的 pdf 或 cdf 感興趣,i.e. η(θ)=f(x0;θ)\eta(\theta)=f(x_0;\theta) or F(x0;θ)F(x_0;\theta)

Note: F(x0;θ)=P(Xx0)=E[I(Xx0)]F(x_0;\theta)=P(X\le x_0)=E[I(X\le x_0)]

    \implies The UMVUE for F(x0;θ)F(x_0;\theta) is E[I(Xx0)T]=P(Xx0T)=δ(T)E[I(X\le x_0)|T]=P(X\le x_0|T)=\delta(T)

    \implies The UMVUE for f(x0;θ)f(x_0;\theta) is x0δ(T)\frac{\partial}{\partial x_0}\delta(T)

Cramer-Rao Lower Bound

u(X~)\forall u(\utilde{X}) is unbiased for η(θ)\eta(\theta), 如果我們有一個下界 LB(θ)LB(\theta) 使得 Var(u(X~))LB(θ),θVar(u(\utilde{X}))\ge LB(\theta),\forall \theta,並且存在一個無偏估計 δ(X~)\delta^*(\utilde{X}) 使得 Var(δ(X~))=LB(θ)Var(\delta^*(\utilde{X}))=LB(\theta)

    δ(X~)\implies \delta^*(\utilde{X}) is the UMVUE for η(θ)\eta(\theta).

Recall that (cov(Y,W))2Var(Y)Var(W)    Var(W)(cov(Y,W))2Var(Y)(cov(Y,W))^2 \le Var(Y)Var(W) \iff Var(W)\ge\frac{(cov(Y,W))^2}{Var(Y)}

Take W=u(X~)W=u(\utilde{X}) is unbiased for η(θ)\eta(\theta) and Y=θlogf(X~;θ)Y=\frac{\partial}{\partial\theta}\log f(\utilde{X};\theta)

Note that

Eθ(Y)=Eθ(θlogf(X~;θ))=Eθ(1f(X~;θ)θf(X~;θ))=Rnθf(x~;θ)f(x~;θ)f(x~;θ)dx~=θRnf(x~;θ)dx~=θ1=0\begin{align*} E_\theta(Y)&=E_\theta(\frac{\partial}{\partial\theta}\log f(\utilde{X};\theta))\\ &=E_\theta(\frac{1}{f(\utilde{X};\theta)}\frac{\partial}{\partial\theta}f(\utilde{X};\theta))\\ &=\int_{R^n}\frac{\frac{\partial}{\partial\theta}f(\utilde{x};\theta)}{f(\utilde{x};\theta)}f(\utilde{x};\theta)d\utilde{x}\\ &=\frac{\partial}{\partial\theta}\int_{R^n}f(\utilde{x};\theta)d\utilde{x}\\ &=\frac{\partial}{\partial\theta}1\\ &=0 \end{align*}     Cov(u(X~,Y))=Eθ(u(X~)Y)Eθ(u(X~))Eθ(Y)=Eθ(u(X~)Y)=Rnu(x~)θf(x~;θ)f(x~;θ)f(x~;θ)dx~=θRnu(x~)f(x~;θ)dx~=θEθ(u(X~))=η(θ)since u(X~) is unbiased for η(θ)\begin{align*} \implies Cov(u(\utilde{X},Y)) &= E_\theta(u(\utilde{X})Y)-E_\theta(u(\utilde{X}))E_\theta(Y)\\ &=E_\theta(u(\utilde{X})Y)\\ &=\int_{R^n}u(\utilde{x})\frac{\frac{\partial}{\partial\theta}f(\utilde{x};\theta)}{f(\utilde{x};\theta)}f(\utilde{x};\theta)d\utilde{x}\\ &=\frac{\partial}{\partial\theta}\int_{R^n}u(\utilde{x})f(\utilde{x};\theta)d\utilde{x}\\ &=\frac{\partial}{\partial\theta}E_\theta(u(\utilde{X}))\\ &=\eta'(\theta) \quad \text{since $u(\utilde{X})$ is unbiased for $\eta(\theta)$} \end{align*} Var(Y)=E(Y2)(E(Y))2=Eθ[(θlogf(X~;θ))2]\begin{align*} Var(Y)&=E(Y^2)-(E(Y))^2\\ &=E_\theta[(\frac{\partial}{\partial\theta}\log f(\utilde{X};\theta))^2]\\ \end{align*}
危險

微分和積分並不總是可以交換順序。以上只能在某些情況下成立。

Definition

the Fisher-info

In(θ)=alsoIX~(θ)Eθ[(θlogf(X~;θ))2]I_n(\theta)\xlongequal{also}I_{\utilde{X}}(\theta)\triangleq E_\theta[(\frac{\partial}{\partial\theta}\log f(\utilde{X};\theta))^2]

Fisher Information 描述了隨機變量 XX 中包含有關 θ\theta 的信息量。如果包含的信息量越多,則 f(x;θ)f(x;\theta) 對於 θ\theta 的變化就越敏感,變化量越大。期望值則消除了隨機性。

Theorem

Cramer-Rao ineq

Let X~iidf(x~;θ),θΩR\utilde{X}\stackrel{\text{iid}}{\sim} f(\utilde{x};\theta), \theta\in\Omega\subseteq\R

Assume the following conditions hold:

  1. Ω\Omega is open in R\R

  2. X={x~:f(x~;θ)>0}θX=\set{\utilde{x}:f(\utilde{x};\theta)>0}\perp\theta

  3. θlogf(x~;θ)\frac{\partial}{\partial\theta}\log f(\utilde{x};\theta) exists x~,θ\forall \utilde{x},\theta

  4. δ(X~)\forall \delta(\utilde{X}) with Eθδ(X~)<,θE_\theta\delta(\utilde{X})<\infty,\forall\theta

    θRnδ(x~)f(x~;θ)dx~=Rnδ(x~)θf(x~;θ)dx~\frac{\partial}{\partial\theta}\int_{R^n}\delta(\utilde{x})f(\utilde{x};\theta)d\utilde{x}=\int_{R^n}\delta(\utilde{x})\frac{\partial}{\partial\theta}f(\utilde{x};\theta)d\utilde{x}

  5. In(θ),θ,n=1,2,I_n(\theta),\forall\theta,n=1,2,\cdots and θη(θ)\frac{\partial}{\partial\theta}\eta(\theta) exists

then u(X~)\forall u(\utilde{X}) is unbiased for η(θ)\eta(\theta)

Varθ(u(X~))(η(θ))2In(θ)Var_\theta(u(\utilde{X}))\ge\frac{(\eta'(\theta))^2}{I_n(\theta)}

Remark:

  1. 如果有一個無偏估計 δ(X~)\delta^*(\utilde{X}) 使得 Var(δ(X~))=(η(θ))2In(θ)Var(\delta^*(\utilde{X}))=\frac{(\eta'(\theta))^2}{I_n(\theta)},則 δ(X~)\delta^*(\utilde{X}) 是 UMVUE,並且我們說 CRLB is attainable。

  2. 如果所有無偏估計的方差都大於 CRLB,則 CRLB is not attainable。


EX: X1,,XniidB(1,P)X_1,\cdots,X_n\overset{\text{iid}}{\sim}B(1,P)

f(x~;P)=Pxi(1P)nxi=PT(1P)nTf(\utilde{x};P)=P^{\sum x_i}(1-P)^{n-\sum x_i}=P^T(1-P)^{n-T} with T=xiB(n,P)T=\sum x_i\sim B(n,P)

Plogf(x~;P)=P(TlogP+(nT)log(1P))=TPnT1P=TnPP(1P)\begin{align*} \frac{\partial}{\partial P}\log f(\utilde{x};P)&=\frac{\partial}{\partial P}\left(T\log P+(n-T)\log(1-P)\right)\\ &=\frac{T}{P}-\frac{n-T}{1-P}\\ &=\frac{T-nP}{P(1-P)} \end{align*} In(P)=E[(Plogf(X~;P))2]=E[(TnPP(1P))2]=1P2(1P)2Var(T)=n1P\begin{align*} I_n(P)&=E[(\frac{\partial}{\partial P}\log f(\utilde{X};P))^2]\\ &=E\left[\left(\frac{T-nP}{P(1-P)}\right)^2\right]\\ &=\frac{1}{P^2(1-P)^2}Var(T)\\ &=\frac{n}{1-P} \end{align*}

To est η(P)=P\eta(P)=P, CRLB =(η(P))2In(P)=P(1P)n=Var(Xˉ)=\frac{(\eta'(P))^2}{I_n(P)}=\frac{P(1-P)}{n}=Var(\bar{X})

    Xˉ\implies\bar{X} is the UMVUE for PP and CRLB is attainable.


Note:

  1. If Xi,i=1,,nX_i,i=1,\cdots,n are independent r.v.s

    f(X~;θ)=Πi=1nfXi(xi;θ)    E[(θlogf(X~;θ))2]=E[(i=1nθlogfXi(xi;θ))2]=i=1nE[(θlogfXi(xi;θ))2]+ijE[(θlogfXi(xi;θ))2(θlogfXj(xj;θ))]=i=1nE[(θlogfXi(xi;θ))2]=i=1nIXi(θ)\begin{align*} &f(\utilde{X};\theta)=\Pi_{i=1}^n f_{X_i}(x_i;\theta)\\ \implies& E[(\frac{\partial}{\partial\theta}\log f(\utilde{X};\theta))^2]= E[(\sum_{i=1}^n\frac{\partial}{\partial\theta}\log f_{X_i}(x_i;\theta))^2]\\ &=\sum_{i=1}^nE[(\frac{\partial}{\partial\theta}\log f_{X_i}(x_i;\theta))^2]+\sum_{i\neq j}E[(\frac{\partial}{\partial\theta}\log f_{X_i}(x_i;\theta))^2(\frac{\partial}{\partial\theta}\log f_{X_j}(x_j;\theta))]\\ &=\sum_{i=1}^nE[(\frac{\partial}{\partial\theta}\log f_{X_i}(x_i;\theta))^2]\\ &=\sum_{i=1}^nI_{X_i}(\theta) \end{align*}
  2. If X1,,XnX_1,\cdots,X_n 獨立服從同分佈

    IX~(θ)=In(θ)=i=1nIXi(θ)=nIX(θ)\begin{align*} I_{\utilde{X}}(\theta)&=I_n(\theta)\\ &=\sum_{i=1}^nI_{X_i}(\theta)\\ &=nI_X(\theta) \end{align*}

重新參數化(reparametrization)

θ=g(ξ)\theta=g(\xi) and gg' exists, i.e. f(x~;θ)=f(x~;g(ξ))f(\utilde{x};\theta)=f(\utilde{x};g(\xi))

IX~(ξ)E[(ξlogf(X~;g(ξ)))2]=E[(θξθlogf(X~;θ))2]=(g(ξ))2In(θ)\begin{align*} I_{\utilde{X}}(\xi)&\triangleq E[(\frac{\partial}{\partial\xi}\log f(\utilde{X};g(\xi)))^2]\\ &=E[(\frac{\partial\theta}{\partial\xi}\frac{\partial}{\partial\theta}\log f(\utilde{X};\theta))^2]\\ &=(g'(\xi))^2I_n(\theta) \end{align*}

i.e. θ=g(ξ)    In(ξ)=(g(ξ))2In(θ)\theta=g(\xi)\implies I_n(\xi)=(g'(\xi))^2I_n(\theta)

如果我們已經知道 In(θ)I_n(\theta) ,如果我們又關心 ξ\xi 的 Fisher-info,可以直接利用這個公式換算。


EX X1,,XniidN(μ,σ2)X_1,\cdots,X_n\overset{\text{iid}}{\sim}N(\mu,\sigma^2)

I(μ)=E[(μlogf(X~;μ,σ2))2]=E[(Xμσ2)2]=1σ4E[(Xμ)2]=1σ4Var(X)=1σ2\begin{align*} I(\mu)&=E[(\frac{\partial}{\partial\mu}\log f(\utilde{X};\mu,\sigma^2))^2]\\ &=E[(\frac{X-\mu}{\sigma^2})^2]=\frac{1}{\sigma^4}E[(X-\mu)^2]\\ &=\frac{1}{\sigma^4}Var(X)=\frac{1}{\sigma^2} \end{align*}

    In(μ)=nI(μ)=nσ2\implies I_n(\mu)=nI(\mu)=\frac{n}{\sigma^2}

CRLB for μ=(μη)2In(μ)=σ2n=Var(Xˉ)\text{CRLB for }\mu=\frac{(\frac{\partial}{\partial\mu}\eta)^2}{I_n(\mu)}=\frac{\sigma^2}{n}=Var(\bar{X})

    Xˉ\implies \bar{X}: UMVUE for μ\mu CRLB is attainable.

I(σ2)=I(θ)=E[(θlogf(X~;μ,θ))2]I(\sigma^2)=I(\theta)=E[(\frac{\partial}{\partial\theta}\log f(\utilde{X};\mu,\theta))^2] θlogf(X~;μ,θ)=θ(ln12π12lnθ(Xμ)22θ)=12θ+(Xμ)22θ2\begin{align*} \frac{\partial}{\partial\theta}\log f(\utilde{X};\mu,\theta)&=\frac{\partial}{\partial\theta}\left(\ln\frac{1}{\sqrt{2\pi}}-\frac{1}{2}\ln\theta-\frac{(X-\mu)^2}{2\theta}\right)\\ &=-\frac{1}{2\theta}+\frac{(X-\mu)^2}{2\theta^2} \end{align*}     I(θ)=E[(12θ+(Xμ)22θ2)2]=θ=σ2E[(Xμ)44σ812σ6E(Xμ)2+14σ4]=14σ4E[(Xμ)4σ4]12σ4E[(Xμ)2σ2]+14σ4=14σ4EY212σ4EY+14σ4where Y=(Xμσ)2χ12=34σ412σ4+14σ4=12σ4\begin{align*} \implies I(\theta)&=E\left[\left(-\frac{1}{2\theta}+\frac{(X-\mu)^2}{2\theta^2}\right)^2\right]\\ &\xlongequal{\theta=\sigma^2}E\left[\frac{(X-\mu)^4}{4\sigma^8}-\frac{1}{2\sigma^6}E(X-\mu)^2+\frac{1}{4\sigma^4}\right]\\ &=\frac{1}{4\sigma^4}E\left[\frac{(X-\mu)^4}{\sigma^4}\right]-\frac{1}{2\sigma^4}E\left[\frac{(X-\mu)^2}{\sigma^2}\right]+\frac{1}{4\sigma^4}\\ &=\frac{1}{4\sigma^4}EY^2-\frac{1}{2\sigma^4}EY+\frac{1}{4\sigma^4} \quad \text{where } Y=(\frac{X-\mu}{\sigma})^2\sim\chi^2_1\\ &=\frac{3}{4\sigma^4}-\frac{1}{2\sigma^4}+\frac{1}{4\sigma^4}\\ &=\frac{1}{2\sigma^4} \end{align*}

    IX~(σ2)=nI(σ2)=n2σ4\implies I_{\utilde{X}}(\sigma^2)=nI(\sigma^2)=\frac{n}{2\sigma^4}

CRLB for σ2=(σ2η)2In(σ2)=1n2σ4=2σ4n\text{CRLB for }\sigma^2=\frac{(\frac{\partial}{\partial\sigma^2}\eta)^2}{I_n(\sigma^2)}=\frac{1}{\frac{n}{2\sigma^4}}=\frac{2\sigma^4}{n}
  1. μ=μ0\mu=\mu_0 known, the UMVUE for σ2\sigma^2 is 1n(Xiμ0)2\frac{1}{n}\sum(X_i-\mu_0)^2

    Var(1n(Xiμ0)2)=σ4n2Var(Xiμ0σ2)2=2σ4n=CRLBVar(\frac{1}{n}\sum(X_i-\mu_0)^2)=\frac{\sigma^4}{n^2}\sum Var\left(\frac{X_i-\mu_0}{\sigma^2} \right)^2=\frac{2\sigma^4}{n}=\text{CRLB}
  2. μ\mu unknown, the UMVUE for σ2\sigma^2 is S2S^2

    Note that (n1)S2σχn12\frac{(n-1)S^2}{\sigma}\sim\chi^2_{n-1}

    Var(S2)=σ4(n1)2Var((n1)S2σ2)2=2σ4n1>2σ4n=CRLBVar(S^2)=\frac{\sigma^4}{(n-1)^2}Var\left(\frac{(n-1)S^2}{\sigma^2}\right)^2=\frac{2\sigma^4}{n-1}>\frac{2\sigma^4}{n}=\text{CRLB}

    CRLB is not attainable.

  3. To est I(σ)I(\sigma), let ξ=σ    σ2=ξ2=g(ξ)\xi=\sigma\implies\sigma^2=\xi^2=g(\xi)

        I(ξ)=(g(ξ))2I(σ2)=4ξ212σ4=2σ2\implies I(\xi)=(g'(\xi))^2I(\sigma^2)=4\xi^2\frac{1}{2\sigma^4}=\frac{2}{\sigma^2}

        In(σ)=2nσ2\implies I_n(\sigma)=\frac{2n}{\sigma^2}


EX:

  1. XiidP(λ)X\stackrel{\text{iid}}{\sim} P(\lambda)
λlogf(x;λ)=λlogeλλxx!=λ(λ+xlogλlog(x!))=xλ1=xλλ\frac{\partial}{\partial\lambda}\log f(x;\lambda)=\frac{\partial}{\partial\lambda}\log\frac{e^\lambda\lambda^x}{x!}=\frac{\partial}{\partial\lambda}(-\lambda+x\log\lambda-\log(x!))=\frac{x}{\lambda}-1=\frac{x-\lambda}{\lambda}     I(λ)=E[(λlogf(x;λ))2]=E[(xλλ)2]=1λ2E[(xλ)2]=1λ2Var(X)=1λ\begin{align*} \implies I(\lambda)&=E[(\frac{\partial}{\partial\lambda}\log f(x;\lambda))^2]\\ &=E[(\frac{x-\lambda}{\lambda})^2]\\ &=\frac{1}{\lambda^2}E[(x-\lambda)^2]=\frac{1}{\lambda^2}Var(X)\\ &=\frac{1}{\lambda} \end{align*}
  1. Show that the Fisher-info X~=(X1,,Xn)\utilde{X}=(X_1,\cdots,X_n) from P(λ)P(\lambda) contains about λ\sqrt\lambda is independent of λ\lambda.

    know In(λ)=nIX(λ)=nλI_n(\lambda)=nI_X(\lambda)=\frac{n}{\lambda}

    ξ=λ    λ=ξ2=g(ξ)\xi=\sqrt\lambda\implies\lambda=\xi^2=g(\xi)     In(ξ)=(g(ξ))2In(λ)=(2ξ)2nλ=4(λ)2nλ=4n\implies I_n(\xi)=(g'(\xi))^2I_n(\lambda)=(2\xi)^2\frac{n}{\lambda}=4(\sqrt{\lambda})^2\frac{n}{\lambda}=4n


有的時候,我們關心的 η(θ)\eta(\theta) 不存在無偏估計,e.g. B(1,p),η(p)=1pB(1,p),\eta(p)=\frac{1}{p}。又或者存在無偏估計,但 UMVUE 超出了參數的範圍。

EX(#12.3.11): f(x;θ)=θ(1θ)x,x=0,1,,θΩ=(0,1)f(x;\theta)=\theta(1-\theta)^x, x=0,1,\cdots,\theta\in\Omega=(0,1)

U(X)={1ifX=00ifX0U(X)= \begin{cases} 1 & if X=0\\ 0 & if X\neq 0 \end{cases}

U(X)U(X) 是 UMVUE,但 U(X)ΩU(X)\notin\Omega