例题与证明三

离散平稳无记忆信源X的N次扩展信源的熵为离散 信源X的熵的N倍。H(Xn) = NH(X)证明：H(Xn) = -EP(X)log P(X)= -Ep(a )log p(a )2 i 2 iX NX N工求和是对信源XN中所有nN个元素求和，可以等 效成N个求和，而其中的每一个又是对X中的n个 元素求和，所以有：工p (a )-艺艺艺 p (x ) p (x ) p (x )iiii12NX Ni -1 i -1i -1X12Ni2-艺 p(x )工 p(x )p(x ) - 1iii12Ni -1i -1i -112NH(Xn)= 一工p(a )log p(x )p(x )p(x )ci2i1i2iN=一乞 p (a )log p (x ) 一工 p (a )log p (x ) 一/i 2i1i 2i2X NX N丄 p (a )log p (x )i 2iNXN共有 N 项，考察其中第一项工p(a )log p(x )=工p(x )p(x ) p(x )log p(x )i2i1i1i2iN2i1X NX N二工p(x )log p(x )工p(x )工p(x )ci12i1i2iNi=1J=1i-N=1n因为工 p(x )二 1 k 二 1,2,Nikik =1所以工 p( )log p(x )=工 p(x )log p(x ) = H(X)2 i1i 2i1i1XNii =1同理其余各项均等于 H(X)证毕。故有： H(X N ) = NH(X)L.x , x , xX123=V1 1 1 P( X)2 4 4 J例3. 1有一离散平稳无记忆信源r3 p(x.)=】，求此信源的二次扩展ii=1信源 X 2 的熵。先求出此离散平稳无记忆信源的二次扩展信源。扩 展信源的每个元素是信源 X 的输出长度为 2 的消息序 列。由于扩展信源是无记忆的，故p(a ) = p(x )p(x )(i ,i = 1,2,3;i = 1,2,，9)i i21 2X 2信源的兀素a1a2a3a4a5a6a7a8a9对应的消息序列x xx xx xx xx xx xx xx xx x1 11 21 32 12 22 33 13 23 3概率p(a )111111111i4888161681616根据熵的定义，二次扩展信源的熵为H(X2) = H(XX )12=H (1/16,1/8,1/16,1/ 8,1/4,1/8,1/16,1/8,1/16)=3(bit / symbol) = 2H (X) = 2H (1/4,1/2,1/4)结论：计算扩展信源的熵时，不必构造新的信源，可直接从原信源X的熵导出。即离散平稳无记忆信源X的N次扩展信源的熵为离散信源X的熵的N倍。思考证明二维离散有记忆信源的熵不大于二维平稳无记忆信源的熵？H (X X ) _ P( X)_、4936 一源模型为，X=X X 中前后两个符号的12条件概率为xxxx7/92/90x1/83/41/8x02/119/11原始信源的熵为：H (X) = -z3 p(x )log p(x ) = 1.542(bit/ symbol)i2ii=1由条件概率确定的条件熵为：H(X / X )=-z3z 3 p(x )p(x /x )log p(x /x ) = 0.870(bit / symbol)21ii i2i ii1=1 i2=1条件熵比信源熵(无条件熵)减少了 0.672bit/symbol, 正是由于符号之间的依赖性所造成的。信源X= xx平均每发一个消息所能提供的信息量，即12 联合熵H(XX ) = H(X ) + H(X /X ) = 2.412(bit/symbol) 1 2 1 2 1 则每一个信源符号所提供的平均信息量H2(X)=H(X X ) = 1.206(bit / symbol)12小于信源X所提供的平均信息量H(X),这同样是由于 符号之间的统计相关性所引起的。将二维离散平稳有记忆信源推广到N维情况，可证H( x ) = H( XX X )1 2 N=H(X)+H(X /X)+H(X /XX )+H(X /XX X ) 12 13 1 2N 1 2N -1证明：H (x ) = H (X X X )1 2 N令 Y = X X X, Y = X X X , Y = X X1 1 2N -1 2 1 2N -2 N-21 2则H(X) = H(YX ) = H(Y) + H(X /Y)1 N 1 N 1=H(XX X) + H(X /XX X )1 2N1N 1 2N1=H (Y ) + H (X/ Y ) + H (X / XX X )2N 12N 1 2N1=H(Y ) + H(X /Y ) + + H(X /XX X )N 23 N 2N 1 2N 1=H(XX ) + H(X /XX ) + + H(X /XX X )1 23 1 2N 1 2N1=H(X ) + H(X /X ) + H(X /XX ) + + H(X /XX X )1213 1 2N 1 2N1 |H(X ) + H(X ) + H(X ) + H(X )1213N表明：多符号离散平稳有记忆信源X的熵H(X)是X中起始时刻随机变量X1的熵与各阶条件熵之和。l=J|=|证明离散平稳信源条件熵随变量数N增大而减小的非递增性，即H(XI XX .X ) H (XI XX .X)N 12N1N112N2由信源熵不等式=1=1-工 P log P -工 P log Piii=1ii=1其中 1Lp = 1； = 1；i =1i =1令 P = P(xI x x .x)，iN 1 2N 1P = P( xiN 1Ixx .x )1 2 N 2代入上式不等式一工P(x I x x .x)log P(x I x x .x)N 1 2N 一1N 1 2N 一1X NP(x Ixx .x)logP(x Ixx .x )N 1 2N 一1N 一11 2N 一2X N两边乘上 P( x x .x)工P(x x .x x )log P(x I x x .x)1 2N-1 NN 1 2N-1XNP(xx .xx )logP(x Ixx .x)1 2N 一1 NN 一11 2 N 一2两边对x1兀2-1求和-XX工 P( x x .xx )log P( x I x x .x)1 2N 一1 NN 1 2N 一1X1X2 一X NXX .XP( x x .x x )log P(xI x x .x)1 2N 一1 NN 一1 1 2N 一 2X X X 12N=-XX X P( x x .x )log P( xI x x .x)1 2N -1N -1 1 2 N - 2X X X1 2N -1即 |H(XI XX .X) H (X I XX .X)NN 12N-1根据平均符号熵的定义1H (X) = H(X , X X )NN12N1=H(X ) + H(X I X ) + H(X I X X ) +.N 1 2 1 3 1 2+ H(X I X ,X X )N 12N -11 _ N - H(X I X , X ,X ) NN 12N-1=H(X I X , X X )N 12N -1证明平均符号熵随N单调下降N - H (X) = H(X , X X )=艺 p(a )log p(a )N12Niii=1I丄 p(a a .a )log p(a a .a )i1 i 2iNi1 i2iNi1=1 iN =1I丄 p(a a .a )log p(a a .a) p(aI a a .a)i1 i2iNi1 i2iN-1iN i1 i2iN-1i1=1 iN =1.aiN -1=-Y.工 p(a a .a)log p(a a .a)i1 i2iN-1i1 i2 iN-1.ai1 i 2 iN.a)i1 i2iN -1iNi1=1 iN-1=1 -工丄 p(a a .a )log p(aI a a .i1=1 iN =1=H(X , X X ) + H(X I X , X X )12N -1N 12N -1N -1 (N -1) - H(X) + H (X)N -1N=(N -1)- H(X) + |H(X I X , X X)N 12N-1也就是(N -1) - H (X) (N -1) - H (X) NN-1最后：H (X) H (X)NN -1