Formal definition of entropyExample: entropy of Bernoulli RVEntropy & Differential EntropyJoint entropyConditional entropyInformation propagation in a channelCross Entropy (or Mutual Information)Cross Entropy (or Mutual Information)Formal definition of cross-entropyImage Mutual Information (IMI)IMI for rectangular matrices (1)IMI for rectangular matrices (2)IMI for rectangular matrices (3)Formal definition of entropyEntropyEntropy in thermodynamics (discrete systems):•log2[how many are the possible states of the system?]E.g. two-state system: fair coin, outcome=heads (H) or tails (T)Entropy=log22=1Unfair coin: seems more reasonable to “weigh” the two statesaccording to their frequencies of occurence (i.e., probabilities)()()statelogstateEntropy2statespp∑−=MIT 2.71704/13/05 – wk10-b-1Example: entropy of Bernoulli RV• Fair coin: p(H)=1/2; p(T)=1/2bit 121log2121log21Entropy22=−−=• Unfair coin: p(H)=1/4; p(T)=3/4bits 81.043log4341log41Entropy22=−−=Maximum entropy Maximum entropy ⇔⇔Maximum uncertaintyMaximum uncertaintyMIT 2.71704/13/05 – wk10-b-2Entropy & Differential Entropy• Discrete objects (can take values among a discrete set of states)– definition of entropy– unit: 1 bit (=entropy value of a YES/NO question with 50% uncertainty)• Continuous objects (can take values from among a continuum)– definition of differential entropy– unit: 1 nat (=diff. entropy value of a significant digit in the representation of a random number, divided by ln10)()()kkkxpxp2logEntropy∑−=()()()xxpxpXd lnEntropy Diff.∫Ω−=MIT 2.71704/13/05 – wk10-b-3Joint entropyJoint EntropyJoint Entropylog2[how many are the possible states of a combined variableobtained from the Cartesian product of two variables?]()()()yxpyxpYXXxYy,log, ,EntropyJoint 2states states∑∑∈∈−=MIT 2.71704/13/05 – wk10-b-4hardwarechannel“physicalattributes”(measurement)objectfieldpropagationdetectiongHf()?,EntropyJoint E.g. =GFConditional entropyConditional EntropyConditional Entropylog2[how many are the possible states of a combined variablegiven the actual state of one of the two variables?]()()()xypyxpXYXxYy|log, |Entropy Cond.2states states∑∑∈∈−=MIT 2.71704/13/05 – wk10-b-5hardwarechannel“physicalattributes”(measurement)objectfieldpropagationdetectiongHf()?|Entropy Cond. E.g. =FGInformation propagation in a channelhardwarechannel“physicalattributes”(measurement)objectfieldpropagationdetectiongHfNoise adds uncertaintyadds uncertainty to the measurement wrt the object⇔eliminates informationeliminates information from the measurement wrt objectMIT 2.71704/13/05 – wk10-b-6Cross Entropy (or Mutual Information)uncertainty added due to noise()FEntropy()GEntropy()GF |Entropy Cond.()FG |Entropy Cond.informationcontainedin the measurementrepresentation bySeth Lloyd, 2.100MIT 2.71704/13/05 – wk10-b-7informationcontainedin the objectinformation eliminated due to noise()GF ,Ccross-entropy(aka mutual information)Cross Entropy (or Mutual Information)()FEntropy()GEntropy()GF ,EntropyJoint ()GF |Entropy Cond.()FG |Entropy Cond.()GF ,CMIT 2.71704/13/05 – wk10-b-8Formal definition of cross-entropyFFGGinformationinformationreceiverreceiver(measurement)(measurement)Corruption source (Noise)Corruption source (Noise)Physical ChannelPhysical Channel(transform)(transform)informationinformationsourcesource(object)(object)),(EntropyJoint )(Entropy)(Entropy )|(Entropy Cond.)(Entropy )|(Entropy Cond.)(Entropy),(CGFGFFGGGFFGF−+=−=−=MIT 2.71704/13/05 – wk10-b-9Image Mutual Information (IMI)hardwarechannel“physicalattributes”(measurement)objectfieldpropagationdetectiongHfMIT 2.71704/13/05 – wk10-b-10Assumptions: (a) f has Gaussian statistics;(b) white additive Gaussian noise (waGn)i.e. g=Hf+nwhere n is a Gaussian random vector, independent of f, with correlation matrix of the form σ2 I.Thenquantifies information transfer between f and g.()∑=⎟⎟⎠⎞⎜⎜⎝⎛+=nkk1221ln21,CσµgfH of seigenvalue :kµIMI for rectangular matrices (1)MIT 2.71704/13/05 – wk10-b-11= =HHunderdeterminedunderdetermined(more unknowns thanmeasurements)overdeterminedoverdetermined(more measurementsthan unknowns)eigenvalues cannot be computed, but insteadwe compute the singular valuessingular values of therectangular matrixIMI for rectangular matrices (2)HTHsquare matrix=()gfT1TˆHHH−=recall pseudo-inverseinversion operation associated with rank of()()HHH aluessingular vseigenvalueT≡MIT 2.71704/13/05 – wk10-b-12IMI for rectangular matrices (3)hardwarechannel“physicalattributes”(measurement)objectfieldpropagationdetectiongHfunder/over determined∑=⎟⎠⎞⎜⎝⎛+=nkk121ln21Cστsingular valuesof HMIT 2.71704/13/05 –