# WUSTL ESE 523 - ESE523Lect2-6(1) (77 pages)

Previewing pages*1, 2, 3, 4, 5, 36, 37, 38, 39, 40, 73, 74, 75, 76, 77*of 77 page document

**View the full content.**## ESE523Lect2-6(1)

Previewing pages *1, 2, 3, 4, 5, 36, 37, 38, 39, 40, 73, 74, 75, 76, 77*
of
actual document.

**View the full content.**View Full Document

## ESE523Lect2-6(1)

0 0 50 views

- Pages:
- 77
- School:
- Washington University in St. Louis
- Course:
- Ese 523 - Information Theory

**Unformatted text preview: **

ESE 523 Information Theory Joseph A O Sullivan Samuel C Sachs Professor Electrical and Systems Engineering Washington University 211 Urbauer Hall 2120E Green Hall 314 935 4173 jao wustl edu 09 3 13 J A O Sullivan ESE 523 Lecture 2 6 1 Binary Entropy Function 1 Outline p 1 p H p p log p 1 p log 1 p Entropy 0 8 0 6 0 4 0 2 0 0 0 2 0 4 0 6 Probability of One 0 8 1 H X p x log p x Entropy x X H X Y p x y log p x y Joint Entropy x X y Y Conditional Entropy H X Y p x y log p x y Relative Entropy D p q p x log x X y Y x X Mutual Information I X Y x X y Y 09 3 13 J A O Sullivan ESE 523 Lecture 2 6 p x q x p x y p x y log p x p y 2 Notation X Random variable R V Alphabet discrete X x1 x2 xn Probability mass function P X xi pi p i p xi pi 0 log log2 p x X i 1 Biased coin flip X h t p x p 1 p Two dice X 2 3 4 5 6 7 8 9 10 11 12 p x 1 2 3 4 5 6 5 4 3 2 1 36 1 Powerball 09 3 13 59 1 p x 39 195 249 054 5 3 J A O Sullivan ESE 523 Lecture 2 6 Measure of Information Entropy The entropy of X H X is H X p x log p x x X Units are bits Measure of uncertainty of a R V H X E log p X 1 E log p X 09 3 13 the eerily self referential expectation Cover and Thomas p 14 J A O Sullivan ESE 523 Lecture 2 6 4 Entropy Example 1 Deterministic R V p x i 1 and p x j 0 j i H X 0 No information gained from observing the outcome 1 log 1 1 0 0 0 log 0 lim log 0 0 Proof uses l Hopital s rule log 1 1 lim log lim lim log e 0 2 0 0 0 1 1 09 3 13 J A O Sullivan ESE 523 Lecture 2 6 5 Entropy Example 2 Flip a fair coin 1 X h t p h p t 2 1 1 1 1 H X log log 2 2 2 2 1 bit 09 3 13 J A O Sullivan ESE 523 Lecture 2 6 6 Entropy Example 3 Flip a fair coin n times X h h h h h t t t t 1 p xi n i 1 2 2n 2 2n 1 1 H X n log n 2 i 1 2 n bits 09 3 13 J A O Sullivan ESE

View Full Document