WUSTL ESE 523 - ESE523Lect2-6(1) (77 pages)

Previewing pages 1, 2, 3, 4, 5, 36, 37, 38, 39, 40, 73, 74, 75, 76, 77 of 77 page document View the full content.
View Full Document

ESE523Lect2-6(1)



Previewing pages 1, 2, 3, 4, 5, 36, 37, 38, 39, 40, 73, 74, 75, 76, 77 of actual document.

View the full content.
View Full Document
View Full Document

ESE523Lect2-6(1)

42 views


Pages:
77
School:
Washington University in St. Louis
Course:
Ese 523 - Information Theory
Information Theory Documents

Unformatted text preview:

ESE 523 Information Theory Joseph A O Sullivan Samuel C Sachs Professor Electrical and Systems Engineering Washington University 211 Urbauer Hall 2120E Green Hall 314 935 4173 jao wustl edu 09 3 13 J A O Sullivan ESE 523 Lecture 2 6 1 Binary Entropy Function 1 Outline p 1 p H p p log p 1 p log 1 p Entropy 0 8 0 6 0 4 0 2 0 0 0 2 0 4 0 6 Probability of One 0 8 1 H X p x log p x Entropy x X H X Y p x y log p x y Joint Entropy x X y Y Conditional Entropy H X Y p x y log p x y Relative Entropy D p q p x log x X y Y x X Mutual Information I X Y x X y Y 09 3 13 J A O Sullivan ESE 523 Lecture 2 6 p x q x p x y p x y log p x p y 2 Notation X Random variable R V Alphabet discrete X x1 x2 xn Probability mass function P X xi pi p i p xi pi 0 log log2 p x X i 1 Biased coin flip X h t p x p 1 p Two dice X 2 3 4 5 6 7 8 9 10 11 12 p x 1 2 3 4 5 6 5 4 3 2 1 36 1 Powerball 09 3 13 59 1 p x 39 195 249 054 5 3 J A O Sullivan ESE 523 Lecture 2 6 Measure of Information Entropy The entropy of X H X is H X p x log p x x X Units are bits Measure of uncertainty of a R V H X E log p X 1 E log p X 09 3 13 the eerily self referential expectation Cover and Thomas p 14 J A O Sullivan ESE 523 Lecture 2 6 4 Entropy Example 1 Deterministic R V p x i 1 and p x j 0 j i H X 0 No information gained from observing the outcome 1 log 1 1 0 0 0 log 0 lim log 0 0 Proof uses l Hopital s rule log 1 1 lim log lim lim log e 0 2 0 0 0 1 1 09 3 13 J A O Sullivan ESE 523 Lecture 2 6 5 Entropy Example 2 Flip a fair coin 1 X h t p h p t 2 1 1 1 1 H X log log 2 2 2 2 1 bit 09 3 13 J A O Sullivan ESE 523 Lecture 2 6 6 Entropy Example 3 Flip a fair coin n times X h h h h h t t t t 1 p xi n i 1 2 2n 2 2n 1 1 H X n log n 2 i 1 2 n bits 09 3 13 J A O Sullivan ESE



View Full Document

Access the best Study Guides, Lecture Notes and Practice Exams

Loading Unlocking...
Login

Join to view ESE523Lect2-6(1) and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view ESE523Lect2-6(1) and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?