Foundations of Computer SecurityLecture 34: Fundamental TheoremsDr. Bill YoungDepartment of Computer SciencesUniversity of Texas at AustinLecture 34: 1 Fundamental TheoremsSource EntropyThe entropy of a language is a measure of the most efficientpossible encoding of the language.The entropy of a message source is the amount of informationcontent that the source can produce in a given period. This ismeasured in bits per second.Any channel can transmit an arbitrary amount of information,given enough time and unbounded buffering capacity. But can agiven channel transmit the information in real time?Lecture 34: 2 Fundamental TheoremsChannel CapacityThe capacity of a channel is the number of bits that can be sentper second over the channel. This is a property of thecommunication medium.Fundamental Theorem of the Noiseless Channel. (Shannon):If a language has entropy h (bits per symbol) and a channel cantransmit C bits per second, then it is possible to encode the signalis such a way as to transmit at an average rate of (C /h) − ǫsymbols per second, where ǫ can be made arbitrarily small. It isimpossible to transmit at an average rate greater than C /h.Lecture 34: 3 Fundamental TheoremsWhat the Theorem MeansSuppose a channel can handle 100 bits / second and your languagehas entropy 5 (bit per symbol).Given a perfect encoding and a noiseless channel, you’d expect tobe able to transmit 20 symbols / second though the channel, onaverage. Right?But you may not have a perfect encoding. Doesn’t matter. Youcan always find a better encoding to get within ǫ of that limit.Lecture 34: 4 Fundamental TheoremsNoisy ChannelsIf the channel is noisy, the capacity is reduced by the noise. Butthe following is true:Fundamental Theorem of a Noisy Channel (Shannon): Let adiscrete channel have a capacity C and a discrete source anentropy h (bits per second). If h < C there exists a coding systemsuch that the output of the source can be transmitted over thechannel with an arbitrarily small frequency of errors.If the channel (with noise factored in) can physically handle themessage traffic, then it is possible to transmit with arbitrarily smallerror rate.Lecture 34: 5 Fundamental TheoremsWhat this Theorem MeansThe upshot of this is that a message can be transmitted reliablyover even a very noisy channel by increasing the redundancy of thecoding scheme.For example, covert channels in the system cannot be dismissedwith the argument that they are noisy and hence useless. You canalways get the message through by finding a more redundantencoding.Lecture 34: 6 Fundamental TheoremsLessonsEntropy provides a bound on coding efficiency.But Shannon’s theorems show that it is always possible toapproach that limit arbitrarily closely.Next lecture: Entropy of EnglishLecture 34: 7 Fundamental
View Full Document
Unlocking...