"You should call it entropy for two reasons: first because that is what the formula is in statistical mechanics but second and more important, as nobody knows what entropy is, whenever you use the term you will always be at an advantage!"
John von Neumman
ECE5583/TCOM5583: Information Theory
Prerequisites: ECE 4523 or instructor permission
This is a graduate introductory course in information theory targeted to graduate and senior students. The goal is to both inspire students with the thoughtprovoking ideas of information theory and provide them enough knowledge necessary for further study of the subject.
Course Outline:
 Overview
 Review of probability theory
 Lossless source coding theory, Huffmann coding, and introduction to Arithmetic coding
 Asymptotic equipartition property (AEP), typicality and joint typicality
 Entropy, conditional entropy, mutual information, and their properties
 Introducing project
 Channel coding theory, capacity, and Fano’s inequality
 Continuous random variables, differential entropy, Gaussian source, and Gaussian channel
 Bandlimited AWGN channel, parallel Gaussian channels, waterfilling and inverse waterfilling
 Error correcting codes, linear codes, and introduction to lowdensity parity check code
 Digital fountain, luby code, raptor code
 Lossy source coding theory and ratedistortion function
 Duality of source and channel coding, separation theorem
 Method of type, large deviation theory, maximum entropy principle
 Network information theory
Textbook
“Elements of Information Theory,” by Cover and Thomas, New York: Wiley.
Second edition is quite a bit better. But 1st edition is okay too, given that it is way cheaper in Amazon. You probably can get one below 20 bucks.
Auxiliary and Reference Material:
 Shannon, C. E. (1948) A mathematical theory of communication. Bell Sys. Tech. J. 27: 379423, 623656.
 “Information Theory, Inference, and Learning Algorithms,” by David J.C. Mackay, Cambridge: Cambridge University Press.
 R. W. Yeung, “On entropy, information inequalities, and Groups.”
 "Law of Large Number," by Terry Tao.
 “A First Course in Information Theory,” by Raymond W. Yeung, New York: Springer.
 “Information Theory and Reliable Communication,” by R. Gallager, New York: Wiley.
 “Information Theory,” by Csiszar and Korner, New York: Academic Press.
 “Entropy and Information Theory,” by R. M. Gray, SpringerVerlag, 1990.
 “Probability, Random Processes, and Ergodic Properties,” by R. M. Gray, SpringerVerlag, 1988.
 J. S. Yedidia, W. T. Freeman, and Y. Weiss, "Understanding Belief Propagation and its Generalizations," in Exploring Artificial Intelligence in the New Millennium: Science and Technology Books, 2003.
 P. A. Chou and Y. Wu, "Network Coding for the Internet and Wireless Networks," Signal Processing Magazine, IEEE, vol. 24, pp. 7785, 2007.
 S. Katti, S. Shintre, S. Jaggi, D. Katabi, and M. Medard, "Real Network Codes," in Allerton, 2007.
 D. J. C. MacKay, "Fountain codes," IEE Communications, vol. 152, pp. 10621068, 2005.
 S. Verdu, "Fifty years of Shannon theory," Information Theory, IEEE Transactions on, vol. 44, pp. 20572078, 1998.
 A. R. Calderbank, "The art of signaling: fifty years of coding theory," Information Theory, IEEE Transactions on, vol. 44, pp. 25612595, 1998.
 I. Csiszar, "The method of types [information theory]," Information Theory, IEEE Transactions on, vol. 44, pp. 25052523, 1998.
 Information Theoretic Inequality Prover
 Xian Qian, Xiaoqian Jiang, Qi Zhang, Xuanjng Huang, and Lide Wu, "Sparse higher order conditional random fields for improved sequence labeling," ICML, 2009.
Grading
Like last year, I want to make grading very concrete this time. You will get
 A: Exam average > 85%
 B: Exam average > 70%
 C: Exam average > 50%
 D: Otherwise.
Be forewarned, getting an A in this course is neither automatic nor easy. You will need some real effort and high maturity on your math ability.
Project (Extra Credit)
I expect a screencast and a report on a relevant topic including what, why, and how. A good report should reflect the understanding of the author on the topic. You may do a group project if you want and you may submit one project report but you need to show how your work is separated. But I expect one screencast for each individual even for a group project. I will grade your project based on:
 The relevancy and depth (difficulty) of the chosen topic
 How much you understand the materials that I can perceive
 How much effort you have paid that I can perceive
Here are some suggestions for potential projects.
 Polar Code
 Gaussian Process Regression
 Finite Rate of Innovation
 Coprime Sampling
 Relevance Vector Machine
 Locality Sensitive Hashing
You should submit your report through d2l (there is a dropbox folder "project upload" where you can upload to) and there is a hard deadline of 11:59 PM, Dec 15, 2014. In the report, you also have to specify the link(s) to your screencast(s). You may use any online video server. But I guess YouTube is the probably easiest. Be sure to set the video to either unlisted or public so that others can access to it.
Calendar (Tentative)

Summary 
Course notes 
Homework 
Reference 
8182014 
Overview,
Shannon Entropy, An Example, Lossless Source Coding, Instantaneous Codes 
01 overview.rar 

02 probability.pdf 
8202014 
Kraft Inequality, Uniquely Decodability and Kraft Inequality 
03 lossless source coding.zip 

[ASH '65 p. 511], axiom_note.pdf 
8252014 
SFE Coding, Arithmetic Coding 



8272014 
AEP, An Example, Lower Bound of Source Coding (Part 1, Part 2), Summary of Source Coding Theory 
04 AEP.rar 


932014 
Mutual Information, Conditional Mutual Information, Conditional Entropy, Other Information Measures, Bounds on Information Measures 



982014 
More on Mutual Information and Conditional Mutual Information, An Example, Data Processing Inequality 
05 information measures.rar 


9102014 
Introduction to Channel Coding Theorem, BSC 


conditional relative entropy, supplimentary for Jensen Inequality 
9152014 
Forward proof of Channel Coding Theorem 



9172014 
Fano's inequality, Converse Proof of Channel Coding Theorem 
channel_coding_theorem.pdf 


9222014 
MAP estimator, Maximum Likelihood Estimator 



9242014 
Conjugate Prior, Multivariate Gaussian Distribution 
Notes on Multivariate Gaussian 


9292014 
Marginalization of Multivariate Gussian Distribution 



1012014 
Conditioning of Multivariate Gaussian Distribution




1062014 
Product of Gaussian Pdfs 
07 differential entropy.rar 


1082014 
Mixture of Gaussians 
08 Gaussian channel.rar 


10132014 
Gaussian Process 
channel coding chapter, 


10152014 
Test 1 



10202014 
Differential Entropy, Gaussian Channel 



10222014 
Bandlimited Gaussian Chanel 
09 error correcting codes.rar 


10272014 
Principal Component Analysis 



10292014 
Waterfilling for Parallel Gaussian Channels, Linear Code, Hamming Code, Trellis Coded Modulation, Viterbi Algorithm 



1112014 
Belief Propagation, Forward Backward Algorithm 



1162014 
Low Density Parity Check Code 



11102014 
Convolutional Codes 



11122014 
Turbo Codes 



11172014 
Campus closed 



11192014 
Bayesian Compressed Sensing 


Bayesian CS paper 
11242014 
ICA 



11262014 
Thanskgiving 



1282014 
Final 



Some lecture notes are available in Windows Journal Viewer format.
