Mutual Information in Machine Learning (1/3)

Presenter(s)

Chung Chan

City University of Hong Kong

2021 Croucher Summer Course in Information Theory, The Chinese University of Hong Kong

Lecture

Date

Aug 23, 2021

Abstract

Mutual information is a fundamental quantity in information theory. It is widely used in machine learning to measure statistical dependency among different features in data. Applications are numerous, ranging from classification, clustering, representation learning, and other tasks that require the selection/extraction of lower-dimensional features of the data without losing valuable information. Although mutual information has a precise formula defined in terms of a probability model, it must be estimated for real-world data with an unknown probability model. In this lecture series, we will dive into some of the applications and estimations of mutual information in machine learning. Registered participants will have hands-on coding experience using the virtual teaching and learning environment DIVE offered by CityU CS Department.

Biography

Chung Chan received the B.Sc., M.Eng. and Ph.D. from the EECS Department at MIT in 2004, 2005 and 2010 respectively. He was a Research Assistant Professor at the Institute of Network Coding, the Chinese University of Hong Kong from 2013 to 2017. He is currently an Assistant Professor at the Department of Computer Science, City University of Hong Kong. His research interest is to develop general information measures and flow models from network information theory that are applicable to practical problems. His research topics include the development of network link models using matroids, the derivation of theoretical limits and optimal strategies for the problems of multiterminal source coding, data exchange, and secret generation. His most significant work is the extension of Shannon’s mutual information to the multivariate case, and the discovery of its connections to various problems in information theory and machine learning. During the COVID-19 pandemic in 2020/21, he received the outstanding teaching award from CityU College of Engineering for the DIVE virtual learning and teaching environment.