Jethro's Braindump

hjelm_learning_2019: Learning deep representations by mutual information estimation and maximization

Learning deep representations by mutual information estimation and maximization

Local MI Maximization

Maximizing Mutual Information between the input and output may not be useful for the task. Trivial pixel-level noise is useless for image classification. By maximizing the average mutual information between the high-level representation and local patches of the image, one can encourage the encoding aspects of data that is shared across patches.

Links to this note