Maximum Likelihood Estimation of information measures

Jiantao Jiao, Kartik Venkat, Yanjun Han, Tsachy Weissman

Research output: Chapter in Book/Report/Conference proceedingConference contribution


The Maximum Likelihood Estimator (MLE) is widely used in estimating information measures, and involves 'plugging-in' the empirical distribution of the data to estimate a given functional of the unknown distribution. In this work we propose a general framework and procedure to analyze the nonasymptotic performance of the MLE in estimating functionals of discrete distributions, under the worst-case mean squared error criterion. We show that existing theory is insufficient for analyzing the bias of the MLE, and propose to apply the theory of approximation using positive linear operators to study this bias. The variance is controlled using the well-known tools from the literature on concentration inequalities. Our techniques completely characterize the maximum L2 risk incurred by the MLE in estimating the Shannon entropy H(P) = σi=1S -piln pi, and Fα(P) = σi=1Spiα up to a multiplicative constant. As a corollary, for Shannon entropy estimation, we show that it is necessary and sufficient to have n ≪ S observations for the MLE to be consistent, where S represents the support size. In addition, we obtain that it is necessary and sufficient to consider n ≪ S1/α samples for the MLE to consistently estimate Fα(P); 0 <α < 1. The minimax rate-optimal estimators for both problems require S/ln S and S1/α / ln S samples, which implies that the MLE is strictly sub-optimal. When 1 < α < 3/2, we show that the maximum L2 rate of convergence for the MLE is n-2(α-1) for infinite support size, while the minimax L2 rate is (n ln n)-2(α-1). When α ≥ 3/2, the MLE achieves the minimax optimal L2 convergence rate n-1 regardless of the support size.

Original languageEnglish (US)
Title of host publicationProceedings - 2015 IEEE International Symposium on Information Theory, ISIT 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages5
ISBN (Electronic)9781467377041
StatePublished - Sep 28 2015
EventIEEE International Symposium on Information Theory, ISIT 2015 - Hong Kong, Hong Kong
Duration: Jun 14 2015Jun 19 2015

Publication series

NameIEEE International Symposium on Information Theory - Proceedings
ISSN (Print)2157-8095


OtherIEEE International Symposium on Information Theory, ISIT 2015
Country/TerritoryHong Kong
CityHong Kong

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Information Systems
  • Modeling and Simulation
  • Applied Mathematics


Dive into the research topics of 'Maximum Likelihood Estimation of information measures'. Together they form a unique fingerprint.

Cite this