Optimality of the Plug-in Estimator for Differential Entropy Estimation under Gaussian Convolutions

Ziv Goldfeld, Kristjan Greenewald, Jonathan Weed, Yury Polyanskiy

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper establishes the optimality of the plugin estimator for the problem of differential entropy estimation under Gaussian convolutions. Specifically, we consider the estimation of the differential entropy h(X + Z), where X and Z are independent d-dimensional random variables with Z{\sim}\mathcal{N}( {0,{σ ^2}{{\text{I}}-d}} ). The distribution of X is unknown and belongs to some nonparametric class, but n independently and identically distributed samples from it are available. We first show that despite the regularizing effect of noise, any good estimator (within an additive gap) for this problem must have an exponential in d sample complexity. We then analyze the absolute-error risk of the plug-in estimator and show that it converges as frac{{{c^d}}}{{n }}, thus attaining the parametric estimation rate. This implies the optimality of the plug-in estimator for the considered problem. We provide numerical results comparing the performance of the plug-in estimator to general-purpose (unstructured) differential entropy estimators (based on kernel density estimation (KDE) or k nearest neighbors (kNN) techniques) applied to samples of X + Z. These results reveal a significant empirical superiority of the plug-in to state-of-the-art KDE- and kNN-based methods.

Original languageEnglish (US)
Title of host publication2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages892-896
Number of pages5
ISBN (Electronic)9781538692912
DOIs
StatePublished - Jul 2019
Event2019 IEEE International Symposium on Information Theory, ISIT 2019 - Paris, France
Duration: Jul 7 2019Jul 12 2019

Publication series

NameIEEE International Symposium on Information Theory - Proceedings
Volume2019-July
ISSN (Print)2157-8095

Conference

Conference2019 IEEE International Symposium on Information Theory, ISIT 2019
CountryFrance
CityParis
Period7/7/197/12/19

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Information Systems
  • Modeling and Simulation
  • Applied Mathematics

Fingerprint Dive into the research topics of 'Optimality of the Plug-in Estimator for Differential Entropy Estimation under Gaussian Convolutions'. Together they form a unique fingerprint.

  • Cite this

    Goldfeld, Z., Greenewald, K., Weed, J., & Polyanskiy, Y. (2019). Optimality of the Plug-in Estimator for Differential Entropy Estimation under Gaussian Convolutions. In 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings (pp. 892-896). [8849414] (IEEE International Symposium on Information Theory - Proceedings; Vol. 2019-July). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISIT.2019.8849414