TY - GEN

T1 - A Geometric Characterization of Fisher Information from Quantized Samples with Applications to Distributed Statistical Estimation

AU - Barnes, Leighton Pate

AU - Han, Yanjun

AU - Ozgur, Ayfer

N1 - Funding Information:
II. ACKNOWLEDGEMENT This work was supported in part by NSF award CCF-1704624, by the Center for Science of Information (CSoI), an NSF Science and Technology Center, under grant agreement CCF-0939370, and by a generous gift from Huawei Technologies.
Publisher Copyright:
© 2018 IEEE.

PY - 2018/7/2

Y1 - 2018/7/2

N2 - Consider the Fisher information for estimating a vector \theta \in \mathbb {R}^{d} from the quantized version of a statistical sample X \sim f(x|\theta). Let M be a k-bit quantization of X. We provide a geometric characterization of the trace of the Fisher information matrix I-{M}(\theta) in terms of the score function S-{\theta }(X). When k=1, we exactly solve the extremal problem of maximizing this geometric quantity for the Gaussian location model, which allows us to conclude that in this model, a half-space quantization is the one-bit quantization that maximizes Tr(I-{M}(\theta)). Under assumptions on the tail of the distribution of S-{\theta }(X) projected onto any unit vector in \mathbb {R}^{d}, we give upper bounds demonstrating how Tr(I-{M}(\theta)) can scale with k. We apply these results to find lower bounds on the minimax risk of estimating \theta from multiple quantized samples of X, for example in a distributed setting where the samples are distributed across multiple nodes and each node has a total budget of k-bits to communicate its sample to a centralized estimator. Our bounds apply in a unified way to many common statistical models including the Gaussian location model and discrete distribution estimation, and they recover and generalize existing results in the literature with simpler and more transparent proofs.

AB - Consider the Fisher information for estimating a vector \theta \in \mathbb {R}^{d} from the quantized version of a statistical sample X \sim f(x|\theta). Let M be a k-bit quantization of X. We provide a geometric characterization of the trace of the Fisher information matrix I-{M}(\theta) in terms of the score function S-{\theta }(X). When k=1, we exactly solve the extremal problem of maximizing this geometric quantity for the Gaussian location model, which allows us to conclude that in this model, a half-space quantization is the one-bit quantization that maximizes Tr(I-{M}(\theta)). Under assumptions on the tail of the distribution of S-{\theta }(X) projected onto any unit vector in \mathbb {R}^{d}, we give upper bounds demonstrating how Tr(I-{M}(\theta)) can scale with k. We apply these results to find lower bounds on the minimax risk of estimating \theta from multiple quantized samples of X, for example in a distributed setting where the samples are distributed across multiple nodes and each node has a total budget of k-bits to communicate its sample to a centralized estimator. Our bounds apply in a unified way to many common statistical models including the Gaussian location model and discrete distribution estimation, and they recover and generalize existing results in the literature with simpler and more transparent proofs.

UR - http://www.scopus.com/inward/record.url?scp=85062879483&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85062879483&partnerID=8YFLogxK

U2 - 10.1109/ALLERTON.2018.8635899

DO - 10.1109/ALLERTON.2018.8635899

M3 - Conference contribution

AN - SCOPUS:85062879483

T3 - 2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018

SP - 16

EP - 23

BT - 2018 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 56th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2018

Y2 - 2 October 2018 through 5 October 2018

ER -