Energy-inspired models: Learning with sampler-induced distributions

Energy-based models (EBMs) are powerful probabilistic models [8, 44], but suffer from intractable sampling and density evaluation due to the partition function. As a result, inference in EBMs relies on approximate sampling algorithms, leading to a mismatch between the model and inference. Motivated by this, we consider the sampler-induced distribution as the model of interest and maximize the likelihood of this model. This yields a class of energy-inspired models (EIMs) that incorporate learned energy functions while still providing exact samples and tractable log-likelihood lower bounds. We describe and evaluate three instantiations of such models based on truncated rejection sampling, self-normalized importance sampling, and Hamiltonian importance sampling. These models outperform or perform comparably to the recently proposed Learned Accept/Reject Sampling algorithm [5] and provide new insights on ranking Noise Contrastive Estimation [34, 46] and Contrastive Predictive Coding [57]. Moreover, EIMs allow us to generalize a recent connection between multi-sample variational lower bounds [9] and auxiliary variable variational inference [1, 63, 59, 47]. We show how recent variational bounds [9, 49, 52, 42, 73, 51, 65] can be unified with EIMs as the variational family.