Unpaired Depth Super-Resolution in the Wild

Depth images captured with commodity sensors commonly suffer from low quality and resolution and require enhancing to be used in many applications. State-of-the-art data-driven methods for depth super-resolution rely on registered pairs of low- and high-resolution depth images of the same scenes. Acquisition of such real-world paired data requires specialized setups. On the other hand, generating low-resolution depth images from respective high-resolution versions by subsampling, adding noise and other artificial degradation methods, does not fully capture the characteristics of real-world depth data. As a consequence, supervised learning methods trained on such artificial paired data may not perform well on real-world low-resolution inputs. We propose an approach to depth super-resolution based on learning from unpaired data. We show that image-based unpaired techniques that have been proposed for depth super-resolution fail to perform effective hole-filling or reconstruct accurate surface normals in the output depth images. Aiming to improve upon these approaches, we propose an unpaired learning method for depth super-resolution based on a learnable degradation model and including a dedicated enhancement component which integrates surface quality measures to produce more accurate depth images. We propose a benchmark for unpaired depth super-resolution and demonstrate that our method outperforms existing unpaired methods and performs on par with paired ones. In particular, our method shows 28% improvement in terms of a perceptual MSEv quality measure, compared to state-of-the-art unpaired depth enhancement techniques adapted to perform super-resolution [e.g., Gu et al. (2020)]. The implementation of our method is publicly available at https://github.com/keqpan/udsr.