Next Best View Planning via Reinforcement Learning for Scanning of Arbitrary 3D Shapes

Abstract: Reconstructing 3D objects from scanned measurements is a fundamental task in computer vision. A central factor for the effectiveness of 3D reconstruction is the selection of sensor views for scanning. The latter remains an open problem in the 3D geometry processing area, known as the next-best-view planning problem, and is commonly approached by combinatorial or greedy methods. In this work, we propose a reinforcement learning-based approach to sequential next-best-view planning. The method is implemented based on the gym environment including 3D reconstruction, next-best-scan planning, and image acquisition features. We demonstrate this method to outperform the baselines in terms of the number of required scans and the obtained 3D mesh reconstruction accuracy.