We propose a method to improve the understanding and visualization of the actual condition of a construction site by automatically developing an as-is BIM using site-appearance information from images and the as-planned BIM model. This is achieved by generating point clouds (PCs) from site images applying structure from motion (SfM). The corresponding elements between PCs and the 3D BIM were automatically determined using geometric and position assets to register PCs into the as-planned BIM accurately. Moreover, material condition classification can be done using information from the point cloud data. This way, the as-is BIM can be enriched with additional information such as the actual material conditions. The proposed method has been demonstrated using a construction environment where the as-is BIM was developed automatically from a set of 130 site images and the as-planned BIM. The as-is model has been used to identify deviations between as-planned and as-is conditions.