The entrenched instability of solar power output throttles its further integration into power grids worldwide. Thus the precise solar power forecasting (SPF) is helpful for the improvement of power grid stability and better exploitation of clean solar energy. As an important role of ultra-short-term SPF, sky images always contain volatile clouds, which results in tempestuous variation of the output of PV plants. Therefore, an accurate model that can capture the mapping relationship between sky image data and solar irradiance data is significant for fulfilling the ultra-short-term SPF. To fill the gap in the content of this research field, this paper proposes two end to end irradiance mapping models based on deep learning technologies, namely convolutional neural network (CNN) and long short-term memory (LSTM) neural network. Then the mapping performance of the above two mapping models is compared to that of traditional artificial neural network (ANN) based model. In all the aforementioned models, it should be noted that the solar irradiance data output by mapping models is in one-to-one correspondence with the input sky image data in time. The deterministic and probability methods are applied to statistically evaluate the mapping result of CNN and LSTM models. Our case study shows that deep learning architectures, especially the CNN based model, are good at mapping sky images to corresponding surface solar irradiance.