Automatic segmentation of the whole heart in computed tomography(CT) image is crucial for efficient treatment planning of thoracic radiotherapy. In this paper, we propose a fully automatic method for whole heart segmentation of thoracic CT images. A robust active shape model (Robust ASM) is proposed using shape models developed from training data to reduce outliers due to similar intensity of neighboring organs. A novel shape constrained active contour model is presented to further improve the segmentation result. A mean point-to-surface error of 2.37mm was measured based on 38 images. The averaged Dice index is 0.90.