The morphology of the retinal vascular structure in fundus images is of great importance for ocular disease diagnosis. However, due to the poor fundus image quality and domain shifts between datasets, retinal vessel segmentation has long been regarded as a problematic machine-learning task. This work proposes a novel algorithm High-frequency Guided Cascaded Network (HGC-Net) to address the above issues. In our algorithm, a self-supervision mechanism is designed to improve the generalizability and robustness of the model. We apply Fourier Augmented Co-Teacher (FACT) augmentation to convert the style of fundus images, and extract high-frequency component (HFC) to highlight the vascular structure. The main structure of the algorithm is two cascaded U-nets, in which the first U-net generates a domain-invariant high-frequency map of fundus images, thus improving the segmentation stability of the second U-net. Comparison with the state-of-the-art methods and ablation study are conducted to demonstrate the excellent performance of our proposed HGC-Net.