Recent developments in immersive audio technology have motivated a proliferation of binaural renderers used for creating spatial audio content. Binaural renderers leverage psychoacoustic features of human hearing to reproduce a 3D sound image over headphones. In this paper, a methodology for the comparative evaluation of different binaural renderers is presented. The methodological approach is threefold: a subjective evaluation of 1) quantitative characteristics (such as front/back and up/down discrimination and localization); 2) qualitative characteristics (such as naturalness and spaciousness); and 3) overall preference. The main objective of the methodology is to help to elucidate the most meaningful factors for the performance of binaural renderers and to provide insight on possible improvements in the rendering process.