We present a study aimed at understanding how human observers judge scatter plot similarity when presented with a large set of iconic scatter plot representations. The work we present involves 18 participants with a scientific background in a similarity perception study. The study asks participants to group a carefully selected set of plots according to their subjective perceptual judgement of similarity, and it integrates the results into a consensus similarity grouping. We then use this consensus grouping to generate insights on similarity perception. The main output of this work is a list of concepts we derive to describe major perceptual features, and a description of how these concepts relate and rank. We also evaluate scagnostics (scatter plot diagnostics), a popular and established set of scatter plot descriptors, and show that they do not reliably reproduce our participants judgements. Finally, we discuss the major implications of this study and how these results can be used for future research.