The approaching 5G era of cellular communications is posing stringent performance requirements. New groundbreaking applications can be enabled only by means of multi-Gbps data rates and ultra-low latencies. The spectrum scarcity at frequencies below 6 GHz stimulated a new wave of wireless research that focuses on higher bands, namely mmWave. Directionality and high penetration loss represent the key challenges when operating with such carriers. The resulting intermittent connectivity makes the user association problem even more complex and critical than in previous generations of cellular systems, where the channel was better behaved. In this paper, we aim at deriving an optimal and fair cell selection policy that encapsulates the reallocation cost of potential handovers, and captures the erratic nature of the mmWave channel. An important conclusion is that (i) if there is no, or minimal, reallocation cost, each user associates with a single BS, while (ii) for higher handover cost values, users tend to connect to multiple base stations simultaneously.