In this work we propose a Voice over IP (VoIP) call tracking scheme based on robust hashing of VoIP flows. In the proposed scheme the audio content of a possibly encrypted VoIP flow is identified by a short binary string, called the robust hash, using variations on the flow's bitrate over time. These robust hashes are then used to detect pairs of parties communicating with each other. In summary, if two parties are communication with each other then they have a pair of VoIP flows having similar robust hashes with high probability. The basic intuition behind the proposed hash function is twofolds: i)The variable bitrate codec employed in most VoIP applications result in a distinctive bitrate variations over time for each VoIP flow depending on the underlying audio content. ii) Encryption typically doesn't change the bitrate of a VoIP flow. Our experiments show that the proposed scheme is able to identify Skype VoIP flows even under various network impairments such as packet delays, jitter and packet drops.