Recently, some concerns have been raised regarding whether the rapid growth of the UDP traffic due to the Skype-like applications will throttle down regular TCP users because UDP flows are unresponsive to network congestion, i.e., UDP senders do not reduce their sending rates when congestion occurs in the network. However, most UDP based real time applications, such as Voice-over-IP (VoIP), drop out completely if the user perceived quality becomes unacceptable as the network congestion increases. This user back-off mechanism effectively reduces UDP traffic rate at the aggregated level when the network is congested. In this paper, we investigate the fairness issue between VoIP flows and TCP flows under different network environment. Our study explicitly takes into account both transport layer congestion control mechanism built-into TCP protocol stack and the self-adaptiveness of VoIP users. Various models are developed to characterize VoIP user back-off behaviors in response to call quality degradation resulted from packet loss and delay inside the network. The comparison between VoIP user back-off and TCP rate adaption shows that not only VoIP flows consume much less bandwidth than TCP flows, but also VoIP traffic is indeed very responsive to congestion when the network is overloaded.