Real time anomaly detection has been attracting considerable interest in traffic control and management applications. As a low-cost, high-efficiency emerging technology, automated video analysis has been adopted by agencies that are convinced of the feasibility of this new technology due to the impressive developments in computer vision and image processing fields. This paper presents a two-layer learning framework for unsupervised anomaly detection based on traffic surveillance videos. The static layer uses background extraction and blurring of moving vehicles to identify static vehicles. The dynamic layer uses proposed time dependent vector field cross correlation method, processes the traffic flow into vector fields and, compares the difference between average vector fields and instant fields to determine anomaly events. This framework has been shown to have a low training cost, acceptable accuracy, and relatively high level of adaptability for real-life video sets obtained from different locations.