The ever increasing leakage power of memories in a system has motivated researches for exploiting unconventional memory architectures. Non-Volatile Memory (NVM) used in conjunction with the conventional on-chip SRAMs has given birth to the hybrid memory paradigm, which can be intelligently exploited to reduce the energy consumption while tackling the high read and write latencies of NVMs. We present a novel scheme AMBER that aims at minimizing the total memory energy consumption of a video processing system by leveraging the application-specific properties and distinct latency and power properties of different memory types. AMBER also features architectural support for data-fetching from external memory and adaptively filling the different on-chip memories. We employ AMBER in the next-generation High Efficiency Video Coding (HEVC) standard to minimize the energy consumption of the new complex motion prediction process. Experimental results demonstrate that our AMBER scheme achieves significant energy savings (average 43%) for the on-chip memory.