The key to designing better home energy management systems is in-depth understanding of the context underlying energy usage. The common method of inferring the underlying context is data collection through extensive sensor deployments and then deriving contextual ties between factors like occupancy and energy consumption. There is, therefore, a lack of studies that use first principle approaches like interviewing households to understand the major factors that influence energy consumption. In this work-in-progress paper, we present preliminary results from an interview-based study on households in low-income neighborhoods in Baltimore City. We show that there are several factors like house insulation, use of old appliances, and specific activities that influence energy consumption. Moreover, we have found that households in these neighborhoods are willing to volunteer their homes as testbeds for collecting contextual data and are primarily incentivized by reduction in their electricity bill.