For decades, researchers have investigated and developed technologies that support independent navigation for people who are blind. This has led to systems that primarily aid in detecting routes, landmarks, and building features. However, there has been relatively little inquiry regarding how technologies might support navigation around and in the presence of other people. What visual information, if any, do blind navigators wish they had about people on their path? To address this question, we surveyed 58 blind and low vision individuals and interviewed 10 blind individuals. We discovered our participants were interested in using visual information about others to increase their physical safety. For example, they wanted to know if a passerby was holding a weapon, if a presumed official had a proper uniform or badge, and how to describe visual aspects of a criminal to law enforcement. This paper presents one of the only reports documenting accessibility challenges related to physical safety posed by others, including how future assistive tools can empower individuals with disabilities to more actively increase their sense of safety. We call this emerging area Personal Safety Management and contribute a set of four broad subareas that deserve further exploration by researchers and designers working within the blind and broader disabilities communities.