Control Barrier Functions (CBFs) when paired with Quadratic Programming offer an efficient way to generate safety-critical controllers. In this paper, we utilize CBFs for guiding multiple robots to their goals while avoiding collisions with the environment and among themselves. However, in more complex scenarios, with many robots and non-convex obstacles, these approaches often fail to guide the robots towards their desired goals because there can be other stable and undesirable equilibrium points in the system other than the desired one (reaching the goal). The proposed approach in this paper mitigates this issue by including constraints in the formulation that force the robots to circulate the boundary of the obstacles as well as each other when in close proximity. This ensures that the system does not get stuck in an undesirable equilibrium. Simulation studies show the efficacy of the proposed approach for a multi-agent problem.