One of the rapidly emerging research topics in synthetic biology focuses on how genetic modules are affected by their context, a fundamental challenge in the modular design of large-scale genetic systems. A major source of such context-dependence is due to the sharing of scarce common cellular resources, such as transcriptional and translational machinery. Since toggle switches are one of the fundamental building blocks of genetic systems, in this paper we reveal how competition for shared resources affects the robustness of bistable toggle switches to noise. To this end, we study the mean transition time between the two stable equilibria by leveraging the Eyring-Kramers law, and show that it decreases due to the scarcity of shared resources, thus the system becomes less robust to noise. In order to achieve this, we define a quasi-potential function that allows us to formulate the problem as the overdamped motion of a Brownian particle in a potential field. In addition to revealing how various parameters affect the mean transition time, thus the robustness of the toggle switch, we also develop explicit design guidelines for creating toggle switches that are minimally affected by their context.