To improve the broadband transduction capabilities of vibratory energy harvesters (VEHs) under random and non-stationary excitations, many researchers have resorted to purposefully introducing nonlinearities into the restoring force of the harvester. While performing this task, it is often very challenging to maintain a perfectly symmetric restoring force which usually yields a VEH with an asymmetric potential energy function. This paper investigates the influence of potential function asymmetries on the performance of nonlinear VEHs under white noise inputs. To that end, a quadratic nonlinearity is introduced into the restoring force of the harvester and its influence on the mean power for both mono-and bi-stable potentials is investigated. It is shown that, for VEHs with a mono-stable potential function, the mean output power increases with the degree of potential function asymmetry. On the other hand, for energy harvesters with a bi-stable potential function, asymmetries in the restoring force appear to worsen performance especially for low to moderate noise intensities. When the noise intensity becomes sufficiently large, the influence of the potential function's asymmetry on the mean power diminishes. Results also reveal that a VEH with a symmetric bi-stable potential function produces higher mean power levels than the one with the most asymmetric mono-stable potential. As such, it is concluded that a VEH with a bi-stable symmetric potential is most desirable to improve performance under white noise.