We propose a multi-task learning approach for estimating both the location and magnitude of damage occurring on an experimental bridge using acceleration signals collected from a passing vehicle. This is a low-cost and low-maintenance indirect structural health monitoring approach in which sensors on the vehicle are used to detect bridge damage. Recently, signal processing and machine learning approaches have been shown to perform well in achieving higher-level structural health monitoring objectives, such as damage localization and quantification. However, these methods not only lack robustness to measurement and model noises, but also require more physical insights. Guided by a theoretical formulation of a simple vehicle-bridge interaction system, our approach preserves the non-linearity of the trend of the acceleration signals as severity changes, and simultaneously localizes and quantifies the damage for minimizing uncertainties propagating from the location estimation. We evaluate our model on an experimental dataset. In the experiments, the damage is represented by a mass with gradually changing magnitude attached at different positions on the bridge. The results show that it can estimate locations of the damage with an accuracy of 0.08 m (3.30% of the total length of the bridge) and changes in severity level with an accuracy of 17.81 grams (8.9% of the maximum severity mass).