Tests of statistical significance and background estimation in gamma-ray air shower experiments

In this paper we discuss established methods of significance calculation for testing the existence of a signal in the presence of unknown background and point out the limits of their applicability. We then introduce a new self-consistent scheme for source detection and discuss some of its properties. The method overcomes weaknesses of those used previously and allows incorporating background anisotropies by vetoing existing localized sources and sinks on the sky and compensating for known large-scale anisotropies. By giving an example using the Milagro gamma-ray observatory data, we demonstrate how the method can be employed to relax the detector stability assumption. The new method is universal and can be used with any large field-of-view detector, in which the object of investigation, steady or transient, point or extended, traverses its field of view.