Integral formulas with distribution kernels for irreducible projections in L² of a nilmanifold

Let N be a simply connected nilpotent Lie group and Γ a discrete uniform subgroup. The authors consider irreducible representations σ in the spectrum of the quasi-regular representation N × L²(Γ/N) → L²(Γ→) which are induced from normal maximal subordinate subgroups M ⊆ N. The primary projection P_σ and all irreducible projections P ≤ P_σ are given by convolutions involving right Γ-invariant distributions D on Γ→, Pf(Γn) = D * f(Γn) = <D, n · f>all f ε{lunate} C^∞(Γ/N), where n · f(ζ) = f(ζ · n). Extending earlier work of Auslander and Brezin, and L. Richardson, the authors give explicit character formulas for the distributions, interpreting them as sums of characters on the torus T^κ = (Γ ∩ M) · [M, M]{minus 45 degree rule}M. By examining these structural formulas, they obtain fairly sharp estimates on the order of the distributions: if σ is associated with an orbit O ⊆ n^* and if V ⊆ n^* is the largest subspace which saturates θ in the sense that f ε{lunate} O ⇒ f + V ⊆ O. As a corollary they obtain Richardson's criterion for a projection to map C⁰(Γ→) into itself. The authors also resolve a conjecture of Brezin, proving a Zero-One law which says, among other things, that if the primary projection P_σ maps C^r(Γ→) into C⁰(Γ→), so do all irreducible projections P ≤ P_σ. This proof is based on a classical lemma on the extent to which integral points on a polynomial graph in Rⁿ lie in the coset ring of Zⁿ (the finitely additive Boolean algebra generated by cosets of subgroups in Zⁿ). This lemma may be useful in other investigations of nilmanifolds.