Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoff

Previous work [Jac23] has shown that DNNs with large depth L and L₂-regularization are biased towards learning low-dimensional representations of the inputs, which can be interpreted as minimizing a notion of rank R⁽⁰⁾(f) of the learned function f, conjectured to be the Bottleneck rank. We compute finite depth corrections to this result, revealing a measure R⁽¹⁾ of regularity which bounds the pseudo-determinant of the Jacobian |Jf(x)|₊ and is subadditive under composition and addition. This formalizes a balance between learning low-dimensional representations and minimizing complexity/irregularity in the feature maps, allowing the network to learn the 'right' inner dimension. Finally, we prove the conjectured bottleneck structure in the learned features as L → ∞: for large depths, almost all hidden representations are approximately R⁽⁰⁾(f)-dimensional, and almost all weight matrices W_ℓ have R⁽⁰⁾(f) singular values close to 1 while the others are O(L^-1/2). Interestingly, the use of large learning rates is required to guarantee an order O(L) NTK which in turns guarantees infinite depth convergence of the representations of almost all layers.