On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions - Inria EPFL
Preprints, Working Papers, ... Year : 2024

On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions

Abstract

We investigate the out-of-domain generalization of random feature (RF) models and Transformers. We first prove that in the `generalization on the unseen (GOTU)' setting, where training data is fully seen in some part of the domain but testing is made on another part, and for RF models in the small feature regime, the convergence takes place to interpolators of minimal degree as in the Boolean case (Abbe et al., 2023). We then consider the sparse target regime and explain how this regime relates to the small feature regime, but with a different regularization term that can alter the picture in the non-Boolean case. We show two different outcomes for the sparse regime with q-ary data tokens: (1) if the data is embedded with roots of unities, then a min-degree interpolator is learned like in the Boolean case for RF models, (2) if the data is not embedded as such, e.g., simply as integers, then RF models and Transformers may not learn minimal degree interpolators. This shows that the Boolean setting and its roots of unities generalization are special cases where the minimal degree interpolator offers a rare characterization of how learning takes place. For more general integer and real-valued settings, a more nuanced picture remains to be fully characterized.
Fichier principal
Vignette du fichier
ICML_2024_camera_ready-1.pdf (653.04 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04619375 , version 1 (20-06-2024)

Identifiers

  • HAL Id : hal-04619375 , version 1

Cite

Denys Pushkin, Raphaël Berthier, Emmanuel Abbe. On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions. 2024. ⟨hal-04619375⟩
123 View
24 Download

Share

More