跳至正文

子课题 C：深度学习泛化之谜

目录

日期：2026-06-23 任务：总课题「归纳问题与『无免费午餐』——为什么机器学习能泛化」之子课题 C 前期文献核查 状态：初步文献笔记（非最终报告） 标签：深度学习泛化 / 随机标签 / 双下降 / 良性过拟合 / grokking / 隐式偏置 / 彩票_ticket / 神经正切核

1. 说明与去重

本笔记覆盖子课题 C 的七个分支：

  1. Zhang et al. (2017) Rethinking Generalization；
  2. 双下降（double descent）；
  3. 良性过拟合（benign overfitting）；
  4. Grokking；
  5. 隐式偏置 / 隐式正则化；
  6. 彩票_ticket 假设；
  7. 神经正切核（NTK）与 lazy/rich 训练。

旧线已覆盖、本处仅接口：

  • 2026-05-05-overfitting-mind-learning-meditation-ai.md：已详细覆盖双下降、benign/tempered/catastrophic overfitting、grokking 玻璃弛豫 vs 一阶相变、隐式偏置；
  • 2026-06-05-neural-scaling-laws-first-principles.md：已覆盖 NTK、核谱幂律、lazy→feature learning 与标度律接口；
  • 2026-06-23-criticism-redteam-nfl-implicit-bias-world-structure-bayesian-priors.md：子课题 E 已收录 Zhang 2017、Dinh 2017、Geiping 2021 等隐式偏置红队文献。

因此本笔记避免重复展开哲学/红队论证，重点提供可验证的原始关键句、数字、URL/DOI 与证据标签，为最终裁决做文献底板。

2. 关键来源

2.1 Zhang et al. (2017) — 深度学习需要重新思考泛化

来源：Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2017). “Understanding deep learning requires rethinking generalization.” International Conference on Learning Representations (ICLR). URL：https://arxiv.org/abs/1611.03530 OpenReview：https://openreview.net/forum?id=Sy8gdB9xx

中文摘要：通过系统实验表明，SOTA 卷积网络能轻易拟合随机标签和随机像素输入；显式正则化（weight decay、dropout、data augmentation）对这一现象只有定性影响，说明传统复杂度/正则化框架不足以解释深度学习的泛化。

原始关键句/数字：

“Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization, and occurs even if we replace the true images by completely unstructured random noise.”

“Theorem 1. There exists a two-layer neural network with ReLU activations and 2n + d weights that can represent any function on a sample of size n in d dimensions.”

数学/实验边界：

  • 实验：CIFAR-10、ImageNet 等图像分类任务；
  • 理论构造：两层 ReLU，参数数 p = 2n + d 即可对任意 n 个 d 维样本实现任意标签；
  • 结论边界：说明「模型族复杂度 + 显式正则化」不是泛化的充分解释，但不直接刻画 SGD 的隐式偏置具体偏好哪类函数。

证据标签：[文献较稳]

2.2 Belkin et al. (2019) — 双下降曲线

来源：Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). “Reconciling modern machine-learning practice and the classical bias–variance trade-off.” Proceedings of the National Academy of Sciences, 116(32), 15849–15854. DOI：https://doi.org/10.1073/pnas.1903070116 arXiv：https://arxiv.org/abs/1812.11118

中文摘要：提出「双下降」风险曲线，将经典 U 形 bias-variance 曲线延伸到插值阈值之外：模型容量超过插值点后，测试风险再次下降，从而调和现代实践「越大越好」与经典理论的矛盾。

原始关键句：

“The main finding of this work is a pattern for how performance on unseen data depends on model capacity and the mechanism underlying its emergence. This dependence, empirically witnessed with important model classes including neural networks and a range of datasets, is summarized in the ‘double descent’ risk curve shown in Figure 1(b). The curve subsumes the classical U-shaped risk curve from Figure 1(a) by extending it beyond the point of interpolation.”

数学/实验边界：

  • 展示决策树、随机特征、两层神经网络、MNIST/CIFAR-10 上的双下降；
  • 未给出统一严格证明，但提出机制猜想。

证据标签：[文献较稳]

2.3 Nakkiran et al. (2020/2021) — 深度双下降：有效模型复杂度

来源：Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2020). “Deep double descent: Where bigger models and more data hurt.” ICLR 2020; 扩展版 Journal of Statistical Mechanics: Theory and Experiment, 2021(12), 124003. URL：https://arxiv.org/abs/1912.02292 OpenReview (ICLR 2020)：https://openreview.net/forum?id=B1g5sA4twr JSTAT DOI：https://doi.org/10.1088/1742-5468/ac3a74

中文摘要：双下降不仅随模型容量出现，还随训练轮数（epoch-wise double descent）和样本量（sample non-monotonicity）出现；提出「有效模型复杂度（EMC）」统一三类现象，并发现增加样本量在临界区附近反而可能损害测试性能。

原始关键句/数字：

“We show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure.”

“Definition 1 (Effective Model Complexity) The Effective Model Complexity (EMC) of a training procedure T, with respect to distribution D and parameter ε > 0, is defined as the maximum number of samples n on which T achieves on average ≈ 0 training error.”

“For example, Figure 3 demonstrates cases in which increasing the number of samples by a factor of 4.5 results in worse test performance.”

数学/实验边界：

  • EMC 是经验性定义，依赖 ε；
  • 双下降在 ResNet、Transformer 等多种架构上观察到；
  • 「更多数据更差」是特定模型尺寸/任务下的现象，非普遍规律。

证据标签：[文献较稳]

2.4 Bartlett et al. (2020) — 线性回归中的良性过拟合

来源：Bartlett, P. L., Long, P. M., Lugosi, G., & Tsigler, A. (2020). “Benign overfitting in linear regression.” Proceedings of the National Academy of Sciences, 117(48), 30063–30070. DOI：https://doi.org/10.1073/pnas.1907378117 arXiv：https://arxiv.org/abs/1906.11300

中文摘要：在高维线性回归中，最小范数插值解的泛化误差由数据协方差 Σ 的两个「有效秩」r_k 与 R_k 决定；当低方差方向足够多（过参数化）时，即使完美拟合含噪训练数据，预测误差仍可接近最优。

原始关键句/数字：

“We give a characterization of linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of the effective rank of the data covariance. It shows that overparameterization is essential for benign overfitting in this setting: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size.”

“Theorem 4. For any σ_x there are b, c, c_1 > 1 for which the following holds. Consider a linear regression problem from Definition 1. Define k^* = min{k ≥ 0 : r_k(Σ) ≥ bn}, where the minimum of the empty set is defined as ∞. Suppose δ < 1 with log(1/δ) < n/c. If k^* ≥ n/c_1, then ER(θ̂) ≥ σ^2/c. Otherwise, R(θ̂) ≤ c ||θ^||^2 ||Σ|| max{ k^/n, n/R_{k^*}(Σ), … } + … with probability at least 1 − δ.”

数学定理边界：

  • 模型：高斯线性模型 y = x^T θ^* + η，最小范数插值 θ̂ = X^T (XX^T)^{-1} y；
  • 关键量：r_k(Σ) = (Σ_{i>k} λ_i) / λ_{k+1}，R_k(Σ) = (Σ_{i>k} λ_i)^2 / Σ_{i>k} λ_i^2；
  • 条件：低方差方向的有效秩 R_{k^*}(Σ) 必须远大于样本量 n；
  • 边界：结果针对线性回归，不能直接外推到非凸深度网络。

证据标签：[文献较稳]

2.5 Mallinar et al. (2022) — 过拟合的三分类法

来源：Mallinar, N., Simon, J. B., Pandit, P., Abedsoltan, A., Belkin, M., & Nakkiran, P. (2022). “Benign, tempered, or catastrophic: A taxonomy of overfitting.” NeurIPS 2022. URL：https://arxiv.org/abs/2207.06569

中文摘要：提出过拟合的三分法：良性（渐近 Bayes 最优）、 tempered（渐近风险有限但高于 Bayes）、灾难性（风险发散）。证明核回归在幂律谱下是 tempered，并发现实际训练到插值的深度网络多为 tempered 而非 benign。

原始关键句/数字：

“We first explore this phenomenon in the context of kernel (ridge) regression (KR) by obtaining conditions on the ridge parameter and kernel eigenspectrum under which KR exhibits each of the three behaviors. We find that kernels with powerlaw spectra, including Laplace kernels and ReLU neural tangent kernels, exhibit tempered overfitting. We then empirically study deep neural networks through the lens of our taxonomy, and find that those trained to interpolation are tempered, while those stopped early are benign.”

“Theorem 3.1 (KR trichotomy). … (b) If δ = 0 and λ_i = i^{−α} for some α > 1, then lim_{n→∞} E_n = α σ^2.”

“Table 1: Our taxonomy of (over)fitting. Benign: lim R_n = R^. Tempered: lim R_n ∈ (R^, ∞). Catastrophic: lim R_n = ∞ (regression) or lim R_n = 1 − 1/K (classification).”

数学/实验边界：

  • 定理针对核岭回归，δ 为 ridge 参数；
  • 幂律谱 λ_i ∼ i^{−α} 的 ridgeless KR 渐近风险为 ασ²；
  • DNN 实验为经验观察，未严格证明。

证据标签：[文献较稳]

2.6 Power et al. (2022) — Grokking：过拟合后的突然泛化

来源：Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. (2022). “Grokking: Generalization beyond overfitting on small algorithmic datasets.” arXiv:2201.02177; ICLR 2022 MATH-AI Workshop. URL：https://arxiv.org/abs/2201.02177

中文摘要：在小规模算法数据集（模运算表）上训练 Transformer，网络会先完美记忆训练集但验证准确率接近随机，经过远多于记忆所需步数后突然跳到完美泛化；数据越少，泛化所需优化步数增长越快。

原始关键句/数字：

“In some situations we show that neural networks learn through a process of ‘grokking’ a pattern in the data, improving generalization performance from random chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting.”

“There we see that validation accuracy starts increasing beyond chance level only after 1000 times more optimization steps than are required for training accuracy to get close to optimal.”

“In the vicinity of 25-30% of data, a decrease of 1% of training data leads to an increase of 40-50% in median time to generalization.”

实验边界：

  • 任务：模 97 除法等二进制运算表；
  • 架构：小型 Transformer；
  • 现象对数据比例、正则化（尤其是 weight decay）敏感。

证据标签：[文献较稳]

2.7 Rubin et al. (2024) — Grokking 作为两层网络的一阶相变

来源：Rubin, N., Seroussi, I., & Ringel, Z. (2024). “Grokking as a first order phase transition in two layer networks.” ICLR 2024. URL：https://arxiv.org/abs/2310.03789 OpenReview：https://openreview.net/forum?id=3ROGsTX3IR

中文摘要：用自适应核（adaptive kernel）方法分析两层教师-学生网络与模加任务，将 grokking 映射到一阶相变：grokking 后网络处于「混合相」，其内部表示与 GP/ lazy 极限尖锐不同。

原始关键句：

“We show that after Grokking, the state of the DNN is analogous to the mixed phase following a first-order phase transition. In this mixed phase, the DNN generates useful internal representations of the teacher that are sharply distinct from those before the transition.”

“For both our models, we simplify the task of learning high-dimensional representations to solving a non-linear equation involving either two (cubic teacher) or one (modular addition teacher) variables. Moreover, for the latter, we determine the location of the phase transition analytically.”

数学/实验边界：

  • 模型：两层非线性网络（Erf 激活）与模加任务；
  • 方法：平均场 / 自适应核；
  • 边界：结果在特定教师-学生设置下解析可解，对一般 Transformer 的适用性为外推。

证据标签：[理论整合]

2.8 Kumar et al. (2024) — Grokking 作为 lazy → rich 转换

来源：Kumar, T., Bordelon, B., Gershman, S. J., & Pehlevan, C. (2024). “Grokking as the transition from lazy to rich training dynamics.” ICLR 2024. URL：https://arxiv.org/abs/2310.06110 OpenReview：https://openreview.net/forum?id=vt5mnLVIVo

中文摘要：提出 grokking 可由「lazy 训练（用初始特征做核回归）→ rich 训练（特征学习）」的延迟转换解释；在多项式回归的两层 MLP 上，即使没有正则化也出现 grokking，且权重范数在转换后上升（与「weight norm 下降至 Goldilocks 区」的解释相反）。

原始关键句：

“We propose that the grokking phenomenon … can arise due to a neural network transitioning from lazy training dynamics to a rich, feature learning regime.”

“We find that the key determinants of grokking are the rate of feature learning—which can be controlled precisely by parameters that scale the network output—and the alignment of the initial features with the target function y(x).”

“The fact that parameter weight norm increases (dashed orange) cannot be explained by any existing theories of grokking.”

数学/实验边界：

  • 设置：两层 MLP + 多项式回归，无正则化；
  • 关键机制：初始 NTK 与目标不对齐 + 数据集大小适中 + 网络起始于 lazy 区；
  • 边界：对更复杂架构（Transformer）的迁移为经验观察。

证据标签：[文献较稳]

2.9 Lyu et al. (2024) — 早期/晚期隐式偏置二分导致 Grokking

来源：Lyu, K., Jin, J., Li, Z., Du, S. S., Lee, J. D., & Hu, W. (2024). “Dichotomy of early and late phase implicit biases can provably induce grokking.” ICLR 2024. URL：https://arxiv.org/abs/2311.18817

中文摘要：在齐次神经网络（large initialization + small weight decay）中，训练早期被隐式偏置推向核预测器（记忆解），晚期被推向最小范数/最大间隔解（泛化解），两者之间的竞争导致 grokking 的尖锐转换；并构造出「misgrokking」反例。

原始关键句：

“Specifically, when training homogeneous neural nets with large initialization and small weight decay on both classification and regression tasks, we prove that the training process gets trapped at a solution corresponding to a kernel predictor for a long time, and then a very sharp transition to min-norm/max-margin predictors occurs, leading to a dramatic change in test accuracy.”

“This implicit bias result holds for homogeneous neural nets, a broad class of neural nets that include commonly used MLPs and CNNs with homogeneous activation.”

数学/实验边界：

  • 证明对象：齐次网络、大初始化、小 weight decay；
  • 给出 diagonal linear nets 与低秩矩阵补全上的严格证明；
  • 对一般深度网络的精确迁移仍需验证。

证据标签：[文献较稳]

2.10 Zhang et al. (2025) — Grokking 作为计算玻璃弛豫

来源：Zhang, X., Shang, Y., Yang, E., & Zhang, G. (2025). “Is grokking a computational glass relaxation?” NeurIPS 2025 (Spotlight). URL：https://arxiv.org/abs/2505.11411 NeurIPS poster：https://neurips.cc/virtual/2025/poster/117824

中文摘要：把 grokking 重新框架为「计算玻璃弛豫」：记忆阶段对应液体快速淬火成非平衡玻璃，泛化阶段对应向高熵稳定构型的缓慢弛豫；实验表明 memorization→generalization 转换没有熵势垒，挑战了 Rubin 等人的一阶相变解释。

原始关键句：

“Here we propose an interpretation for grokking by framing it as a computational glass relaxation: viewing NNs as a physical system where parameters are the degrees of freedom and train loss is the system energy, we find memorization process resembles a rapid cooling of liquid into non-equilibrium glassy state at low temperature and the later generalization is like a slow relaxation towards a more stable configuration.”

“Our experiments in transformers on arithmetic tasks suggests that there is NO entropy barrier in the memorization-to-generalization transition of grokking, challenging previous theory that defines grokking as a first-order phase transition.”

“This provides strictly-defined counterexamples to theory attributing grokking solely to weight norm evolution towards the Goldilocks zone.”

边界：

  • 与 Rubin 2024 的「一阶相变」解释直接冲突；
  • 高熵优势的定量刻画与 WanD 优化器为探索性工作。

证据标签：[理论整合]

2.11 Neyshabur et al. (2015) — 神经网络的范数容量控制

来源：Neyshabur, B., Tomioka, R., & Srebro, N. (2015). “Norm-based capacity control in neural networks.” COLT 2015. URL：http://proceedings.mlr.press/v40/Neyshabur15.html PDF：http://proceedings.mlr.press/v40/Neyshabur15.pdf

中文摘要：研究以 group-norm 约束神经网络的容量：per-unit ℓ1 正则可实现 size-independent 容量控制；overall ℓp 正则仅在 p ≤ 2 时可 size-independent；任何情况下深度都以指数方式进入容量界。

原始关键句：

“A central question we ask is: can we bound the capacity of feed-forward network in terms of norm-based regularization alone, without relying on network size and even if the network size (number of nodes or edges) is unbounded or infinite?”

“In terms of capacity control, we show that per-unit regularization allows size-independent capacity-control only with a per-unit ℓ1-norm, and that overall ℓp regularization allows for size-independent capacity control only when p ≤ 2, even if the depth is bounded. In any case, even if we bound the sum of all magnitudes in the system, we show that an exponential dependence on the depth is unavoidable.”

数学边界：

  • 网络：前馈 ReLU/hinge 网络；
  • 关键工具：非负齐次性带来的权重缩放不变性；
  • 结论：范数控制容量在深度上指数恶化。

证据标签：[文献较稳]

2.12 Soudry et al. (2018) — 可分数据上梯度下降的隐式偏置

来源：Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., & Srebro, N. (2018). “The implicit bias of gradient descent on separable data.” Journal of Machine Learning Research, 19(70), 1–57. URL：https://www.jmlr.org/papers/v19/18-188.html PDF：https://www.jmlr.org/papers/volume19/18-188/18-188.pdf arXiv：https://arxiv.org/abs/1710.10345

中文摘要：在可分数据上，无正则化逻辑回归配合梯度下降，参数方向收敛到 ℓ2 最大间隔分类器（硬间隔 SVM）；收敛速度极慢，仅 O(1/log t)。

原始关键句/数字：

“In this paper, we show that even without any explicit regularization, for all linearly separable datasets, when minimizing logistic regression problems using gradient descent, we have that w(t)/||w(t)|| converges to the L2 maximum margin separator, i.e. to the solution of the hard margin SVM for homogeneous linear predictors.”

“Theorem 3. … the gradient descent iterates … will behave as: w(t) = ŵ log t + ρ(t), where ŵ is the L2 max margin vector (the solution to the hard margin SVM): ŵ = argmin ||w||_2 s.t. w^T x_n ≥ 1.”

“…the distance to the max-margin predictor decreasing only as O(1/log(t)).”

数学定理边界：

  • 数据：线性可分；
  • 损失：logistic / exp-loss 等具有紧指数尾的下降损失；
  • 步长：η < 2β^{−1} σ_max^{−2}(X)；
  • 结论：参数方向收敛到 max-margin，但范数发散，速度对数慢。

证据标签：[文献较稳]

2.13 Keskar et al. (2017) — 大批量训练与尖锐极小值

来源：Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., & Tang, P. T. P. (2017). “On large-batch training for deep learning: Generalization gap and sharp minima.” ICLR 2017. URL：https://arxiv.org/abs/1609.04836

中文摘要：大批量训练倾向于收敛到损失函数的尖锐极小值，而小批量 SGD 由于梯度估计噪声收敛到平坦极小值；尖锐极小值在训练与测试函数上的 Hessian 有较大正特征值，泛化更差。

原始关键句：

“The lack of generalization ability is due to the fact that large-batch methods tend to converge to sharp minimizers of the training function. These minimizers are characterized by a significant number of large positive eigenvalues in ∇² f(x), and tend to generalize less well. In contrast, small-batch methods converge to flat minimizers characterized by having numerous small eigenvalues of ∇² f(x).”

边界：

  • 为经验观察 + 启发式解释；
  • Dinh et al. (2017) 指出 flatness 度量在 ReLU 网络对称性下不不变。

证据标签：[理论整合]

2.14 Dinh et al. (2017) — 尖锐极小值也能泛化

来源：Dinh, L., Pascanu, R., Bengio, S., & Bengio, Y. (2017). “Sharp minima can generalize for deep nets.” ICML 2017. URL：https://arxiv.org/abs/1703.04933

中文摘要：对 ReLU 网络，利用非负齐次性/神经元置换等对称性，可以把一个平坦极小值重参数化为任意尖锐的等价极小值；因此常见 flatness 定义不能直接用于解释泛化。

原始关键句：

“This paper argues that most notions of flatness are problematic for deep models and can not be directly applied to explain generalization. Specifically, when focusing on deep networks with rectifier units, we can exploit the particular geometry of parameter space induced by the inherent symmetries that these architectures exhibit to build equivalent models corresponding to arbitrarily sharper minima. Furthermore, if we allow to reparametrize a function, the geometry of its parameters can change drastically without affecting its generalization properties.”

证据标签：[文献较稳]

2.15 Frankle & Carbin (2019) — 彩票_ticket 假设

来源：Frankle, J., & Carbin, M. (2019). “The lottery ticket hypothesis: Finding sparse, trainable neural networks.” ICLR 2019. URL：https://arxiv.org/abs/1803.03635

中文摘要：稠密随机初始化网络中包含「中奖子网络」：当单独训练时，这些小得多的子网络能在相同迭代次数内达到与原网络相当的测试精度；中奖_ticket 的关键在于保留训练后的稀疏 mask 并回退到初始化权重。

原始关键句/数字：

“The Lottery Ticket Hypothesis. A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.”

“We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.”

实验边界：

  • 数据集：MNIST、CIFAR-10；
  • 架构：全连接、卷积网络；
  • 方法：迭代幅度剪枝 + 权重回退到初始化；
  • 对极深网络/大学习率需要 warmup。

证据标签：[文献较稳]

2.16 Jacot et al. (2018) — 神经正切核

来源：Jacot, A., Gabriel, F., & Hongler, C. (2018). “Neural tangent kernel: Convergence and generalization in neural networks.” NeurIPS 2018. URL：https://arxiv.org/abs/1806.07572

中文摘要：在无限宽度极限下，神经网络的训练动态可由一个确定性核——神经正切核（NTK）——描述；网络函数在函数空间中沿 NTK 梯度下降收敛，最小二乘回归时满足线性 ODE，收敛速度由 NTK 主成分决定。

原始关键句/数字：

“We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function f_θ … follows the kernel gradient of the functional cost … w.r.t. a new kernel: the Neural Tangent Kernel (NTK).”

“While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training.”

“For a least-squares regression loss, the network function f_θ follows a linear differential equation in the infinite-width limit, and the eigenfunctions of the Jacobian are the kernel principal components of the input data.”

数学定理边界：

  • 网络：全连接网络，Lipschitz、二阶可导非线性；
  • 参数化：含 1/√n_ℓ 缩放与偏置因子 β；
  • 极限：隐藏层宽度趋于无穷；
  • 正定：数据在单位球面上、非多项式激活；
  • 边界：有限宽度网络只在训练早期近似 NTK，特征学习对应离开 lazy 区。

证据标签：[文献较稳]

3. 主要结论

3.1 深度学习泛化确实突破了经典图景

  • Zhang 2017 的随机标签实验是硬事实：CNN+SGD 能拟合任意标签与随机噪声，说明「模型族复杂度 + 显式正则化」不足以解释泛化 [文献较稳]。
  • Belkin 2019 / Nakkiran 2020 表明，越过插值阈值后测试风险再次下降，且该现象可随模型容量、训练轮数、样本量出现；EMC 提供了统一描述框架 [文献较稳]。
  • Bartlett 2020 给出线性回归中良性过拟合的精确条件：低方差方向的有效秩必须显著超过样本量 [文献较稳]。
  • Mallinar 2022 进一步指出，实际训练到插值的深度网络多处于 tempered 区，而非 benign 区 [文献较稳]。

3.2 Grokking 是理解隐式偏置的时间窗口

  • Power 2022 发现 grokking：训练准确率饱和后，验证准确率从随机水平突然跳到完美泛化 [文献较稳]。
  • Rubin 2024 将其映射为一阶相变；Kumar 2024 将其解释为 lazy→rich 的特征学习转换；Lyu 2024 用早期/晚期隐式偏置二分严格证明 grokking 可在简单网络上出现 [文献较稳 / 理论整合]。
  • Zhang 2025 以「计算玻璃弛豫」挑战一阶相变解释，提出无熵势垒、高熵优势，并给出反例说明 grokking 不 sole 归因于 weight norm 降至 Goldilocks 区 [理论整合]。
  • 当前 grokking 的物理/几何解释并存且部分冲突，尚未有统一理论。

3.3 隐式偏置是核心但尚未闭合

  • Soudry 2018 证明线性可分逻辑回归中，GD 的方向收敛到 ℓ2 max-margin SVM，但速度极慢 O(1/log t) [文献较稳]。
  • Neyshabur 2015 显示范数容量控制可行，但深度导致指数依赖 [文献较稳]。
  • Keskar 2017 提出小批量 SGD → 平坦极小值 → 好泛化，但 Dinh 2017 指出 ReLU 网络对称性破坏常见 flatness 度量的解释力 [文献较稳]。
  • 隐式偏置的真实机制仍涉及 SGD 噪声、架构对称性、数据结构与损失景观的耦合，不能简单归结为单一因素。

3.4 Lottery Ticket 与 NTK 是两个边界条件

  • Frankle & Carbin 2019：中奖子网络的存在说明初始化与稀疏结构共同决定可训练性，但 LTH 在极大规模预训练中的适用范围仍受限 [文献较稳]。
  • Jacot 2018：无限宽度网络在 lazy 区等价于核方法，解释了「为何大模型早期像核回归」；但真实深度网络的特征学习发生在离开 lazy 区之后 [文献较稳]。

3.5 证据状态汇总

论断 证据强度 关键来源
DNN 能拟合随机标签 [文献较稳] Zhang 2017
双下降/EMC 是经验规律 [文献较稳] Belkin 2019, Nakkiran 2020
线性回归良性过拟合有精确条件 [文献较稳] Bartlett 2020
实际 DNN 多为 tempered overfitting [文献较稳] Mallinar 2022
Grokking 由 lazy→rich / 隐式偏置二分导致 [文献较稳] Kumar 2024, Lyu 2024
Grokking 是一阶相变 [理论整合] Rubin 2024
Grokking 是玻璃弛豫 [理论整合] Zhang 2025
平坦极小值解释泛化 [仍不确定/弱] Keskar 2017 vs Dinh 2017
隐式偏置 = 单一机制 [仍不确定/弱] 多源冲突

4. 与母裁决的关系

母裁决雏形：「归纳问题哲学上仍硬；NFL/PAC/VC/Solomonoff 形式定理真而解释窄；深度学习泛化突破经典图景但机制未闭合；『世界简单/AGI 必然』是过度外推；真正答案是形式约束 + 隐式偏置 + 世界结构三足鼎立。」

本子课题 C 为母裁决提供以下支撑：

  1. 「突破经典图景」部分：Zhang 2017、Belkin 2019、Nakkiran 2020、Bartlett 2020、Mallinar 2022 共同构成经验与形式证据，说明「偏差-方差权衡」「插值即过拟合」的经典教条在过参数化、隐式偏置、特定数据谱结构下失效。
  1. 「机制未闭合」部分：
  • Grokking 的相变/玻璃弛豫/隐式偏置二分三种解释并存冲突；
  • 隐式偏置从 max-margin（Soudry 2018）、范数控制（Neyshabur 2015）到 flatness（Keskar 2017）再到对称性破坏（Dinh 2017），没有单一赢家；
  • NTK 解释无限宽度 lazy 区，但真实网络在 rich 区的特征学习仍缺乏普适理论。
  1. 「隐式偏置」是三足之一：本笔记确认的隐式偏置文献（Soudry 2018、Kumar 2024、Lyu 2024）支持「优化算法本身选择解」这一支柱，但不支持「隐式偏置 alone 解释一切」的过度简化。
  1. 与世界结构接口：Bartlett 2020 和 Mallinar 2022 显示数据协方差/核谱结构决定过拟合是 benign、temed 还是 catastrophic，说明「世界结构」支柱不可或缺。
  1. 与 NFL 接口：Zhang 2017 说明 DNN 的表达能力足以破坏任何仅依赖模型族的泛化界；NFL（子课题 A/E）与此兼容：NFL 在说「没有统一偏置则平均无差别」，而深度学习正是在特定数据结构与隐式偏置下找到了有效偏置。

5. 诚实缺口与待核点

5.1 全文未直接细读的文献

  • Nakkiran 2021 JSTAT 扩展版：已读 ICLR 2020 版，JSTAT 版的额外理论细节未逐字核对。
  • Rubin 2024 全文：仅读摘要与引言/主要结果片段，平均场方程细节未核对。
  • Kumar 2024 与 Lyu 2024 的定理完整证明：已读摘要、引言与主要定理陈述，完整证明未逐字跟踪。
  • Zhang 2025 玻璃弛豫全文：已读摘要与引言，WanD 优化器与熵采样细节未细读。
  • Neyshabur 2017 博士论文《Implicit Regularization in Deep Learning》：仅读摘要，全文为博士论文，未系统阅读。
  • Jacot 2018 全文证明：已读摘要、引言、主要定理陈述，完整 NTK 收敛证明未逐字跟踪。

5.2 数学边界待进一步核实

  • Bartlett 2020 中有效秩 r_k、R_k 的定义与定理 4 的上界常数 c, b, c_1 的具体依赖关系；
  • Mallinar 2022 定理 3.1 的「universality」假设具体是什么；
  • Lyu 2024 中「misgrokking」构造的完整条件；
  • Soudry 2018 中 stepsize 条件 η < 2β^{−1} σ_max^{−2}(X) 与指数尾假设的精确表述。

5.3 经验外推的诚实缺口

  • Grokking 的 toy 任务（模运算、多项式回归）与真实语言/视觉任务的迁移关系未建立；
  • Lottery Ticket 假设在 GPT-scale 预训练中的有效性尚无系统证据；
  • NTK 理论对 finite-width、finite-time、非平方损失的真实 Transformer 仅近似成立。

5.4 与旧线重复/接口的缺口

  • 子课题 C 与 2026-05-05-overfitting-mind-learning-meditation-ai.md 在双下降、grokking、隐式偏置上有大面积重叠，最终裁决需明确划分「本笔记补充」与「旧线继承」；
  • 与 2026-06-05-neural-scaling-laws-first-principles.md 在 NTK/核谱/幂律接口上需统一术语与结论。

6. 关联笔记

  • 2026-05-05-overfitting-mind-learning-meditation-ai.md
  • 2026-05-25-overfitting-double-descent-grokking-addendum.md
  • 2026-06-05-neural-scaling-laws-first-principles.md
  • 2026-06-23-criticism-redteam-nfl-implicit-bias-world-structure-bayesian-priors.md