Breakdown Point Analysis for Deep Neural Network Estimators Under Heavy-Tailed Noise
Abdullah Mohammed Rashid
Department of Accounting and Financial Control, College of Business Economics, Al-Nahrain University, Iraq.
Hadeel Fawzi Mohammed
Al-Kindy College of Medicine, University of Baghdad, Iraq.
Hasan Talib Hendi
Science of College, University of Baghdad, Iraq.
Receiving Date:
2026-02-28
Acceptance Date:
2026-04-05
Publication Date:
2026-04-25
Download PDF
http://doi.org/10.37648/ijrst.v16i02.005
Abstract
In this paper, we experimentally assess the breakdown point of deep neural network regression estimators trained in the presence of heavy-tailed label contamination. On the standard California Housing dataset, we randomly replace between 0% and 50% of the training labels with independent Cauchy noise (scale ? = 50). We consider Mean Squared Error (MSE) loss versus Huber loss (? = 1), training a 3-layer MLP at each corruption level for 100 epochs using Adam. Models are scored on a held-out clean test set using Mean Absolute Error (MAE). We observe that MSE trained networks experience catastrophic breakdown at as little as 5% corruption (tested in 5% increments), where test MAE grows by 3,221% over that of a model trained without label corruption. In stark contrast, Huber trained models are resilient to corruption across all levels of label contamination, growing by only 29% at 50% corruption. Specifically, at our maximal observed difference of 35% corruption, MSE incurs an error 340x that of Huber. This illustrates an empirical breakdown point of zero for MSE loss, and that Huber loss dramatically increases robustness of DNN regression estimators to adversarial heavy-tailed noise.
Keywords:
breakdown point; robust regression; Huber loss; deep neural networks; heavy-tailed noise; outlier robustness.
References
- Chen, J. (2024). Robust nonparametric regression based on deep ReLU neural networks. Statistics & Probability Letters, 208, 110046. https://doi.org/10.1016/j.spl.2024.110046
- Chen, S., Koehler, F., Moitra, A., & Yau, M. (2022). Online and distribution-free robustness: Regression and contextual bandits with Huber contamination. In Proceedings of the 62nd IEEE Annual Symposium on Foundations of Computer Science (FOCS) (pp. 684–695). IEEE. https://doi.org/10.1109/FOCS52979.2021.00070
- Chen, Y., Wang, Z., Liu, X., Li, H., & Zhang, J. (2025). A survey on learning from data with label noise via deep neural networks. Systems Science & Control Engineering, 13(1), Article 2488120. https://doi.org/10.1080/21642583.2025.2488120
- Fan, J., Gu, Y., & Zhou, W.-X. (2024). How do noise tails impact on deep ReLU networks? The Annals of Statistics, 52(4), 1845–1871. https://doi.org/10.1214/24-AOS2428
- Feng, Y., & Wu, Q. (2025). Understanding robust machine learning for nonparametric regression with heavy-tailed noise (arXiv preprint arXiv:2510.09888).
- Hampel, F. R. (1971). A general qualitative definition of robustness. The Annals of Mathematical Statistics, 42(6), 1887–1896. https://doi.org/10.1214/aoms/1177693054
- He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) (pp. 1026–1034). IEEE. https://doi.org/10.1109/ICCV.2015.123
- Huber, P. J. (1964). Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1), 73–101. https://doi.org/10.1214/aoms/1177703732
- Jiao, Y., Shen, G., Lin, Y., & Huang, J. (2023). Deep nonparametric regression on approximate manifolds: Nonasymptotic error bounds with polynomial prefactors. The Annals of Statistics, 51(2), 691–716. https://doi.org/10.1214/23-AOS2266
- Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1412.6980
- Liu, T., Xu, Z., & Ling, Q. (2023). Functional linear regression with Huber loss. Journal of Complexity, 74, 101696. https://doi.org/10.1016/j.jco.2022.101696
- Nguyen, T., & Oijen, R. (2024). Deep regression learning with optimal loss function. Journal of the American Statistical Association. https://doi.org/10.1080/01621459.2024.2412364
- Pace, R. K., & Barry, R. (1997). Sparse spatial autoregressions. Statistics & Probability Letters, 33(3), 291–297. https://doi.org/10.1016/S0167-7152(96)00140-X
- Paszke, A., et al. (2019). PyTorch: An imperative style, high-performance deep learning library. In H. Wallach et al. (Eds.), Advances in Neural Information Processing Systems (Vol. 32, pp. 8024–8035). Curran Associates.
- Pedregosa, F., et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.
- Rubio, A., & Dorronsoro, J. R. (2023). Robust losses in deep regression. In Hybrid Artificial Intelligent Systems (Lecture Notes in Computer Science, Vol. 14001, pp. 259–270). Springer. https://doi.org/10.1007/978-3-031-40725-3_22
- Schmidt-Hieber, J. (2020). Nonparametric regression using deep neural networks with ReLU activation function. The Annals of Statistics, 48(4), 1875–1897. https://doi.org/10.1214/19-AOS1875
- Shen, G., Jiao, Y., Lin, Y., & Huang, J. (2021). Robust nonparametric regression with deep neural networks (arXiv preprint arXiv:2107.10343).
- Song, H., Kim, M., Park, D., Shin, Y., & Lee, J.-G. (2023). Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 34(11), 8135–8153. https://doi.org/10.1109/TNNLS.2022.3152527
- Tyralis, H., Papacharalampous, G., Langousis, A., & Papalexiou, S. M. (2025). Deep Huber quantile regression networks. Neural Networks, 186, 107364. https://doi.org/10.1016/j.neunet.2025.107364
- Werner, T. (2024). Global quantitative robustness of regression feed-forward neural networks. Neural Computing and Applications, 36, 19967–19988. https://doi.org/10.1007/s00521-024-10289-w
- Xu, H., An, Y., & Lu, J. (2022). Robust learning of Huber loss under weak conditional moment. Neurocomputing, 507, 191–201. https://doi.org/10.1016/j.neucom.2022.08.012
Back
