PriMera Scientific Engineering (ISSN: 2834-2550)

Review Article

Volume 7 Issue 3

The Properties of the Least Squares Estimate in Regression Models with an Increasing Number of Unknowns Parameters

Asaf Hajiyev* and Jiuping Xu

September 03, 2025

View PDF

Abstract

Regression models with an increasing number of unknown parameters and different and unknown values of variances of random errors of observations are considered. The specificity of such regression models is that there is no more than one response at each observation point, which does not allow estimation of the variances of random errors of observations. For the calculation of the l.s.e, the iteration process has been constructed. It is shown that the l.s.e are unbiased and consistent. Using these results, the approach for the construction of a confidence band for the unknown function in regression models is suggested.

Keywords: regression models with an increasing number of unknown parameters; least squares estimator; Gauss-Newton approach; iterative process

References

  1. MK Acharjee and K Das. “Polynomial Spline Regression: Theory and Application”. Statistics > Methodology (2023).
  2. J Aldrich. “Fisher and Regression”. Statist. Sci 20.4 (2005): 401-417.
  3. A Belloni and V Chernozhukov. “1-penalized quantile regression in high- dimensional sparse models”. The Annals of Statistics 39 (2011): 82-130.
  4. RA Bellman. “Introduction to matrix analysis”. NY, McGraw-Hill (1962).
  5. PL Davies. “Aspects of robust linear regression”. Ann. Statist 21 (1993): 1843-1899.
  6. E Demidenko. “Mixed Models”. Theory and Applications with R. Wiley Series in Probability and Statistics (1981).
  7. LY Deng and CFJ Wu. “Estimation of variance of regression estimator”. J.American Statist. Assoc 82 (1987): 568-576.
  8. H Drygas. “BLUE in linear regression models”. Probab.Mathem. Statist.. Fasc 2 (1985): 177-186.
  9. H Drygas. “Weak and strong consistency of least squares estimators in linear models”. Z. Probability theory. Administrative (1976): 119- 128.
  10. L Fahrmeir., et al. “Regression”. Models, Methods and Applications. Springer (2021).
  11. AH Hajiyev. “Regression Models with Increasing Numbers of Unknown Parameters”. In: International Encyclopedia. Statist. Sciences, Springer (2011): 1208-1211.
  12. AH Hajiyev and J Xu. “Gauss-Newton method for a nonlinear regression model with an increasing number of unknown parameters”. Azerbaijan J. Mathematics 15 (2025): 2.
  13. PJ Huber. “Data analysis. What can be learned from the past 50 years”. Wiley. Probab.Statist (2011).
  14. VI Krylov, VV Bobkov and PI Monastirski. “Theory of computing methods”. Minsk, Nauka I Tekhnika (1985). (in Russian).
  15. A-M Legendre. “On Least Squares”. Translated from the French by Professor Henry A Ruger and Professor Helen M Walker, Teachers College, Columbia University, New York City (1805).
  16. CR Rao. “Linear Statistical Inference and its Applications”. John Wiley Series in Probability and Statistics.
  17. Rao C.R. “Estimation of heteroscedastic variances in linear regression”. J. American Statistician. Association 65 (1970): 161-172.
  18. PJ Rousseeuw and A Leroy. “Robust regression and outlier detection”. Wiley (1987).
  19. GF Seber. “Linear Regression Analysis”. NY, Wiley Ser. Probab. and Statistics (2003).
  20. O Sheinin. “Gauss and the method of least squares”. Yearbooks of Economics and Statistics (1999).
  21. AN Shiryaev. “Probability”. Springer-Verlag, New York-Berlin--Heidelberg (2013).
  22. S Stigler. “Gauss and the Invention of Least Squares”. Ann. Statist 9.3 (1981): 465-474.
  23. Zhang Xiaoqin, Hao Hongxia and Liang Jiye. “A new nonparametric estimation method of the variance in a heteroscedastic model”. Hacettepe Journal of Mathematics and Statistics 44.1 (2015): 239-245.
  24. A Hajiyev and J Xu. “High-dimensional regression analysis and Artificial Intelligence”. Springer (2025).