|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| {{Regression bar}}
| | Many individuals have this habit of doing all stuff by themselves, irrespective of how important or simple they are! These folks won't allow others interfere in their matters. While this stance might work inside other regions of lifetime, it is very really not the number one way to reply when you need to fix the Windows registry. There are some jobs such as removing spywares, virus and also obsolete registry entries, that are best left to expert softwares. In this article I may tell we why it's important to fix Windows registry NOW!<br><br>If it is actually not as big of the problem as we think it is, it may probably be resolved easily by running a Startup Repair or by System Restore Utility. Again it can be as effortless because running an anti-virus check or cleaning the registry.<br><br>It doesn't matter whether you are not extremely clear regarding what rundll32.exe is. However remember that it plays an important character in preserving the stability of our computers plus the integrity of the program. When several software or hardware can not respond usually to a program operation, comes the rundll32 exe error, which may be caused by corrupted files or lost information in registry. Usually, error message may shows up at booting or the beginning of running a system.<br><br>There may be several factors why your computer may lose speed. Normal computer use, such as surfing the Internet will receive your running program in a condition where it has no choice but to slow down. The constant entering and deleting of temporary files that occur whenever you surf the Web leave our registries with thousands of false indicators inside the operating system's registry.<br><br>The final step is to make sure which you clean the registry of the computer. The "registry" is a large database that shops significant files, settings & choices, plus information. Windows reads the files it needs in order for it to run programs by this database. If the registry gets damaged, afflicted, or clogged up, then Windows are not able to correctly access the files it needs for it to load up programs. As this arises, difficulties and mistakes like the d3d9.dll error occur. To fix this and avoid future setbacks, we have to download and run a registry cleaning tool. The highly recommended software is the "Frontline [http://bestregistrycleanerfix.com registry cleaners]".<br><br>Although I constantly utilize the latest adaptation of browser, often different extensions plus plugins become the cause of errors with my browser and the program. The same is the story with my browser that was crashing frequently perhaps due to the Flash player error.<br><br>Most likely should you are experiencing a slow computer it could be a couple years older. You moreover may not have been told which whilst you use the computer everyday; there are certain points that it requires to continue running in its best performance. You furthermore could not even own any diagnostic tools that will get your PC running like new again. So never let which stop we from getting a program cleaned. With access to the internet you will find the tools that will assist we receive the system running like brand-new again.<br><br>Registry cleaners have been crafted to fix all broken files inside the program, allowing the computer to read any file it wants, whenever it wants. They work by scanning by the registry and checking each registry file. If the cleaner sees which it is corrupt, then it will substitute it automatically. |
| The method of '''least squares''' is a standard approach to the approximate solution of [[overdetermined system]]s, i.e., sets of equations in which there are more equations than unknowns. "Least squares" means that the overall solution minimizes the sum of the squares of the errors made in the results of every single equation.
| |
| | |
| The most important application is in [[curve fitting|data fitting]]. The best fit in the least-squares sense minimizes the sum of squared [[errors and residuals in statistics|residuals]], a residual being the difference between an observed value and the fitted value provided by a model. When the problem has substantial uncertainties in the [[independent variable]] (the 'x' variable), then simple regression and least squares methods have problems; in such cases, the methodology required for fitting [[errors-in-variables models]] may be considered instead of that for least squares.
| |
| | |
| Least squares problems fall into two categories: linear or [[ordinary least squares]] and [[non-linear least squares]], depending on whether or not the residuals are linear in all unknowns. The linear least-squares problem occurs in statistical [[regression analysis]]; it has a closed-form solution. A closed-form solution (or [[closed-form expression]]) is any formula that can be evaluated in a finite number of standard operations. The non-linear problem has no closed-form solution and is usually solved by iterative refinement; at each iteration the system is approximated by a linear one, and thus the core calculation is similar in both cases.
| |
| | |
| When the observations come from an [[exponential family]] and mild conditions are satisfied, least-squares estimates and [[Maximum likelihood|maximum-likelihood]] estimates are identical.<ref>{{cite doi|10.1080/01621459.1976.10481508}}</ref> The method of least squares can also be derived as a [[method of moments (statistics)|method of moments]] estimator.
| |
| | |
| The following discussion is mostly presented in terms of [[linear]] functions but the use of least-squares is valid and practical for more general families of functions. Also, by iteratively applying local [[quadratic approximation]] to the likelihood (through the [[Fisher information]]), the least-squares method may be used to fit a [[generalized linear model]].
| |
| | |
| For the topic of approximating a function by a sum of others using an objective function based on squared distances, see [[least squares (function approximation)]].
| |
| | |
| [[File:Linear least squares2.png|right|thumb|The result of fitting a set of data points with a quadratic function.]]
| |
| | |
| The least-squares method is usually credited to [[Carl Friedrich Gauss]] (1795),<ref name=brertscher>{{cite book|author = Bretscher, Otto|title = Linear Algebra With Applications, 3rd ed.|publisher = Prentice Hall|year = 1995|location = Upper Saddle River NJ}}</ref> but it was first published by [[Adrien-Marie Legendre]].
| |
| | |
| ==History==
| |
| | |
| ===Context===
| |
| The method of least squares grew out of the fields of [[astronomy]] and [[geodesy]] as scientists and mathematicians sought to provide solutions to the challenges of navigating the Earth's oceans during the [[Age of Exploration]]. The accurate description of the behavior of celestial bodies was the key to enabling ships to sail in open seas, where sailors could no longer rely on land sightings for navigation.
| |
| | |
| The method was the culmination of several advances that took place during the course of the eighteenth century:<ref name=stigler>{{cite book
| |
| | author = Stigler, Stephen M.
| |
| | title = The History of Statistics: The Measurement of Uncertainty Before 1900
| |
| | publisher = Belknap Press of Harvard University Press
| |
| | year = 1986
| |
| | location = Cambridge, MA
| |
| | isbn = 0-674-40340-1
| |
| }}</ref>
| |
| | |
| *The combination of different observations as being the best estimate of the true value; errors decrease with aggregation rather than increase, perhaps first expressed by [[Roger Cotes]] in 1722.
| |
| *The combination of different observations taken under the ''same'' conditions contrary to simply trying one's best to observe and record a single observation accurately. The approach was known as the method of averages. This approach was notably used by [[Tobias Mayer]] while studying the [[libration]]s of the moon in 1750, and by [[Pierre-Simon Laplace]] in his work in explaining the differences in motion of [[Jupiter]] and [[Saturn]] in 1788.
| |
| *The combination of different observations taken under ''different'' conditions. The method came to be known as the method of least absolute deviation. It was notably performed by [[Roger Joseph Boscovich]] in his work on the shape of the earth in 1757 and by [[Pierre-Simon Laplace]] for the same problem in 1799.
| |
| *The development of a criterion that can be evaluated to determine when the solution with the minimum error has been achieved. Laplace tried to specify a mathematical form of the probability density for the errors and define a method of estimation that minimizes the error of estimation. For this purpose, Laplace used a symmetric two sided exponential distribution we now call [[Laplace distribution]] to model the error distribution and used the sum of absolute deviation as error of estimation. He felt these to be the simplest assumptions he could make, and he had hoped to obtain the arithmetic mean as the best estimate. Instead, his estimator was the posterior median.
| |
| | |
| ===The method===
| |
| [[File:Bendixen - Carl Friedrich Gauß, 1828.jpg|thumb|[[Carl Friedrich Gauss]]]]
| |
| | |
| The first clear and concise exposition of the method of least squares was published by [[Adrien-Marie Legendre|Legendre]] in 1805.<ref>{{Citation |first=Adrien-Marie |last=Legendre |title=Nouvelles méthodes pour la détermination des orbites des comètes |trans_title=New Methods for the Determination of the Orbits of Comets |language=French |publisher=F. Didot |location=Paris |year=1805 |url=http://books.google.com/books/about/Nouvelles_m%C3%A9thodes_pour_la_d%C3%A9terminati.html?id=FRcOAAAAQAAJ }}</ref> The technique is described as an algebraic procedure for fitting linear equations to data and Legendre demonstrates the new method by analyzing the same data as Laplace for the shape of the earth. The value of Legendre's method of least squares was immediately recognized by leading astronomers and geodesists of the time.
| |
| | |
| In 1809 [[Carl Friedrich Gauss]] published his method of calculating the orbits of celestial bodies. In that work he claimed to have been in possession of the method of least squares since 1795. This naturally led to a priority dispute with Legendre. However to Gauss's credit, he went beyond Legendre and succeeded in connecting the method of least squares with the principles of probability and to the normal distribution. He had managed to complete Laplace's program of specifying a mathematical form of the probability density for the observations, depending on a finite number of unknown parameters, and define a method of estimation that minimizes the error of estimation. Gauss showed that arithmetic mean is indeed the best estimate of the location parameter by changing both the probability density and the method of estimation. He then turned the problem around by asking what form the density should have and what method of estimation should be used to get the arithmetic mean as estimate of the location parameter. In this attempt, he invented the normal distribution.
| |
| | |
| An early demonstration of the strength of Gauss's method came when it was used to predict the future location of the newly discovered asteroid [[Ceres (asteroid)|Ceres]]. On 1 January 1801, the Italian astronomer [[Giuseppe Piazzi]] discovered Ceres and was able to track its path for 40 days before it was lost in the glare of the sun. Based on this data, astronomers desired to determine the location of Ceres after it emerged from behind the sun without solving the complicated [[Kepler's laws of planetary motion|Kepler's nonlinear equations]] of planetary motion. The only predictions that successfully allowed Hungarian astronomer [[Franz Xaver von Zach]] to relocate Ceres were those performed by the 24-year-old Gauss using least-squares analysis.
| |
| | |
| In 1810, after reading Gauss's work, Laplace, after proving the [[central limit theorem]], used it to give a large sample justification for the method of least square and the normal distribution. In 1822, Gauss was able to state that the least-squares approach to regression analysis is optimal in the sense that in a linear model where the errors have a mean of zero, are uncorrelated, and have equal variances, the best linear unbiased estimator of the coefficients is the least-squares estimator. This result is known as the [[Gauss–Markov theorem]].
| |
| | |
| The idea of least-squares analysis was also independently formulated by the Frenchman [[Adrien-Marie Legendre]] in 1805 and the American [[Robert Adrain]] in 1808. In the next two centuries workers in the theory of errors and in statistics found many different ways of implementing least squares.<ref>{{cite journal|doi=10.1111/j.1751-5823.1998.tb00406.x|author=J. Aldrich|year=1998|title=Doing Least Squares: Perspectives from Gauss and Yule|journal=International Statistical Review|volume=66|issue=1|pages= 61–81}}</ref>
| |
| | |
| ==Problem statement==
| |
| {{Unreferenced section|date=February 2012}}
| |
| The objective consists of adjusting the parameters of a model function to best fit a data set. A simple data set consists of ''n'' points (data pairs) <math>(x_i,y_i)\!</math>, ''i'' = 1, ..., ''n'', where <math>x_i\!</math> is an [[independent variable]] and <math>y_i\!</math> is a [[dependent variable]] whose value is found by observation. The model function has the form <math>f(x,\beta)</math>, where the ''m'' adjustable parameters are held in the vector <math>\boldsymbol \beta</math>. The goal is to find the parameter values for the model which "best" fits the data. The least squares method finds its optimum when the sum, ''S'', of squared residuals
| |
| :<math>S=\sum_{i=1}^{n}{r_i}^2</math>
| |
| is a minimum. A [[errors and residuals in statistics|residual]] is defined as the difference between the actual value of the dependent variable and the value predicted by the model.
| |
| | |
| :<math>r_i=y_i-f(x_i,\boldsymbol \beta)</math>.
| |
| | |
| An example of a model is that of the straight line in two dimensions. Denoting the intercept as <math>\beta_0</math> and the slope as <math>\beta_1</math>, the model function is given by <math>f(x,\boldsymbol \beta)=\beta_0+\beta_1 x</math>. See [[Linear_least_squares_(mathematics)#Motivational_example|linear least squares]] for a fully worked out example of this model.
| |
| | |
| A data point may consist of more than one independent variable. For example, when fitting a plane to a set of height measurements, the plane is a function of two independent variables, ''x'' and ''z'', say. In the most general case there may be one or more independent variables and one or more dependent variables at each data point. <!-- Also, the residuals may be weighted to take into account differences in the reliability of the measurements.
| |
| math> S = \sum_{i=1}^{n} w_ir_i^2 </math>.
| |
| This is called '''weighted least squares,''' in contrast to '''ordinary least squares''' in which unit weights are used. -->
| |
| | |
| == Limitations ==
| |
| This regression formulation considers only residuals in the dependent variable. There are two rather different contexts in which different implications apply:
| |
| *Regression for prediction. Here a model is fitted to provide a prediction rule for application in a similar situation to which the data used for fitting apply. Here the dependent variables corresponding to such future application would be subject to the same types of observation error as those in the data used for fitting. It is therefore logically consistent to use the least-squares prediction rule for such data.
| |
| *Regression for fitting a "true relationship". In standard [[regression analysis]], that leads to fitting by least squares, there is an implicit assumption that errors in the [[independent variable]] are zero or strictly controlled so as to be negligible. When errors in the [[independent variable]] are non-negligible, [[Errors-in-variables models|models of measurement error]] can be used; such methods can lead to [[parameter estimation|parameter estimates]], [[hypothesis testing]] and [[confidence interval]]s that take into account the presence of observation errors in the independent variables.{{Citation needed|date=February 2012}} An alternative approach is to fit a model by [[total least squares]]; this can be viewed as taking a pragmatic approach to balancing the effects of the different sources of error in formulating an objective function for use in model-fitting.
| |
| | |
| ==Solving the least squares problem==
| |
| {{Unreferenced section|date=February 2012}}
| |
| The [[Maxima and minima|minimum]] of the sum of squares is found by setting the [[gradient]] to zero. Since the model contains ''m'' parameters there are ''m'' gradient equations.
| |
| | |
| :<math>\frac{\partial S}{\partial \beta_j}=2\sum_i r_i\frac{\partial r_i}{\partial \beta_j}=0,\ j=1,\ldots,m</math>
| |
| | |
| and since <math>r_i=y_i-f(x_i,\boldsymbol \beta)\,</math> the gradient equations become
| |
| | |
| :<math>-2\sum_i r_i\frac{\partial f(x_i,\boldsymbol \beta)}{\partial \beta_j}=0,\ j=1,\ldots,m</math>.
| |
| | |
| The gradient equations apply to all least squares problems. Each particular problem requires particular expressions for the model and its partial derivatives.
| |
| | |
| === Linear least squares ===
| |
| {{main|Linear_least_squares_(mathematics)|l1=Linear least squares}}
| |
| | |
| A regression model is a linear one when the model comprises a [[linear combination]] of the parameters, i.e.,
| |
| | |
| :<math> f(x, \beta) = \sum_{j = 1}^{m} \beta_j \phi_j(x)</math>
| |
| | |
| where the functions, <math>\phi_{j}</math>, are functions of <math> x </math>.
| |
| | |
| Letting
| |
| | |
| :<math> X_{ij}= \frac{\partial f(x_i,\boldsymbol \beta)}{\partial \beta_j}= \phi_j(x_{i}) . \, </math>
| |
| | |
| we can then see that in that case the least square estimate (or estimator, in the context of a random sample), <math> \boldsymbol \beta</math> is given by
| |
| | |
| :<math> \boldsymbol{\hat\beta} =( X ^TX)^{-1}X^{T}\boldsymbol y.</math>
| |
| | |
| For a derivation of this estimate see [[Linear least squares (mathematics)]].
| |
| | |
| === Non-linear least squares ===
| |
| {{main|Non-linear least squares}}
| |
| | |
| There is no closed-form solution to a non-linear least squares problem. Instead, numerical algorithms are used to find the value of the parameters <math>\beta</math> which minimize the objective. Most algorithms involve choosing initial values for the parameters. Then, the parameters are refined iteratively, that is, the values are obtained by successive approximation.
| |
| :<math>{\beta_j}^{k+1}={\beta_j}^k+\Delta \beta_j</math>
| |
| ''k'' is an iteration number and the vector of increments, <math>\Delta \beta_j\,</math> is known as the shift vector. In some commonly used algorithms, at each iteration the model may be linearized by approximation to a first-order [[Taylor series]] expansion about <math> \boldsymbol \beta^k\!</math>
| |
| | |
| :<math>
| |
| \begin{align}
| |
| f(x_i,\boldsymbol \beta) & = f^k(x_i,\boldsymbol \beta) +\sum_j \frac{\partial f(x_i,\boldsymbol \beta)}{\partial \beta_j} \left(\beta_j-{\beta_j}^k \right) \\
| |
| & = f^k(x_i,\boldsymbol \beta) +\sum_j J_{ij} \Delta\beta_j.
| |
| \end{align}
| |
| </math>
| |
| | |
| The [[Jacobian matrix and determinant|Jacobian]], '''J''', is a function of constants, the independent variable ''and'' the parameters, so it changes from one iteration to the next. The residuals are given by
| |
| | |
| :<math>r_i=y_i- f^k(x_i,\boldsymbol \beta)- \sum_{k=1}^{m} J_{ik}\Delta\beta_k=\Delta y_i- \sum_{j=1}^{m} J_{ij}\Delta\beta_j</math>.
| |
| | |
| To minimize the sum of squares of <math>r_i</math>, the gradient equation is set to zero and solved for <math> \Delta \beta_j\!</math>
| |
| | |
| :<math>-2\sum_{i=1}^{n}J_{ij} \left( \Delta y_i-\sum_{k=1}^{m} J_{ik}\Delta \beta_k \right)=0</math>
| |
| | |
| which, on rearrangement, become ''m'' simultaneous linear equations, the '''normal equations'''.
| |
| | |
| :<math>\sum_{i=1}^{n}\sum_{k=1}^{m} J_{ij}J_{ik}\Delta \beta_k=\sum_{i=1}^{n} J_{ij}\Delta y_i \qquad (j=1,\ldots,m)\,</math>
| |
| | |
| The normal equations are written in matrix notation as
| |
| | |
| :<math>\mathbf{\left(J^TJ\right)\Delta \boldsymbol \beta=J^T\Delta y}.\,</math>
| |
| <!-- or
| |
| :<math>\mathbf{\left(J^TWJ\right)\Delta \boldsymbol \beta=J^TW\Delta y}</math>
| |
| if weights are used. -->
| |
| | |
| These are the defining equations of the [[Gauss–Newton algorithm]].
| |
| | |
| === Differences between linear and non-linear least squares ===
| |
| * The model function, ''f'', in LLSQ (linear least squares) is a linear combination of parameters of the form <math>f = X_{i1}\beta_1 + X_{i2}\beta_2 +\cdots</math> The model may represent a straight line, a parabola or any other linear combination of functions. In NLLSQ (non-linear least squares) the parameters appear as functions, such as <math>\beta^2, e^{\beta x}</math> and so forth. If the derivatives <math>\partial f /\partial \beta_j</math> are either constant or depend only on the values of the independent variable, the model is linear in the parameters. Otherwise the model is non-linear.
| |
| *Algorithms for finding the solution to a NLLSQ problem require initial values for the parameters, LLSQ does not.
| |
| *Like LLSQ, solution algorithms for NLLSQ often require that the Jacobian be calculated. Analytical expressions for the partial derivatives can be complicated. If analytical expressions are impossible to obtain either the partial derivatives must be calculated by numerical approximation or an estimate must be made of the Jacobian.
| |
| *In NLLSQ non-convergence (failure of the algorithm to find a minimum) is a common phenomenon whereas the LLSQ is globally concave so non-convergence is not an issue.
| |
| *NLLSQ is usually an iterative process. The iterative process has to be terminated when a convergence criterion is satisfied. LLSQ solutions can be computed using direct methods, although problems with large numbers of parameters are typically solved with iterative methods, such as the [[Gauss–Seidel]] method.
| |
| *In LLSQ the solution is unique, but in NLLSQ there may be multiple minima in the sum of squares.
| |
| *Under the condition that the errors are uncorrelated with the predictor variables, LLSQ yields unbiased estimates, but even under that condition NLLSQ estimates are generally biased.
| |
| These differences must be considered whenever the solution to a non-linear least squares problem is being sought.
| |
| | |
| ==Least squares, regression analysis and statistics==
| |
| {{Unreferenced section|date=February 2012}}
| |
| | |
| The methods of least squares and [[regression analysis]] are conceptually different. However, the method of least squares is often used to generate estimators and other statistics in regression analysis. | |
| | |
| Consider a simple example drawn from physics. A spring should obey [[Hooke's law]] which states that the extension of a spring is proportional to the force, ''F'', applied to it.
| |
| :<math>f(F_i,k)=kF_i\!</math>
| |
| constitutes the model, where ''F'' is the independent variable. To estimate the [[force constant]], ''k'', a series of ''n'' measurements with different forces will produce a set of data, <math>(F_i, y_i),\ i=1,\dots,n\!</math>, where ''y<sub>i</sub>'' is a measured spring extension. Each experimental observation will contain some error. If we denote this error <math>\varepsilon</math>, we may specify an empirical model for our observations,
| |
| | |
| : <math> y_i = kF_i + \varepsilon_i. \, </math>
| |
| | |
| There are many methods we might use to estimate the unknown parameter ''k''. Noting that the ''n'' equations in the ''m'' variables in our data comprise an [[overdetermined system]] with one unknown and ''n'' equations, we may choose to estimate ''k'' using least squares. The sum of squares to be minimized is
| |
| | |
| :<math> S = \sum_{i=1}^{n} \left(y_i - kF_i\right)^2. </math>
| |
| | |
| The least squares estimate of the force constant, ''k'', is given by | |
| | |
| :<math>\hat k=\frac{\sum_i F_i y_i}{\sum_i {F_i}^2}.</math>
| |
| | |
| Here it is assumed that application of the force '''''causes''''' the spring to expand and, having derived the force constant by least squares fitting, the extension can be predicted from Hooke's law.
| |
| | |
| In regression analysis the researcher specifies an empirical model. For example, a very common model is the straight line model which is used to test if there is a linear relationship between dependent and independent variable. If a linear relationship is found to exist, the variables are said to be [[correlated]]. However, [[Correlation does not imply causation|correlation does not prove causation]], as both variables may be correlated with other, hidden, variables, or the dependent variable may "reverse" cause the independent variables, or the variables may be otherwise spuriously correlated. For example, suppose there is a correlation between deaths by drowning and the volume of ice cream sales at a particular beach. Yet, both the number of people going swimming and the volume of ice cream sales increase as the weather gets hotter, and presumably the number of deaths by drowning is correlated with the number of people going swimming. Perhaps an increase in swimmers causes both the other variables to increase.
| |
| | |
| In order to make statistical tests on the results it is necessary to make assumptions about the nature of the experimental errors. A common (but not necessary) assumption is that the errors belong to a [[Normal distribution]]. The [[central limit theorem]] supports the idea that this is a good approximation in many cases.
| |
| * The [[Gauss–Markov theorem]]. In a linear model in which the errors have [[expected value|expectation]] zero conditional on the independent variables, are [[uncorrelated]] and have equal [[variance]]s, the best linear [[unbiased]] estimator of any linear combination of the observations, is its least-squares estimator. "Best" means that the least squares estimators of the parameters have minimum variance. The assumption of equal variance is valid when the errors all belong to the same distribution.
| |
| *In a linear model, if the errors belong to a [[Normal distribution]] the least squares estimators are also the [[maximum likelihood estimator]]s.
| |
| | |
| However, if the errors are not normally distributed, a [[central limit theorem]] often nonetheless implies that the parameter estimates will be approximately normally distributed so long as the sample is reasonably large. For this reason, given the important property that the error mean is independent of the independent variables, the distribution of the error term is not an important issue in regression analysis. Specifically, it is not typically important whether the error term follows a normal distribution.
| |
| | |
| In a least squares calculation with unit weights, or in linear regression, the variance on the ''j''th parameter,
| |
| denoted <math>\text{var}(\hat{\beta}_j)</math>, is usually estimated with
| |
| | |
| :<math>\text{var}(\hat{\beta}_j)= \sigma^2\left( \left[X^TX\right]^{-1}\right)_{jj} \approx \frac{S}{n-m}\left( \left[X^TX\right]^{-1}\right)_{jj},</math>
| |
| where the true residual variance σ<sup>2</sup> is replaced by an estimate based on the minimised value of the sum of squares objective function ''S''. The denominator, ''n-m'', is the [[Degrees of freedom (statistics)|statistical degrees of freedom]]; see [[Degrees of freedom (statistics)#Effective degrees of freedom|effective degrees of freedom]] for generalizations.
| |
| | |
| [[Confidence limits]] can be found if the [[probability distribution]] of the parameters is known, or an asymptotic approximation is made, or assumed. Likewise statistical tests on the residuals can be made if the probability distribution of the residuals is known or assumed. The probability distribution of any linear combination of the dependent variables can be derived if the probability distribution of experimental errors is known or assumed. Inference is particularly straightforward if the errors are assumed to follow a normal distribution, which implies that the parameter estimates and residuals will also be normally distributed conditional on the values of the independent variables.
| |
| | |
| ==Weighted least squares==
| |
| {{see also|Weighted mean|Linear least squares (mathematics)#Weighted linear least squares}}
| |
| | |
| A special case of [[Generalized least squares]] called '''weighted least squares''' occurs when all the off-diagonal entries of ''Ω'' (the correlation matrix of the residuals) are 0.
| |
| | |
| The expressions given above are based on the implicit assumption that the errors are uncorrelated with each other and with the independent variables and have equal variance. The [[Gauss–Markov theorem]] shows that, when this is so, <math>\hat{\boldsymbol{\beta}}</math> is a [[best linear unbiased estimator]] (BLUE). If, however, the measurements are uncorrelated but have different uncertainties, a modified approach might be adopted. [[Alexander Aitken|Aitken]] showed that when a weighted sum of squared residuals is minimized, <math>\hat{\boldsymbol{\beta}}</math> is [[Best linear unbiased estimator|BLUE]] if each weight is equal to the reciprocal of the variance of the measurement.
| |
| :<math> S = \sum_{i=1}^{n} W_{ii}{r_i}^2,\qquad W_{ii}=\frac{1}{{\sigma_i}^2} </math>
| |
| The gradient equations for this sum of squares are
| |
| | |
| :<math>-2\sum_i W_{ii}\frac{\partial f(x_i,\boldsymbol {\beta})}{\partial \beta_j} r_i=0,\qquad j=1,\ldots,n</math>
| |
| | |
| which, in a linear least squares system give the modified normal equations,
| |
| | |
| :<math>\sum_{i=1}^{n}\sum_{k=1}^{m} X_{ij}W_{ii}X_{ik}\hat{ \beta}_k=\sum_{i=1}^{n} X_{ij}W_{ii}y_i, \qquad j=1,\ldots,m\,.</math>
| |
| | |
| When the observational errors are uncorrelated and the weight matrix, '''W''', is diagonal, these may be written as
| |
| | |
| :<math>\mathbf{\left(X^TWX\right)\hat {\boldsymbol {\beta}}=X^TWy}.</math>
| |
| | |
| If the errors are correlated, the resulting estimator is BLUE if the weight matrix is equal to the inverse of the [[variance-covariance matrix]] of the observations.
| |
| | |
| When the errors are uncorrelated, it is convenient to simplify the calculations to factor the weight matrix as <math>\mathbf{w_{ii}}=\sqrt{\mathbf{W_{ii}}}</math>. The normal equations can then be written as
| |
| | |
| :<math>\mathbf{\left(X'^TX'\right)\hat{\boldsymbol{\beta}}=X'^Ty'}\,</math>
| |
| | |
| where
| |
| | |
| : <math>\mathbf{X'}=\mathbf{wX}, \mathbf{y'}=\mathbf{wy}.\,</math>
| |
| | |
| For non-linear least squares systems a similar argument shows that the normal equations should be modified as follows.
| |
| | |
| :<math>\mathbf{\left(J^TWJ\right)\boldsymbol \Delta \beta=J^TW \boldsymbol\Delta y}.\,</math>
| |
| | |
| Note that for empirical tests, the appropriate '''W''' is not known for sure and must be
| |
| estimated. For this [[Feasible Generalized Least Squares]] (FGLS) techniques may be used.
| |
| | |
| ==Relationship to principal components==
| |
| The first [[Principal component analysis|principal component]] about the mean of a set of points can be represented by that line which most closely approaches the data points (as measured by squared distance of closest approach, i.e. perpendicular to the line). In contrast, linear least squares tries to minimize the distance in the <math>y</math> direction only. Thus, although the two use a similar error metric, linear least squares is a method that treats one dimension of the data preferentially, while PCA treats all dimensions equally.
| |
| | |
| == Regularized versions ==
| |
| | |
| ===Tikhonov regularization===
| |
| {{Main|Tikhonov regularization}}
| |
| In some contexts a [[Regularization (machine learning)|regularized]] version of the least squares solution may be preferable. [[Tikhonov regularization]] (or [[ridge regression]]) adds a constraint that <math>\|\beta\|^2</math>, the [[L2-norm|L<sup>2</sup>-norm]] of the parameter vector, is not greater than a given value. Equivalently, it may solve an unconstrained minimization of the least-squares penalty with <math>\alpha\|\beta\|^2</math> added, where <math>\alpha</math> is a constant (this is the [[Lagrange multipliers|Lagrangian]] form of the constrained problem). In a [[Bayesian statistics|Bayesian]] context, this is equivalent to placing a zero-mean [[normal distribution|normally distributed]] [[prior distribution|prior]] on the parameter vector.
| |
| | |
| ===Lasso method===
| |
| An alternative [[Regularization (machine learning)|regularized]] version of least squares is ''Lasso'' (least absolute shrinkage and selection operator), which uses the constraint that <math>\|\beta\|_1</math>, the [[L1-norm|L<sup>1</sup>-norm]] of the parameter vector, is no greater than a given value. (As above, this is equivalent to an unconstrained minimization of the least-squares penalty with <math>\alpha\|\beta\|_1</math> added.) In a [[Bayesian statistics|Bayesian]] context, this is equivalent to placing a zero-mean [[Laplace distribution|Laplace]] [[prior distribution]] on the parameter vector. The optimization problem may be solved using [[quadratic programming]] or more general [[convex optimization]] methods, as well as by specific algorithms such as the [[least angle regression]] algorithm.
| |
| | |
| One of the prime differences between Lasso and ridge regression is that in ridge regression, as the penalty is increased, all parameters are reduced while still remaining non-zero, while in Lasso, increasing the penalty will cause more and more of the parameters to be driven to zero.
| |
| | |
| The L<sup>1</sup>-regularized formulation is useful in some contexts due to its tendency to prefer solutions with fewer nonzero parameter values, effectively reducing the number of variables upon which the given solution is dependent.<ref>{{cite journal|author=Tibshirani, R. |year=1996|title=Regression shrinkage and selection via the lasso |journal=[[Journal of the Royal Statistical Society]], Series B|volume= 58|issue= 1| pages =267–288}}</ref> For this reason, the Lasso and its variants are fundamental to the field of [[compressed sensing]]. An extension of this approach is [[elastic net regularization]].
| |
| | |
| ==See also==
| |
| * [[Minimum mean square error|Bayesian MMSE estimator]]
| |
| * [[Gauss–Markov theorem|Best linear unbiased estimator]] (BLUE)
| |
| * [[Best linear unbiased prediction]] (BLUP)
| |
| * [[Gauss-Markov theorem]]
| |
| * [[L2 norm|''L''<sub>2</sub> norm]]
| |
| * [[Least absolute deviation]]
| |
| * [[Measurement uncertainty]]
| |
| * [[Quadratic loss function]]
| |
| * [[Root mean square]]
| |
| * [[Squared deviations]]
| |
| * [[Proximal gradient methods for learning]]
| |
| | |
| ==Notes==
| |
| <references />
| |
| | |
| ==References==
| |
| *{{cite book|author=Å. Björck|isbn=978-0-89871-360-2|title=Numerical Methods for Least Squares Problems|publisher=SIAM|year=1996|url=http://www.ec-securehost.com/SIAM/ot51.html}}
| |
| *{{cite book| author=C.R. Rao, H. Toutenburg, A. Fieger, C. Heumann, T. Nittner and S. Scheid| title=Linear Models: Least Squares and Alternatives| series=Springer Series in Statistics|year=1999}}
| |
| *{{cite book|author=T. Kariya and H. Kurata |title=Generalized Least Squares|publisher= Wiley|year= 2004}}
| |
| *{{cite book|author=J. Wolberg|title=Data Analysis Using the Method of Least Squares: Extracting the Most Information from Experiments|publisher= Springer|year=2005|isbn=3-540-25674-1}}
| |
| *{{cite book|author=T. Strutz| title=Data Fitting and Uncertainty (A practical introduction to weighted least squares and beyond) |publisher=Vieweg+Teubner | year=2010 | isbn= 978-3-8348-1022-9}}
| |
| | |
| {{Least Squares and Regression Analysis|state=expanded}}
| |
| {{Statistics|correlation|state=collapsed}}
| |
| | |
| {{DEFAULTSORT:Least Squares}}
| |
| [[Category:Least squares| ]]
| |
| [[Category:Regression analysis]]
| |
| [[Category:Single-equation methods (econometrics)]]
| |
| [[Category:Mathematical and quantitative methods (economics)]]
| |
| [[Category:Mathematical optimization]]
| |
| | |
| {{Link GA|de}}
| |