|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| {{Regression bar}}
| | Bonus: WP Twin and WP Twin Auto Backup: (link to ) While not a theme, I think this software is essential if you are maintaining your Wordpress blog or regularly create new blog sites. You can either install Word - Press yourself or use free services offered on the web today. One really cool features about this amazing and free wp plugin is that the code it generates is completely portable. s and intelligently including a substantial amount of key words in the title tags, image links, etc. If you are happy with your new look then click "Activate 'New Theme'" in the top right corner. <br><br>Most Word - Press web developers can provide quality CMS website solutions and they price their services at reasonable rates. You do not catch a user's attention through big and large pictures that usually takes a millennium to load up. This plugin allows a blogger get more Facebook fans on the related fan page. By purchasing Word - Press weblogs you can acquire your very own domain title and have total command of your web site. If you beloved this article and you also would like to receive more info with regards to [http://deurl.de/wordpress_backup_plugin_295630 wordpress backup] please visit the website. Moreover, many Word - Press themes need to be purchased and designing your own WP site can be boring. <br><br>Minor and medium sized corporations also have a lot to obtain by shelling out in a very good website. Browse through the popular Wordpress theme clubs like the Elegant Themes, Studio Press, Woo - Themes, Rocket Theme, Simple Themes and many more. I've applied numerous Search engine optimization-ready Word - Press themes and I can say from knowledge that I consider the Genesis Search engine marketing panel one particular of the simplest to use. These frequent updates have created menace in the task of optimization. If you've hosted your Word - Press website on a shared hosting server then it'll be easier for you to confirm the restricted access to your site files. <br><br>Word - Press has plenty of SEO benefits over Joomla and Drupal. The SEOPressor Word - Press SEO Plugin works by analysing each page and post against your chosen keyword (or keyword phrase) and giving a score, with instructions on how to improve it. This allows for keeping the content editing toolbar in place at all times no matter how far down the page is scrolled. Word - Press is the most popular open source content management system (CMS) in the world today. See a product, take a picture, and it gives you an Amazon price for that product, or related products. <br><br>Many developers design websites and give them to the clients, but still the client faces problems to handle the website. If you operate a website that's been built on HTML then you might have to witness traffic losses because such a site isn't competent enough in grabbing the attention of potential consumers. Useful Plugins Uber - Menu Top Megamenu Now it is the time of sticky Top navbar. Working with a Word - Press blog and the appropriate cost-free Word - Press theme, you can get a professional internet site up and published in no time at all. As for performing online business, websites and blogs are the only medium that are available to interact with customers and Word - Press perform this work with the help of cross-blog communication tools, comments and full user registration plug-ins. |
| '''Least absolute deviations''' (LAD), also known as Least Absolute Errors (LAE), Least Absolute Value (LAV), or the [[L1 norm|''L''<sub>1</sub> norm]] problem, is a [[mathematics|mathematical]] [[optimization (mathematics)|optimization]] technique similar to the popular [[least squares]] technique that attempts to find a [[function (mathematics)|function]] which closely approximates a set of data. In the simple case of a set of (''x'',''y'') data, the approximation function is a simple "trend line" in two-dimensional [[Cartesian coordinates]]. The method [[maxima and minima|minimizes]] the sum of absolute errors (SAE) (the sum of the absolute values of the vertical "residuals" between points generated by the function and corresponding points in the data). The least absolute deviations estimate also arises as the [[maximum likelihood]] estimate if the errors have a [[Laplace distribution]].
| |
| | |
| ==Formulation of the problem==
| |
| | |
| Suppose that the [[data set]] consists of the points (''x''<sub>''i''</sub>, ''y''<sub>''i''</sub>) with ''i'' = 1, 2, ..., ''n''. We want to find a function ''f'' such that <math>f(x_i)\approx y_i.</math>
| |
| | |
| To attain this goal, we suppose that the function ''f'' is of a particular form containing some parameters which need to be determined. For instance, the simplest form would be linear: ''f''(''x'') = ''bx'' + ''c'', where ''b'' and ''c'' are parameters whose values are not known but which we would like to estimate. Less simply, suppose that ''f''(''x'') is [[quadratic function|quadratic]], meaning that ''f''(''x'') = ''ax''<sup>2</sup> + ''bx'' + ''c'', where ''a'', ''b'' and ''c'' are not yet known. (More generally, there could be not just one explanator ''x'', but rather multiple explanators, all appearing as arguments of the function ''f''.)
| |
| | |
| We now seek estimated values of the unknown parameters that minimize the sum of the absolute values of the residuals:
| |
| | |
| :<math> S = \sum_{i=1}^n |y_i - f(x_i)|. </math>
| |
| | |
| ==Contrasting Least Squares with Least Absolute Deviations==
| |
| | |
| The following is a table contrasting some properties of the method of least absolute deviations with those of the method of least squares (for non-singular problems).
| |
| | |
| {| border="1" cellpadding="5" cellspacing="0"
| |
| |-
| |
| ! Least Squares Regression || Least Absolute Deviations Regression
| |
| |-
| |
| | rowspan=1 align="center"| Not very robust
| |
| | colspan=2 align="center"| Robust
| |
| |-
| |
| | rowspan=1 align="center"| Stable solution
| |
| | colspan=2 align="center"| Unstable solution
| |
| |-
| |
| | rowspan=1 align="center"| Always one solution
| |
| | colspan=2 align="center"| Possibly multiple solutions
| |
| |-
| |
| |}
| |
| | |
| The method of least absolute deviations finds applications in many areas, due to its robustness compared to the least squares method. Least absolute deviations is robust in that it is resistant to outliers in the data. This may be helpful in studies where outliers may be safely and effectively ignored. If it is important to pay attention to any and all outliers, the method of least squares is a better choice.
| |
| | |
| The instability property of the method of least absolute deviations means that, for a small horizontal adjustment of a [[data|datum]], the regression line may jump a large amount. The method has continuous solutions for some data configurations; however, by moving a datum a small amount, one could "jump past" a configuration which has multiple solutions that span a region. After passing this region of solutions, the least absolute deviations line has a slope that may differ greatly from that of the previous line. In contrast, the least squares solutions is stable in that, for any small adjustment of a data point, the regression line will always move only slightly; that is, the regression parameters are continuous functions of the data.
| |
| | |
| Lastly, for a given data set, the method of least absolute deviations may produce multiple solutions, whereas the method of least squares always produces only one solution (the regression line is unique).
| |
| | |
| For a set of applets that demonstrate these differences, see the following site: http://www.math.wpi.edu/Course_Materials/SAS/lablets/7.3/73_choices.html
| |
| | |
| ==Other properties==
| |
| | |
| There exist other unique properties of the least absolute deviations line. In the case of a set of (''x'',''y'') data, the least absolute deviations line will always pass through at least two of the data points, unless there are multiple solutions. If multiple solutions exist, then the region of valid least absolute deviations solutions will be bounded by at least two lines, each of which passes through at least two data points. More generally, if there are ''k'' [[Dependent and independent variables#Alternative terminology in statistics|regressors]] (including the constant), then at least one optimal regression surface will pass through ''k'' of the data points.<ref>Branham, R. L., Jr., "Alternatives to least squares", ''[[Astronomical Journal]]'' 87, June 1982, 928–937. [http://adsabs.harvard.edu/full/1982AJ.....87..928B] at SAO/NASA Astrophysics Data System (ADS)</ref>{{rp|p.936}}
| |
| | |
| This "latching" of the line to the data points can help to understand the "instability" property: if the line always latches to at least two points, then the line will jump between different sets of points as the data points are altered. The "latching" also helps to understand the "robustness" property: if there exists an outlier, and a least absolute deviations line must latch onto two data points, the outlier will most likely not be one of those two points because that will not minimize the sum of absolute deviations in most cases.
| |
| | |
| One known case in which multiple solutions exist is a set of points symmetric about a horizontal line, as shown in Figure A below.
| |
| | |
| [[File:Least absolute deviations regression method diagram.gif|600px|thumb|center|Figure A: A set of data points with reflection symmetry and multiple least absolute deviations solutions. The “solution area” is shown in green. The vertical blue lines represent the absolute errors from the pink line to each data point. The pink line is one of infinitely many solutions within the green area.]]
| |
| | |
| To understand why there are multiple solutions in the case shown in Figure A, consider the pink line in the green region. Its sum of absolute errors is some value S. If one were to tilt the line upward slightly, while still keeping it within the green region, the sum of errors would still be S. It would not change because the distance from each point to the line grows on one side of the line, while the distance to each point on the opposite side of the line diminishes by exactly the same amount. Thus the sum of absolute errors remains the same. Also, since one can tilt the line in infinitely small increments, this also shows that if there is more than one solution, there are infinitely many solutions.
| |
| | |
| ==Variations, extensions, specializations==
| |
| The least absolute deviation problem may be extended to include multiple explanators, constraints and [[regularization (mathematics)|regularization]], e.g., a linear model with linear constraints:<ref>{{Cite journal
| |
| | author = [[Mingren Shi]] & [[Mark A. Lukas]]
| |
| | title = An ''L<sub>1</sub>'' estimation algorithm with degeneracy and linear constraints
| |
| | journal = [[Computational Statistics & Data Analysis]]
| |
| | volume = 39
| |
| | issue = 1
| |
| |date=March 2002
| |
| | pages = 35–55
| |
| | doi = 10.1016/S0167-9473(01)00049-4
| |
| }}</ref>
| |
| : minimize <math>S(\mathbf{\beta}, b) = \sum_i | \mathbf{x}'_i \mathbf{\beta} + b - y_i |</math>
| |
| : subject to, e.g., <math>\mathbf{x}'_1 \mathbf{\beta} + b - y_1 \leq k</math>
| |
| | |
| where <math>\mathbf{\beta}</math> is a column vector of coefficients to be estimated, ''b'' is an intercept to be estimated, '''x'''<sub>'''i''' </sub> is a column vector of the ''i''<sup>th</sup> observations on the various explanators, ''y''<sub>''i''</sub> is the ''i''<sup>th</sup> observation on the dependent variable, and ''k'' is a known constant.
| |
| | |
| [[Regularization (mathematics)|Regularization]] with [[Least squares#Lasso method|LASSO]] may also be combined with LAD.<ref>{{Cite conference
| |
| | author = Li Wang, Michael D. Gordon & Ji Zhu
| |
| | title = Regularized Least Absolute Deviations Regression and an Efficient Algorithm for Parameter Tuning
| |
| | booktitle = Proceedings of the Sixth International Conference on Data Mining
| |
| |date=December 2006
| |
| | pages = 690–700
| |
| | doi = 10.1109/ICDM.2006.134
| |
| }}</ref>
| |
| | |
| ==Solving methods==
| |
| | |
| Though the idea of least absolute deviations regression is just as straightforward as that of least squares regression, the least absolute deviations line is not as simple to compute efficiently. Unlike least squares regression, least absolute deviations regression does not have an analytical solving method. Therefore, an iterative approach is required. The following is an enumeration of some least absolute deviations solving methods.
| |
| | |
| * [[Simplex algorithm|Simplex-based methods]] (such as the Barrodale-Roberts algorithm<ref>{{Cite journal
| |
| | author = I. Barrodale & F. D. K. Roberts
| |
| | title = An improved algorithm for discrete L<sub>1</sub> linear approximation
| |
| | journal = [[SIAM Journal on Numerical Analysis]]
| |
| | volume = 10
| |
| | year = 1973
| |
| | pages = 839–848
| |
| | jstor = 2156318
| |
| | doi = 10.1137/0710069
| |
| | issue = 5
| |
| |bibcode = 1973SJNA...10..839B }}</ref>)
| |
| ** Because the problem is a [[linear program]], any of the many linear programming techniques (including the simplex method as well as others) can be applied.
| |
| * [[Iteratively re-weighted least squares]]<ref>{{Cite journal
| |
| | author = E. J. Schlossmacher
| |
| | title = An Iterative Technique for Absolute Deviations Curve Fitting
| |
| | journal = [[Journal of the American Statistical Association]]
| |
| | volume = 68
| |
| | issue = 344
| |
| |date=December 1973
| |
| | pages = 857–859
| |
| | jstor = 2284512
| |
| | doi = 10.2307/2284512
| |
| | publisher = Journal of the American Statistical Association, Vol. 68, No. 344
| |
| }}</ref>
| |
| * Wesolowsky’s direct descent method<ref>{{Cite journal
| |
| | author = G. O. Wesolowsky
| |
| | year = 1981
| |
| | title = A new descent algorithm for the least absolute value regression problem
| |
| | journal = Communications in Statistics – Simulation and Computation
| |
| | volume = B10
| |
| | issue = 5
| |
| | pages = 479–491
| |
| | url = http://www.informaworld.com/index/780061825.pdf
| |
| | doi = 10.1080/03610918108812224
| |
| }}</ref>
| |
| * Li-Arce’s maximum likelihood approach<ref>{{Cite journal
| |
| | author = Yinbo Li and Gonzalo R. Arce
| |
| | title = A Maximum Likelihood Approach to Least Absolute Deviation Regression
| |
| | journal = [[EURASIP Journal on Applied Signal Processing]]
| |
| | volume = 2004
| |
| | year = 2004
| |
| | issue = 12
| |
| | pages = 1762–1769
| |
| | doi = 10.1155/S1110865704401139
| |
| | url = http://www.hindawi.com/journals/asp/2004/948982.abs.html
| |
| }}</ref>
| |
| * Check all combinations of point-to-point lines for minimum sum of errors
| |
| | |
| Simplex-based methods are the “preferred” way to solve the least absolute deviations problem.<ref name=Pfeil>William A. Pfeil,
| |
| ''[http://www.wpi.edu/Pubs/E-project/Available/E-project-050506-091720/unrestricted/IQP_Final_Report.pdf Statistical Teaching Aids]'', Bachelor of Science thesis, [[Worcester Polytechnic Institute]], 2006</ref> A Simplex method is a method for solving a problem in linear programming. The most popular algorithm is the Barrodale-Roberts modified Simplex algorithm. The algorithms for IRLS, Wesolowsky's Method, and Li's Method can be found in Appendix A of,<ref name=Pfeil/>
| |
| among other methods. Checking all combinations of lines traversing any two (x,y) data points is another method of finding the least absolute deviations line. Since it is known that at least one least absolute deviations line traverses at least two data points, this method will find a line by comparing the SAE of each line, and choosing the line with the smallest SAE. In addition, if multiple lines have the same, smallest SAE, then the lines outline the region of multiple solutions. Though simple, this final method is inefficient for large sets of data.
| |
| | |
| ===Solving using linear programming===
| |
| | |
| The problem can be solved using any linear programming technique on the following problem specification. We wish to
| |
| | |
| :<math> \text{Minimize} \sum_{i=1}^n |y_i - a_0 - a_1x_{i1} - a_2x_{i2} - \cdots - a_kx_{ik}|</math>
| |
| | |
| with respect to the choice of the values of the parameters <math>a_0,..., a_k</math>, where ''y''<sub>''i''</sub> is the value of the ''i''<sup>th</sup> observation of the dependent variable, and ''x''<sub>''ij''</sub> is the value of the ''i''<sup>th</sup> observation of the ''j''<sup>th</sup> independent variable (''j'' = 1,...,''k''). We rewrite this problem in terms of artificial variables ''u''<sub>''i''</sub> as
| |
| | |
| :<math> \text{Minimize} \sum_{i=1}^n u_i</math>
| |
| | |
| :with respect to <math>a_0,..., a_k</math> and <math>u_1,..., u_n</math>
| |
| | |
| :subject to
| |
| | |
| :<math> u_i \ge y_i - a_0 - a_1x_{i1} - a_2x_{i2} - \cdots - a_kx_{ik} \,\ \,\ \,\ \,\ \,\ \text{for} \,\ i=1,...,n</math>
| |
| | |
| :<math> u_i \ge -[y_i - a_0 - a_1x_{i1} - a_2x_{i2} - \cdots - a_kx_{ik}] \,\ \,\ \text{for} \,\ i=1,...,n.</math>
| |
| | |
| These constraints have the effect of forcing each <math>u_i</math> to equal <math>|y_i - a_0 - a_1x_{i1} - a_2x_{i2} - \cdots - a_kx_{ik}|</math> upon being minimized, so the objective function is equivalent to the original objective function. Since this version of the problem statement does not contain the absolute value operator, it is in a format that can be solved with any linear programming package.
| |
| | |
| ==See also==
| |
| * [[Regression analysis]]
| |
| * [[Linear regression model]]
| |
| * [[Absolute deviation]]
| |
| * [[Ordinary least squares]]
| |
| | |
| ==References==
| |
| {{Reflist}}
| |
| | |
| ==External links==
| |
| * {{Cite journal
| |
| | author = Peter Bloomfield and William Steiger
| |
| | title = Least Absolute Deviations Curve-Fitting | |
| | journal = [[SIAM Journal on Scientific Computing]]
| |
| | year = 1980
| |
| | volume = 1
| |
| | issue = 2
| |
| | pages = pages 290–301
| |
| | doi = 10.1137/0901019
| |
| }}
| |
| * {{Cite journal
| |
| | author = Subhash C. Narula and John F. Wellington
| |
| | title = The Minimum Sum of Absolute Errors Regression: A State of the Art Survey
| |
| | journal = [[International Statistical Review]]
| |
| | volume = 50
| |
| | issue = 3
| |
| | year = 1982
| |
| | pages = 317–326
| |
| | jstor = 1402501
| |
| | doi = 10.2307/1402501
| |
| | publisher = International Statistical Review / Revue Internationale de Statistique, Vol. 50, No. 3
| |
| }}
| |
| * {{Cite journal
| |
| | author = Robert F. Phillips
| |
| | title = Least absolute deviations estimation via the EM algorithm
| |
| | journal = [[Statistics and Computing]]
| |
| | volume = 12
| |
| | issue = 3
| |
| |date=July 2002
| |
| | doi = 10.1023/A:1020759012226
| |
| | pages = 281–285
| |
| }}
| |
| * {{Cite journal
| |
| | author = Enno Siemsen & Kenneth A. Bollen
| |
| | title = Least Absolute Deviation Estimation in Structural Equation Modeling
| |
| | journal = [[Sociological Methods & Research]]
| |
| | volume = 36
| |
| | issue = 2
| |
| | pages = 227–265
| |
| | year = 2007
| |
| | doi = 10.1177/0049124107301946
| |
| | url = http://smr.sagepub.com/cgi/content/abstract/36/2/227
| |
| }}
| |
| | |
| {{DEFAULTSORT:Least Absolute Deviations}}
| |
| [[Category:Mathematical optimization]]
| |
| [[Category:Robust statistics]]
| |
| [[Category:Robust regression]]
| |
| [[Category:Estimation theory]]
| |
Bonus: WP Twin and WP Twin Auto Backup: (link to ) While not a theme, I think this software is essential if you are maintaining your Wordpress blog or regularly create new blog sites. You can either install Word - Press yourself or use free services offered on the web today. One really cool features about this amazing and free wp plugin is that the code it generates is completely portable. s and intelligently including a substantial amount of key words in the title tags, image links, etc. If you are happy with your new look then click "Activate 'New Theme'" in the top right corner.
Most Word - Press web developers can provide quality CMS website solutions and they price their services at reasonable rates. You do not catch a user's attention through big and large pictures that usually takes a millennium to load up. This plugin allows a blogger get more Facebook fans on the related fan page. By purchasing Word - Press weblogs you can acquire your very own domain title and have total command of your web site. If you beloved this article and you also would like to receive more info with regards to wordpress backup please visit the website. Moreover, many Word - Press themes need to be purchased and designing your own WP site can be boring.
Minor and medium sized corporations also have a lot to obtain by shelling out in a very good website. Browse through the popular Wordpress theme clubs like the Elegant Themes, Studio Press, Woo - Themes, Rocket Theme, Simple Themes and many more. I've applied numerous Search engine optimization-ready Word - Press themes and I can say from knowledge that I consider the Genesis Search engine marketing panel one particular of the simplest to use. These frequent updates have created menace in the task of optimization. If you've hosted your Word - Press website on a shared hosting server then it'll be easier for you to confirm the restricted access to your site files.
Word - Press has plenty of SEO benefits over Joomla and Drupal. The SEOPressor Word - Press SEO Plugin works by analysing each page and post against your chosen keyword (or keyword phrase) and giving a score, with instructions on how to improve it. This allows for keeping the content editing toolbar in place at all times no matter how far down the page is scrolled. Word - Press is the most popular open source content management system (CMS) in the world today. See a product, take a picture, and it gives you an Amazon price for that product, or related products.
Many developers design websites and give them to the clients, but still the client faces problems to handle the website. If you operate a website that's been built on HTML then you might have to witness traffic losses because such a site isn't competent enough in grabbing the attention of potential consumers. Useful Plugins Uber - Menu Top Megamenu Now it is the time of sticky Top navbar. Working with a Word - Press blog and the appropriate cost-free Word - Press theme, you can get a professional internet site up and published in no time at all. As for performing online business, websites and blogs are the only medium that are available to interact with customers and Word - Press perform this work with the help of cross-blog communication tools, comments and full user registration plug-ins.