|
|
| Line 1: |
Line 1: |
| {{About|the partition of sums of squares in statistics||Sum of squares (disambiguation){{!}}Sum of squares}}
| | Andrew Simcox is the title his parents gave him and he completely loves this name. Doing ballet is something she would by no means give up. My spouse and I live in Mississippi and I love every day residing right here. My working day occupation is an information officer but I've already applied for another one.<br><br>My web-site: accurate psychic predictions, [http://1.234.36.240/fxac/m001_2/7330 just click the following internet site], |
| | |
| {{Expert-subject|Statistics|date=November 2008}}
| |
| | |
| The '''partition of sums of squares''' is a concept that permeates much of [[inferential statistics]] and [[descriptive statistics]]. More properly, it is the '''partitioning of sums of [[squared deviations]] or errors'''. Mathematically, the sum of squared deviations is an unscaled, or unadjusted measure of [[statistical dispersion|dispersion]] (also called [[statistical variability|variability]]). When scaled for the number of [[Degrees of freedom (statistics)|degrees of freedom]], it estimates the [[variance]], or spread of the observations about their mean value. Partitioning of the sum of squared deviations into various components allows the overall variability in a dataset to be ascribed to different types or sources of variability, with the relative importance of each being quantified by the size of each component of the overall sum of squares.
| |
| | |
| ==Background==
| |
| | |
| The distance from any point in a collection of data, to the mean of the data, is the deviation. This can be written as <math>y_i - \overline{y}</math>, where <math>y_i</math> is the ith data point, and <math>\overline{y}</math> is the estimate of the mean. If all such deviations are squared, then summed, as in <MATH>\sum_{i=1}^n\left(y_i-\overline{y}\,\right)^2</MATH>, this gives the "sum of squares" for these data.
| |
| | |
| When more data are added to the collection the sum of squares will increase, except in unlikely cases such as the new data being equal to the mean. So usually, the sum of squares will grow with the size of the data collection. That is a manifestation of the fact that it is unscaled.
| |
| | |
| In many cases, the number of [[degrees of freedom (statistics)|degrees of freedom]] is simply the number of data in the collection, minus one. We write this as ''n'' − 1, where ''n'' is the number of data.
| |
| | |
| Scaling (also known as normalizing) means adjusting the sum of squares so that it does not grow as the size of the data collection grows. This is important when we want to compare samples of different sizes, such as a sample of 100 people compared to a sample of 20 people. If the sum of squares was not normalized, its value would always be larger for the sample of 100 people than for the sample of 20 people. To scale the sum of squares, we divide it by the degrees of freedom, i.e., calculate the sum of squares per degree of freedom, or variance. [[Standard deviation]], in turn, is the square root of the variance.
| |
| | |
| The above information is how sum of squares is used in descriptive statistics; see the article on [[total sum of squares]] for an application of this broad principle to [[inferential statistics]].
| |
| | |
| ==Partitioning the sum of squares in linear regression==
| |
| | |
| '''Theorem.''' Given a linear regression model <math> y_i = \beta_0 + \beta_1 x_{i1} + \cdots + \beta_p x_{ip} + \varepsilon_i </math> ''including a constant'' based on a sample <math> (y_i, x_{i1}, \ldots, x_{ip}), \, i = 1, \ldots, n </math> containing ''n'' observations, the total sum of squares <math> \sum_{i = 1}^n (y_i - \bar{y})^2 </math> (TSS) can be partitioned as follows into the [[explained sum of squares]] (ESS) and the [[residual sum of squares]] (RSS): | |
| :<math>\mathrm{TSS} = \mathrm{ESS} + \mathrm{RSS},</math> | |
| where this equation is equivalent to each of the following forms:
| |
| | |
| :<math>
| |
| \begin{align}
| |
| \left\| y - \bar{y} \mathbf{1} \right\|^2 &= \left\| \hat{y} - \bar{y} \mathbf{1} \right\|^2 + \left\| \hat{\varepsilon} \right\|^2, \quad \mathbf{1} = (1, 1, \ldots, 1)^T ,\\
| |
| \sum_{i = 1}^n (y_i - \bar{y})^2 &= \sum_{i = 1}^n (\hat{y}_i - \bar{y})^2 + \sum_{i = 1}^n (y_i - \hat{y}_i)^2 ,\\
| |
| \sum_{i = 1}^n (y_i - \bar{y})^2 &= \sum_{i = 1}^n (\hat{y}_i - \bar{y})^2 + \sum_{i = 1}^n \hat{\varepsilon}_i^2 .\\
| |
| \end{align}
| |
| </math>
| |
| | |
| ===Proof===
| |
| | |
| :<math>
| |
| \begin{align}
| |
| \sum_{i = 1}^n (y_i - \overline{y})^2 &= \sum_{i = 1}^n (y_i - \overline{y} + \hat{y}_i - \hat{y}_i)^2
| |
| = \sum_{i = 1}^n ((\hat{y}_i - \bar{y}) + \underbrace{(y_i - \hat{y}_i)}_{\hat{\varepsilon}_i})^2 \\
| |
| &= \sum_{i = 1}^n ((\hat{y}_i - \bar{y})^2 + 2 \hat{\varepsilon}_i (\hat{y}_i - \bar{y}) + \hat{\varepsilon}_i^2) \\
| |
| &= \sum_{i = 1}^n (\hat{y}_i - \bar{y})^2 + \sum_{i = 1}^n \hat{\varepsilon}_i^2 + 2 \sum_{i = 1}^n \hat{\varepsilon}_i (\hat{y}_i - \bar{y}) \\
| |
| &= \sum_{i = 1}^n (\hat{y}_i - \bar{y})^2 + \sum_{i = 1}^n \hat{\varepsilon}_i^2 + 2 \sum_{i = 1}^n \hat{\varepsilon}_i(\hat{\beta}_0 + \hat{\beta}_1 x_{i1} + \cdots + \hat{\beta}_p x_{ip} - \overline{y}) \\
| |
| &= \sum_{i = 1}^n (\hat{y}_i - \bar{y})^2 + \sum_{i = 1}^n \hat{\varepsilon}_i^2 + 2 (\hat{\beta}_0 - \overline{y}) \underbrace{\sum_{i = 1}^n \hat{\varepsilon}_i}_0 + 2 \hat{\beta}_1 \underbrace{\sum_{i = 1}^n \hat{\varepsilon}_i x_{i1}}_0 + \cdots + 2 \hat{\beta}_p \underbrace{\sum_{i = 1}^n \hat{\varepsilon}_i x_{ip}}_0 \\
| |
| &= \sum_{i = 1}^n (\hat{y}_i - \bar{y})^2 + \sum_{i = 1}^n \hat{\varepsilon}_i^2 = \mathrm{ESS} + \mathrm{RSS} \\
| |
| \end{align}
| |
| </math>
| |
| | |
| The requirement that the model includes a constant or equivalently that the design matrix contains a column of ones ensures that <math> \sum_{i = 1}^n \hat{\varepsilon}_i = 0 </math>.
| |
| | |
| Some readers may find the following version of the proof, set in vector form, more enlightening:
| |
| | |
| <math>
| |
| \begin{align}
| |
| SS_{{\text{total}}} = \Vert {{\mathbf{y}} - \bar y{\mathbf{1}}} \Vert^2 & = \Vert {{\mathbf{y}} - \bar y{\mathbf{1}} + {\mathbf{\hat y}} - {\mathbf{\hat y}}} \Vert^2 , \\
| |
| & = \Vert {\left( {{\mathbf{\hat y}} - \bar y{\mathbf{1}}} \right) + \left( {{\mathbf{y}} - {\mathbf{\hat y}}} \right)} \Vert^2 , \\
| |
| & = \Vert {{\mathbf{\hat y}} - \bar y{\mathbf{1}}} \Vert^2 + \Vert{\hat \varepsilon }\Vert^2 + 2{\hat \varepsilon }^T \left( {{\mathbf{\hat y}} - \bar y{\mathbf{1}}} \right) , \\
| |
| & = SS_{{\text{regression}}} + SS_{{\text{error}}} + 2{\hat \varepsilon }^T \left( {X{\hat \beta } - \bar y{\mathbf{1}}} \right) ,\\
| |
| & = SS_{{\text{regression}}} + SS_{{\text{error}}} + 2\left( {\hat \varepsilon ^T X} \right){\hat \beta - }2\bar y{\hat \varepsilon }^T { \mathbf{1} } , \\
| |
| & = SS_{{\text{regression}}} + SS_{{\text{error}}} .\\
| |
| \end{align}
| |
| </math>
| |
| | |
| The elimination of terms in the last line, used the fact that
| |
| | |
| : <math>
| |
| \hat \varepsilon ^T X = \left( {\mathbf{y}} - {\mathbf{\hat y}} \right)^T X
| |
| = {\mathbf{y}}^T\left( {I - X\left( {X^T X} \right)^{ - 1} X^T } \right)X = {\mathbf{y}}^T\left(X-X \right)={\mathbf{0}}.
| |
| </math>
| |
| | |
| ===Further partitioning===
| |
| | |
| Note that the residual sum of squares can be further partitioned as the [[lack-of-fit sum of squares]] plus the sum of squares due to pure error.
| |
| | |
| ==See also==
| |
| * [[Inner-product space]]
| |
| ** [[Hilbert space]]
| |
| *** [[Euclidean space]]
| |
| ** [[Orthogonality]]
| |
| ** [[Orthonormal basis]]
| |
| ***[[Orthogonal complement]], the closed subspace orthogonal to a set (especially a subspace)
| |
| ***[[Orthomodular lattice]] of the subspaces of an inner-product space
| |
| ***[[Orthogonal projection]]
| |
| ** [[Pythagorean theorem]] that the sum of the squared norms of orthogonal summands equals the squared norm of the sum.
| |
| * [[Least squares]]
| |
| * [[Mean squared error]]
| |
| * [[Squared deviations]]
| |
| | |
| ==References==
| |
| | |
| * {{cite book |last=Bailey|first=R. A.|authorlink=Rosemary A. Bailey|title=Design of Comparative Experiments|publisher=[http://www.cambridge.org/uk/catalogue/catalogue.asp?isbn=9780521683579 Cambridge University Press]|year=2008 |isbn=978-0-521-68357-9|url=http://www.maths.qmul.ac.uk/~rab/DOEbook}} Pre-publication chapters are available on-line.
| |
| * {{cite book
| |
| |title=Plane Answers to Complex Questions: The Theory of Linear Models|last=Christensen|first=Ronald|location=New York|publisher=Springer|year=2002| edition=Third|isbn=0-387-95361-2}}
| |
| * {{cite book|title=Prediction and Regulation|last=Whittle|first=Peter|authorlink=Peter Whittle|publisher=English Universities Press|year=1963|isbn=0-8166-1147-5}}
| |
| *:Republished as: {{cite book|title=Prediction and Regulation by Linear Least-Square Methods|author=Whittle, P.|publisher=University of Minnesota Press|year=1983|isbn=0-8166-1148-3}}
| |
| * {{cite book|title=Probability Via Expectation|edition=4th|author=Whittle, P.|publisher=Springer|date=20 April 2000|isbn=0-387-98955-2}}
| |
| | |
| [[Category:Analysis of variance]]
| |
| [[Category:Regression analysis]]
| |
| [[Category:Least squares]]
| |
Andrew Simcox is the title his parents gave him and he completely loves this name. Doing ballet is something she would by no means give up. My spouse and I live in Mississippi and I love every day residing right here. My working day occupation is an information officer but I've already applied for another one.
My web-site: accurate psychic predictions, just click the following internet site,