ā¹ļø Skipped - page is already crawled
| Filter | Status | Condition | Details |
|---|---|---|---|
| HTTP status | PASS | download_http_code = 200 | HTTP 200 |
| Age cutoff | PASS | download_stamp > now() - 6 MONTH | 0.4 months ago (distributed domain, exempt) |
| History drop | PASS | isNull(history_drop_reason) | No drop reason |
| Spam/ban | PASS | fh_dont_index != 1 AND ml_spam_score = 0 | ml_spam_score=0 |
| Canonical | PASS | meta_canonical IS NULL OR = '' OR = src_unparsed | Not set |
| Property | Value | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| URL | https://en.wikipedia.org/wiki/Ordinary_least_squares | |||||||||
| Last Crawled | 2026-04-13 08:10:06 (11 days ago) | |||||||||
| First Indexed | 2013-08-24 22:28:28 (12 years ago) | |||||||||
| HTTP Status Code | 200 | |||||||||
| Content | ||||||||||
| Meta Title | Ordinary least squares - Wikipedia | |||||||||
| Meta Description | null | |||||||||
| Meta Canonical | null | |||||||||
| Boilerpipe Text | Okun's law
in
macroeconomics
states that in an economy the
GDP
growth should depend linearly on the changes in the unemployment rate. Here the ordinary least squares method is used to construct the regression line describing this law.
In
statistics
,
ordinary least squares
(
OLS
) is a type of
linear least squares
method for choosing the unknown
parameters
in a
linear regression
model (with fixed level-one
[
clarification needed
]
effects of a
linear function
of a set of
explanatory variables
) by the principle of
least squares
: minimizing the sum of the squares of the differences between the observed
dependent variable
(values of the variable being observed) in the input
dataset
and the output of the (linear) function of the
independent variable
. Some sources consider OLS to be linear regression.
[
1
]
Geometrically, this is seen as the sum of the squared distances, parallel to the axis of the dependent variable, between each data point in the set and the corresponding point on the regression surfaceāthe smaller the differences, the better the model fits the data. The resulting
estimator
can be expressed by a simple formula, especially in the case of a
simple linear regression
, in which there is a single
regressor
on the right side of the regression equation.
The OLS estimator is
consistent
for the level-one fixed effects when the regressors are
exogenous
and forms perfect
colinearity
(rank condition), consistent for the variance estimate of the residuals when regressors have finite fourth moments
[
2
]
andāby the
GaussāMarkov theorem
ā
optimal in the class of linear unbiased estimators
when the
errors
are
homoscedastic
and
serially uncorrelated
. Under these conditions, the method of OLS provides
minimum-variance mean-unbiased
estimation when the errors have finite
variances
. Under the additional assumption that the errors are
normally distributed
with zero mean, OLS is the
maximum likelihood estimator
that outperforms any non-linear unbiased estimator.
Suppose the data consists of
observations
. Each observation
includes a scalar response
and a column vector
of
parameters (regressors), i.e.,
. In a
linear regression model
, the response variable,
, is a linear function of the regressors:
or in
vector
form,
where
, as introduced previously, is a column vector of the
-th observation of all the explanatory variables;
is a
vector of unknown parameters; and the scalar
represents unobserved random variables (
errors
) of the
-th observation.
accounts for the influences upon the responses
from sources other than the explanatory variables
. This model can also be written in matrix notation as
where
and
are
vectors of the response variables and the errors of the
observations, and
is an
matrix of regressors, also sometimes called the
design matrix
, whose row
is
and contains the
-th observations on all the explanatory variables.
Typically, a constant term is included in the set of regressors
, say, by taking
for all
. The coefficient
corresponding to this regressor is called the
intercept
. Without the intercept, the fitted line is forced to cross the origin when
.
Regressors do not have to be independent for estimation to be consistent e.g. they may be non-linearly dependent. Short of perfect multicollinearity, parameter estimates may still be consistent; however, as multicollinearity rises the standard error around such estimates increases and reduces the precision of such estimates. When there is perfect multicollinearity, it is no longer possible to obtain unique estimates for the coefficients to the related regressors; estimation for these parameters cannot converge (thus, it cannot be consistent).
As a concrete example where regressors are non-linearly dependent yet estimation may still be consistent, we might suspect the response depends linearly both on a value and its square; in which case we would include one regressor whose value is just the square of another regressor. In that case, the model would be
quadratic
in the second regressor, but none-the-less is still considered a
linear
model because the model
is
still linear in the parameters (
).
Matrix/vector formulation
[
edit
]
Consider an
overdetermined system
of
linear equations
in
unknown
coefficients
,
, with
. This can be written in
matrix
form as
where
(Note: for a linear model as above, not all elements in
contains information on the data points. The first column is populated with ones,
. Only the other columns contain actual data. So here
is equal to the number of regressors plus one).
Such a system usually has no exact solution, so the goal is instead to find the coefficients
which fit the equations "best", in the sense of solving the
quadratic
minimization
problem
where the objective function
is given by
A justification for choosing this criterion is given in
Properties
below. This minimization problem has a unique solution, provided that the
columns of the matrix
are
linearly independent
, given by solving the so-called
normal equations
:
The matrix
is known as the
normal matrix
or
Gram matrix
and the matrix
is known as the
moment matrix
of regressand by regressors.
[
3
]
Finally,
is the coefficient vector of the least-squares
hyperplane
, expressed as
or
Suppose
b
is a "candidate" value for the parameter vector
Ī²
. The quantity
y
i
ā
x
i
T
b
, called the
residual
for the
i
-th observation, measures the vertical distance between the data point
(
x
i
,
y
i
)
and the hyperplane
y
=
x
T
b
, and thus assesses the degree of fit between the actual data and the model. The
sum of squared residuals
(
SSR
) (also called the
error sum of squares
(
ESS
) or
residual sum of squares
(
RSS
))
[
4
]
is a measure of the overall model fit:
where
T
denotes the matrix
transpose
, and the rows of
X
, denoting the values of all the independent variables associated with a particular value of the dependent variable, are
X
i
= x
i
T
. The value of
b
which minimizes this sum is called the
OLS estimator for
Ī²
. The function
S
(
b
) is quadratic in
b
with positive-definite
Hessian
, and therefore this function possesses a unique global minimum at
, which can be given by the explicit formula
[
5
]
[proof]
The product
N
=
X
T
X
is a
Gram matrix
, and its inverse,
Q
=
N
ā1
, is the
cofactor matrix
of
Ī²
,
[
6
]
[
7
]
[
8
]
closely related to its
covariance matrix
,
C
Ī²
.
The matrix (
X
T
Ā
X
)
ā1
Ā
X
T
=
Q
Ā
X
T
is called the
MooreāPenrose pseudoinverse
matrix of
X
. This formulation highlights the point that estimation can be carried out if, and only if, there is no perfect
multicollinearity
between the explanatory variables (which would cause the Gram matrix to have no inverse).
After we have estimated
Ī²
, the
fitted values
(or
predicted values
) from the regression will be
where
P
=
X
(
X
T
X
)
ā1
X
T
is the
projection matrix
onto the space
V
spanned by the columns of
X
. This matrix
P
is also sometimes called the
hat matrix
because it "puts a hat" onto the variable
y
. Another matrix, closely related to
P
is the
annihilator
matrix
M
=
I
n
ā
P
; this is a projection matrix onto the space orthogonal to
V
. Both matrices
P
and
M
are
symmetric
and
idempotent
(meaning that
P
2
=
P
and
M
2
=
M
), and relate to the data matrix
X
via identities
PX
=
X
and
MX
= 0
.
[
9
]
Matrix
M
creates the
residuals
from the regression:
The variances of the predicted values
are found in the main diagonal of the
variance-covariance matrix
of predicted values:
where
P
is the projection matrix and
s
2
is the sample variance.
[
10
]
The full matrix is very large; its diagonal elements can be calculated individually as:
where
X
i
is the
i
-th row of matrix
X
.
Using these residuals we can estimate the sample variance
s
2
using the
reduced chi-squared
statistic:
The denominator,
n
ā
p
, is the
statistical degrees of freedom
. The first quantity,
s
2
, is the OLS estimate for
Ļ
2
, whereas the second,
, is the MLE estimate for
Ļ
2
. The two estimators are quite similar in large samples; the first estimator is always
unbiased
, while the second estimator is biased but has a smaller
mean squared error
. In practice
s
2
is used more often, since it is more convenient for the hypothesis testing. The square root of
s
2
is called the
regression standard error
,
[
11
]
standard error of the regression
,
[
12
]
[
13
]
or
standard error of the equation
.
[
9
]
It is common to assess the goodness-of-fit of the OLS regression by comparing how much the initial variation in the sample can be reduced by regressing onto
X
. The
coefficient of determination
R
2
is defined as a ratio of "explained" variance to the "total" variance of the dependent variable
y
, in the cases where the regression sum of squares equals the sum of squares of residuals:
[
14
]
where TSS is the
total sum of squares
for the dependent variable,
, and
is an
n
Ć
n
matrix of ones. (
is a
centering matrix
which is equivalent to regression on a constant; it simply subtracts the mean from a variable.) In order for
R
2
to be meaningful, the matrix
X
of data on regressors must contain a column vector of ones to represent the constant whose coefficient is the regression intercept. In that case,
R
2
will always be a number between 0 and 1, with values close to 1 indicating a good degree of fit.
Simple linear regression model
[
edit
]
If the data matrix
X
contains only two variables, a constant and a scalar regressor
x
i
, then this is called the "simple regression model". This case is often considered in the beginner statistics classes, as it provides much simpler formulas even suitable for manual calculation. The parameters are commonly denoted as
(
Ī±
,
Ī²
)
:
The least squares estimates in this case are given by simple formulas
Alternative derivations
[
edit
]
In the previous section the least squares estimator
was obtained as a value that minimizes the sum of squared residuals of the model. However it is also possible to derive the same estimator from other approaches. In all cases the formula for OLS estimator remains the same:
^
Ī²
= (
X
T
X
)
ā1
X
T
y
; the only difference is in how we interpret this result.
OLS estimation can be viewed as a projection onto the linear space spanned by the regressors. (Here each of
and
refers to a column of the data matrix.)
Least squares as projection of y onto col(X) for three observations; Å· = XĪ² gives fitted values and y ā Å· is the residual.
For mathematicians, OLS is an approximate solution to an overdetermined system of linear equations
XĪ²
ā
y
, where
Ī²
is the unknown. Assuming the system cannot be solved exactly (the number of equations
n
is much larger than the number of unknowns
p
), we are looking for a solution that could provide the smallest discrepancy between the right- and left- hand sides. In other words, we are looking for the solution that satisfies
where
ā
Ā·
ā
is the standard
L
2
Ā norm
in the
n
-dimensional
Euclidean space
R
n
. The predicted quantity
XĪ²
is just a certain linear combination of the vectors of regressors. Thus, the residual vector
y
ā
XĪ²
will have the smallest length when
y
is
projected orthogonally
onto the
linear subspace
spanned
by the columns of
X
. The OLS estimator
in this case can be interpreted as the coefficients of
vector decomposition
of
^
y
=
Py
along the basis of
X
.
In other words, the gradient equations at the minimum can be written as:
A geometrical interpretation of these equations is that the vector of residuals,
is orthogonal to the
column space
of
X
, since the
dot product
is equal to zero for
any
conformal vector,
v
. This means that
is the shortest of all possible vectors
, that is, the variance of the residuals is the minimum possible. This is illustrated at the right.
Introducing
and a matrix
K
with the assumption that a matrix
is non-singular and
K
T
X
= 0 (cf.
Orthogonal projections
), the residual vector should satisfy the following equation:
The equation and solution of linear least squares are thus described as follows:
Another way of looking at it is to consider the regression line to be a weighted average of the lines passing through the combination of any two points in the dataset.
[
15
]
Although this way of calculation is more computationally expensive, it provides a better intuition on OLS.
The OLS estimator is identical to the
maximum likelihood estimator
(MLE) under the normality assumption for the error terms.
[
16
]
[proof]
This normality assumption has historical importance, as it provided the basis for the early work in linear regression analysis by
Yule
and
Pearson
.
[
citation needed
]
From the properties of MLE, we can infer that the OLS estimator is asymptotically efficient (in the sense of attaining the
CramĆ©rāRao bound
for variance) if the normality assumption is satisfied.
[
17
]
Generalized method of moments
[
edit
]
In
iid
case the OLS estimator can also be viewed as a
GMM
estimator arising from the moment conditions
These moment conditions state that the regressors should be uncorrelated with the errors. Since
x
i
is a
p
-vector, the number of moment conditions is equal to the dimension of the parameter vector
Ī²
, and thus the system is exactly identified. This is the so-called classical GMM case, when the estimator does not depend on the choice of the weighting matrix.
Note that the original strict exogeneity assumption
E[
Īµ
i
ā|ā
x
i
] = 0
implies a far richer set of moment conditions than stated above. In particular, this assumption implies that for any vector-function
Ę
, the moment condition
E[
Ę
(
x
i
)Ā·
Īµ
i
] = 0
will hold. However it can be shown using the
GaussāMarkov theorem
that the optimal choice of function
Ę
is to take
Ę
(
x
) =
x
, which results in the moment equation posted above.
There are several different frameworks in which the
linear regression model
can be cast in order to make the OLS technique applicable. Each of these settings produces the same formulas and same results. The only difference is the interpretation and the assumptions which have to be imposed in order for the method to give meaningful results. The choice of the applicable framework depends mostly on the nature of data in hand, and on the inference task which has to be performed.
One of the lines of difference in interpretation is whether to treat the regressors as random variables, or as predefined constants. In the first case (
random design
) the regressors
x
i
are random and sampled together with the
y
i
'
s from some
population
, as in an
observational study
. This approach allows for more natural study of the
asymptotic properties
of the estimators. In the other interpretation (
fixed design
), the regressors
X
are treated as known constants set by a
design
, and
y
is sampled conditionally on the values of
X
as in an
experiment
. For practical purposes, this distinction is often unimportant, since estimation and inference is carried out while conditioning on
X
. All results stated in this article are within the random design framework.
The classical model focuses on the "finite sample" estimation and inference, meaning that the number of observations
n
is fixed. This contrasts with the other approaches, which study the
asymptotic behavior
of OLS, and in which the behavior at a large number of samples is studied.
To prove finite sample unbiasedness of the OLS estimator, we require the following assumptions.
Example of a cubic polynomial regression, which is a type of linear regression. Although
polynomial regression
fits a curve model to the data, as a
statistical estimation
problem it is linear, in the sense that the conditional expectation function
is linear in the unknown
parameters
that are estimated from the
data
. For this reason, polynomial regression is considered to be a special case of
multiple linear regression
.
Exogeneity
. The regressors do not
covary
with the error term:
This requires, for example, that there are no
omitted variables
that covary with observed variables and affect the response variable. An alternative (but stronger) statement that is often required when explaining linear regression in
mathematical statistics
is that the predictor variables
x
can be treated as fixed values, rather than
random variables
. This stronger form means, for example, that the predictor variables are assumed to be error-free, that is, not contaminated with measurement error. Although this assumption is not realistic in many settings, dropping it leads to more complex
errors-in-variables models
,
instrumental variable models
and the like.
Linearity
, or
correct specification
. This means that the mean of the response variable is a
linear combination
of the parameters (regression coefficients) and the predictor variables. Note that this assumption is much less restrictive than it may at first seem. Because the predictor variables are treated as fixed values (see above), linearity is really only a restriction on the parameters. The predictor variables themselves can be arbitrarily transformed, and in fact multiple copies of the same underlying predictor variable can be added, each one transformed differently. This technique is used, for example, in
polynomial regression
, which uses linear regression to fit the response variable as an arbitrary
polynomial
function (up to a given degree) of a predictor variable. With this much flexibility, models such as polynomial regression often have "too much power", in that they tend to
overfit
the data. As a result, some kind of
regularization
must typically be used to prevent unreasonable solutions coming out of the estimation process. Common examples are
ridge regression
and
lasso regression
.
Bayesian linear regression
can also be used, which by its nature is more or less immune to the problem of overfitting. (In fact,
ridge regression
and
lasso regression
can both be viewed as special cases of Bayesian linear regression, with particular types of
prior distributions
placed on the regression coefficients.)
Visualization of heteroscedasticity in a scatter plot against 100 random fitted values using Matlab
Constant variance
or
homoscedasticity
. This means that the variance of the errors does not depend on the values of the predictor variables:
Thus the variability of the responses for given fixed values of the predictors is the same regardless of how large or small the responses are. This is often not the case, as a variable whose mean is large will typically have a greater variance than one whose mean is small. For example, a person whose income is predicted to be $100,000 may easily have an actual income of $80,000 or $120,000āi.e., a
standard deviation
of around $20,000āwhile another person with a predicted income of $10,000 is unlikely to have the same $20,000 standard deviation, since that would imply their actual income could vary anywhere between ā$10,000 and $30,000. (In fact, as this shows, in many casesāoften the same cases where the assumption of normally distributed errors failsāthe variance or standard deviation should be predicted to be proportional to the mean, rather than constant.) The absence of homoscedasticity is called
heteroscedasticity
. In order to check this assumption, a plot of residuals versus predicted values (or the values of each individual predictor) can be examined for a "fanning effect" (i.e., increasing or decreasing vertical spread as one moves left to right on the plot). A plot of the absolute or squared residuals versus the predicted values (or each predictor) can also be examined for a trend or curvature. Formal tests can also be used; see
Heteroscedasticity
. The presence of heteroscedasticity will result in an overall "average" estimate of variance being used instead of one that takes into account the true variance structure. This leads to less precise (but in the case of
ordinary least squares
, not biased) parameter estimates and biased standard errors, resulting in misleading tests and interval estimates. The
mean squared error
for the model will also be wrong. Various estimation techniques including
weighted least squares
and the use of
heteroscedasticity-consistent standard errors
can handle heteroscedasticity in a quite general way.
Bayesian linear regression
techniques can also be used when the variance is assumed to be a function of the mean. It is also possible in some cases to fix the problem by applying a transformation to the response variable (e.g., fitting the
logarithm
of the response variable using a linear regression model, which implies that the response variable itself has a
log-normal distribution
rather than a
normal distribution
).
To check for violations of the assumptions of linearity, constant variance, and independence of errors within a linear regression model, the residuals are typically plotted against the predicted values (or each of the individual predictors). An apparently random scatter of points about the horizontal midline at 0 is ideal, but cannot rule out certain kinds of violations such as
autocorrelation
in the errors or their correlation with one or more covariates.
Uncorrelatedness of errors
. This assumes that the errors of the response variables are uncorrelated with each other:
Some methods such as
generalized least squares
are capable of handling correlated errors, although they typically require significantly more data unless some sort of
regularization
is used to bias the model towards assuming uncorrelated errors.
Bayesian linear regression
is a general way of handling this issue. Full
statistical independence
is a stronger condition than mere lack of correlation and is often not needed, although it implies mean-independence.
Lack of perfect multicollinearity
in the predictors. For standard
least squares
estimation methods, the design matrix
X
must have full
column rank
p
:
[
18
]
If this assumption is violated, perfect
multicollinearity
exists in the predictor variables, meaning a linear relationship exists between two or more predictor variables. Multicollinearity can be caused by accidentally duplicating a variable in the data, using a linear transformation of a variable along with the original (e.g., the same temperature measurements expressed in Fahrenheit and Celsius), or including a linear combination of multiple variables in the model, such as their mean. It can also happen if there is too little data available compared to the number of parameters to be estimated (e.g., fewer data points than regression coefficients). Near violations of this assumption, where predictors are highly but not perfectly correlated, can reduce the precision of parameter estimates (see
Variance inflation factor
). In the case of perfect multicollinearity, the parameter vector
Ī²
will be
non-identifiable
āit has no unique solution. In such a case, only some of the parameters can be identified (i.e., their values can only be estimated within some linear subspace of the full parameter space
R
p
). See
partial least squares regression
. Methods for fitting linear models with multicollinearity have been developed,
[
19
]
[
20
]
[
21
]
[
22
]
some of which require additional assumptions such as "effect sparsity"āthat a large fraction of the effects are exactly zero. Note that the more computationally expensive iterated algorithms for parameter estimation, such as those used in
generalized linear models
, do not suffer from this problem.
Violations of these assumptions can result in biased estimations of
Ī²
, biased standard errors, untrustworthy confidence intervals and significance tests. Beyond these assumptions, several other statistical properties of the data strongly influence the performance of different estimation methods:
The statistical relationship between the error terms and the regressors plays an important role in determining whether an estimation procedure has desirable sampling properties such as being unbiased and consistent.
The arrangement, or
probability distribution
of the predictor variables
x
has a major influence on the precision of estimates of
Ī²
.
Sampling
and
design of experiments
are highly developed subfields of statistics that provide guidance for collecting data in such a way to achieve a precise estimate of
Ī²
.
Finite sample properties
[
edit
]
First of all, under the
strict exogeneity
assumption the OLS estimators
and
s
2
are
unbiased
, meaning that their expected values coincide with the true values of the parameters:
[
23
]
[proof]
If the strict exogeneity does not hold (as is the case with many
time series
models, where exogeneity is assumed only with respect to the past shocks but not the future ones), then these estimators will be biased in finite samples.
The
variance-covariance matrix
(or simply
covariance matrix
) of
is equal to
[
24
]
In particular, the standard error of each coefficient
is equal to square root of the
j
-th diagonal element of this matrix. The estimate of this standard error is obtained by replacing the unknown quantity
Ļ
2
with its estimate
s
2
. Thus,
It can also be easily shown that the estimator
is uncorrelated with the residuals from the model:
[
24
]
The
GaussāMarkov theorem
states that under the
spherical errors
assumption (that is, the errors should be
uncorrelated
and
homoscedastic
) the estimator
is efficient in the class of linear unbiased estimators. This is called the
best linear unbiased estimator
(BLUE). Efficiency should be understood as if we were to find some other estimator
which would be linear in
y
and unbiased, then
[
24
]
in the sense that this is a
nonnegative-definite matrix
. This theorem establishes optimality only in the class of linear unbiased estimators, which is quite restrictive. Depending on the distribution of the error terms
Īµ
, other, non-linear estimators may provide better results than OLS.
The properties listed so far are all valid regardless of the underlying distribution of the error terms. However, if you are willing to assume that the
normality assumption
holds (that is, that
Īµ
~
N
(0,
Ļ
2
I
n
)
), then additional properties of the OLS estimators can be stated.
The estimator
is normally distributed, with mean and variance as given before:
[
25
]
This estimator reaches the
CramĆ©rāRao bound
for the model, and thus is optimal in the class of all unbiased estimators.
[
17
]
Note that unlike the
GaussāMarkov theorem
, this result establishes optimality among both linear and non-linear estimators, but only in the case of normally distributed error terms.
The estimator
s
2
will be proportional to the
chi-squared distribution
:
[
26
]
The variance of this estimator is equal to
2
Ļ
4
/(
n
āāā
p
)
, which does not attain the
CramĆ©rāRao bound
of
2
Ļ
4
/
n
. However it was shown that there are no unbiased estimators of
Ļ
2
with variance smaller than that of the estimator
s
2
.
[
27
]
If we are willing to allow biased estimators, and consider the class of estimators that are proportional to the sum of squared residuals (SSR) of the model, then the best (in the sense of the
mean squared error
) estimator in this class will be
~
Ļ
2
= SSRā
/
ā(
n
āāā
p
ā+ā2)
, which even beats the CramĆ©rāRao bound in case when there is only one regressor (
p
= 1
).
[
28
]
Moreover, the estimators
and
s
2
are
independent
,
[
29
]
the fact which comes in useful when constructing the t- and F-tests for the regression.
Influential observations
[
edit
]
As was mentioned before, the estimator
is linear in
y
, meaning that it represents a linear combination of the dependent variables
y
i
. The weights in this linear combination are functions of the regressors
X
, and generally are unequal. The observations with high weights are called
influential
because they have a more pronounced effect on the value of the estimator.
To analyze which observations are influential we remove a specific
j
-th observation and consider how much the estimated quantities are going to change (similarly to the
jackknife method
). It can be shown that the change in the OLS estimator for
Ī²
will be equal to
[
30
]
where
h
j
=
x
j
T
ā(
X
T
X
)
ā1
x
j
is the
j
-th diagonal element of the hat matrix
P
, and
x
j
is the vector of regressors corresponding to the
j
-th observation. Similarly, the change in the predicted value for
j
-th observation resulting from omitting that observation from the dataset will be equal to
[
30
]
From the properties of the hat matrix,
0 ā¤
h
j
ā¤ 1
, and they sum up to
p
, so that on average
h
j
ā
p/n
. These quantities
h
j
are called the
leverages
, and observations with high
h
j
are called
leverage points
.
[
31
]
Usually the observations with high leverage ought to be scrutinized more carefully, in case they are erroneous, or outliers, or in some other way atypical of the rest of the dataset.
Partitioned regression
[
edit
]
Sometimes the variables and corresponding parameters in the regression can be logically split into two groups, so that the regression takes form
where
X
1
and
X
2
have dimensions
n
Ć
p
1
,
n
Ć
p
2
, and
Ī²
1
,
Ī²
2
are
p
1
Ć1 and
p
2
Ć1 vectors, with
p
1
+
p
2
=
p
.
The
FrischāWaughāLovell theorem
states that in this regression the residuals
and the OLS estimate
will be numerically identical to the residuals and the OLS estimate for
Ī²
2
in the following regression:
[
32
]
where
M
1
is the
annihilator matrix
for regressors
X
1
.
The theorem can be used to establish a number of theoretical results. For example, having a regression with a constant and another regressor is equivalent to subtracting the means from the dependent variable and the regressor and then running the regression for the de-meaned variables but without the constant term.
Large sample properties
[
edit
]
The least squares estimators are
point estimates
of the linear regression model parameters
Ī²
. However, generally we also want to know how close those estimates might be to the true values of parameters. In other words, we want to construct the
interval estimates
.
Since we have not made any assumption about the distribution of error term
Īµ
i
, it is impossible to infer the distribution of the estimators
and
. Nevertheless, we can apply the
central limit theorem
to derive their
asymptotic
properties as sample size
n
goes to infinity. While the sample size is necessarily finite, it is customary to assume that
n
is "large enough" so that the true distribution of the OLS estimator is close to its asymptotic limit.
We can show that under the model assumptions, the least squares estimator for
Ī²
is
consistent
(that is
converges in probability
to
Ī²
) and asymptotically normal:
[proof]
where
Using this asymptotic distribution, approximate two-sided confidence intervals for the
j
-th component of the vector
can be constructed as
Ā at the
1Ā āĀ
Ī±
confidence level,
where
q
denotes the
quantile function
of standard normal distribution, and [Ā·]
jj
is the
j
-th diagonal element of a matrix.
Similarly, the least squares estimator for
Ļ
2
is also consistent and asymptotically normal (provided that the fourth moment of
Īµ
i
exists) with limiting distribution
These asymptotic distributions can be used for prediction, testing hypotheses, constructing other estimators, etc.. As an example consider the problem of prediction. Suppose
is some point within the domain of distribution of the regressors, and one wants to know what the response variable would have been at that point. The
mean response
is the quantity
, whereas the
predicted response
is
. Clearly the predicted response is a random variable, its distribution can be derived from that of
:
which allows construct confidence intervals for mean response
to be constructed:
Ā at the
1Ā āĀ
Ī±
confidence level.
Two hypothesis tests are particularly widely used. First, one wants to know if the estimated regression equation is any better than simply predicting that all values of the response variable equal its sample mean (if not, it is said to have no explanatory power). The
null hypothesis
of no explanatory value of the estimated regression is tested using an
F-test
. If the calculated F-value is found to be large enough to exceed its critical value for the pre-chosen level of significance, the null hypothesis is rejected and the
alternative hypothesis
, that the regression has explanatory power, is accepted. Otherwise, the null hypothesis of no explanatory power is accepted.
Second, for each explanatory variable of interest, one wants to know whether its estimated coefficient differs significantly from zeroāthat is, whether this particular explanatory variable in fact has explanatory power in predicting the response variable. Here the null hypothesis is that the true coefficient is zero. This hypothesis is tested by computing the coefficient's
t-statistic
, as the ratio of the coefficient estimate to its
standard error
. If the t-statistic is larger than a predetermined value, the null hypothesis is rejected and the variable is found to have explanatory power, with its coefficient significantly different from zero. Otherwise, the null hypothesis of a zero value of the true coefficient is accepted.
In addition, the
Chow test
is used to test whether two subsamples both have the same underlying true coefficient values. The sum of squared residuals of regressions on each of the subsets and on the combined data set are compared by computing an F-statistic; if this exceeds a critical value, the null hypothesis of no difference between the two subsets is rejected; otherwise, it is accepted.
Violations of assumptions
[
edit
]
In a
time series
model, we require the
stochastic process
{
x
i
,
y
i
} to be
stationary
and
ergodic
; if {
x
i
,
y
i
} is nonstationary, OLS results are often biased unless {
x
i
,
y
i
} is
co-integrating
.
[
33
]
We still require the regressors to be
strictly exogenous
: E[
x
i
Īµ
i
] = 0 for all
i
= 1, ...,
n
. If they are only
predetermined
, OLS is biased in finite sample;
Finally, the assumptions on the variance take the form of requiring that {
x
i
Īµ
i
} is a
martingale difference sequence
, with a finite matrix of second moments
Q
xxĪµ
Ā²
= E[ā
Īµ
i
2
x
i
āx
i
T
ā]
.
Constrained estimation
[
edit
]
Suppose it is known that the coefficients in the regression satisfy a system of linear equations
where
Q
is a
p
Ć
q
matrix of full rank, and
c
is a
q
Ć1 vector of known constants, where
qā<āp
. In this case least squares estimation is equivalent to minimizing the sum of squared residuals of the model subject to the constraint
A
. The
constrained least squares (CLS)
estimator can be given by an explicit formula:
[
34
]
This expression for the constrained estimator is valid as long as the matrix
X
T
X
is invertible. It was assumed from the beginning of this article that this matrix is of full rank, and it was noted that when the rank condition fails,
Ī²
will not be identifiable. However it may happen that adding the restriction
A
makes
Ī²
identifiable, in which case one would like to find the formula for the estimator. The estimator is equal to
[
35
]
where
R
is a
p
Ć(
p
Ā āĀ
q
) matrix such that the matrix
[
Q R
]
is non-singular, and
R
T
Q
= 0
. Such a matrix can always be found, although generally it is not unique. The second formula coincides with the first in case when
X
T
X
is invertible.
[
35
]
Example with real data
[
edit
]
The following data set gives average heights and weights for American women aged 30ā39 (source:
The World Almanac and Book of Facts, 1975
).
Height (m)
1.47
1.50
1.52
1.55
1.57
Scatterplot
of the data, the relationship is slightly curved but close to linear
Weight (kg)
52.21
53.12
54.48
55.84
57.20
Height (m)
1.60
1.63
1.65
1.68
1.70
Weight (kg)
58.57
59.93
61.29
63.11
64.47
Height (m)
1.73
1.75
1.78
1.80
1.83
Weight (kg)
66.28
68.10
69.92
72.19
74.46
When only one dependent variable is being modeled, a
scatterplot
will suggest the form and strength of the relationship between the dependent variable and regressors. It might also reveal outliers, heteroscedasticity, and other aspects of the data that may complicate the interpretation of a fitted regression model. The scatterplot suggests that the relationship is strong and can be approximated as a quadratic function. OLS can handle non-linear relationships by introducing the regressor
HEIGHT
2
. The regression model then becomes a multiple linear model:
Fitted regression
The output from most popular
statistical packages
will look similar to this:
Method
Least squares
Dependent variable
WEIGHT
Observations
15
Parameter
Value
Std error
t-statistic
p-value
128.8128
16.3083
7.8986
0.0000
ā143.1620
19.8332
ā7.2183
0.0000
61.9603
6.0084
10.3122
0.0000
R
2
0.9989
S.E. of regression
0.2516
Adjusted R
2
0.9987
Model sum-of-sq.
692.61
Log-likelihood
1.0890
Residual sum-of-sq.
0.7595
DurbināWatson stat.
2.1013
Total sum-of-sq.
693.37
Akaike criterion
0.2548
F-statistic
5471.2
Schwarz criterion
0.3964
p-value (F-stat)
0.0000
In this table:
The
Value
column gives the least squares estimates of parameters
Ī²
j
The
Std error
column shows
standard errors
of each coefficient estimate:
The
t-statistic
and
p-value
columns are testing whether any of the coefficients might be equal to zero. The
t
-statistic is calculated simply as
. If the errors Īµ follow a normal distribution,
t
follows a Student-t distribution. Under weaker conditions,
t
is asymptotically normal. Large values of
t
indicate that the null hypothesis can be rejected and that the corresponding coefficient is not zero. The second column,
p
-value
, expresses the results of the hypothesis test as a
significance level
. Conventionally,
p
-values smaller than 0.05 are taken as evidence that the population coefficient is nonzero.
R-squared
is the
coefficient of determination
indicating goodness-of-fit of the regression. This statistic will be equal to one if fit is perfect, and to zero when regressors
X
have no explanatory power whatsoever. This is a biased estimate of the population
R-squared
, and will never decrease if additional regressors are added, even if they are irrelevant.
Adjusted R-squared
is a slightly modified version of
, designed to penalize for the excess number of regressors which do not add to the explanatory power of the regression. This statistic is always smaller than
, can decrease as new regressors are added, and even be negative for poorly fitting models:
Log-likelihood
is calculated under the assumption that errors follow normal distribution. Even though the assumption is not very reasonable, this statistic may still find its use in conducting LR tests.
DurbināWatson statistic
tests whether there is any evidence of serial correlation between the residuals. As a rule of thumb, the value smaller than 2 will be an evidence of positive correlation.
Akaike information criterion
and
Schwarz criterion
are both used for model selection. Generally when comparing two alternative models, smaller values of one of these criteria will indicate a better model.
[
36
]
Standard error of regression
is an estimate of
Ļ
, standard error of the error term.
Total sum of squares
,
model sum of squared
, and
residual sum of squares
tell us how much of the initial variation in the sample were explained by the regression.
F-statistic
tries to test the hypothesis that all coefficients (except the intercept) are equal to zero. This statistic has
F
(
pā1
,
nāp
) distribution under the null hypothesis and normality assumption, and its
p-value
indicates probability that the hypothesis is indeed true. Note that when errors are not normal this statistic becomes invalid, and other tests such as
Wald test
or
LR test
should be used.
Residuals plot
Ordinary least squares analysis often includes the use of diagnostic plots designed to detect departures of the data from the assumed form of the model. These are some of the common diagnostic plots:
Residuals against the explanatory variables in the model. A non-linear relation between these variables suggests that the linearity of the conditional mean function may not hold. Different levels of variability in the residuals for different levels of the explanatory variables suggests possible heteroscedasticity.
Residuals against explanatory variables not in the model. Any relation of the residuals to these variables would suggest considering these variables for inclusion in the model.
Residuals against the fitted values,
.
Residuals against the preceding residual. This plot may identify serial correlations in the residuals.
An important consideration when carrying out statistical inference using regression models is how the data were sampled. In this example, the data are averages rather than measurements on individual women. The fit of the model is very good, but this does not imply that the weight of an individual woman can be predicted with high accuracy based only on her height.
Sensitivity to rounding
[
edit
]
This example also demonstrates that coefficients determined by these calculations are sensitive to how the data is prepared. The heights were originally given rounded to the nearest inch and have been converted and rounded to the nearest centimetre. Since the conversion factor is one inch to 2.54Ā cm this is
not
an exact conversion. The original inches can be recovered by Round(x/0.0254) and then re-converted to metric without rounding. If this is done the results become:
Const
Height
Height
2
Converted to metric with rounding.
128.8128
ā143.162
61.96033
Converted to metric without rounding.
119.0205
ā131.5076
58.5046
Residuals to a quadratic fit for correctly and incorrectly converted data.
Using either of these equations to predict the weight of a 5' 6" (1.6764 m) woman gives similar values: 62.94Ā kg with rounding vs. 62.98Ā kg without rounding. Thus a seemingly small variation in the data has a real effect on the coefficients but a small effect on the results of the equation.
While this may look innocuous in the middle of the data range it could become significant at the extremes or in the case where the fitted model is used to project outside the data range (
extrapolation
).
This highlights a common error: this example is an abuse of OLS which inherently requires that the errors in the independent variable (in this case height) are zero or at least negligible. The initial rounding to nearest inch plus any actual measurement errors constitute a finite and non-negligible error. As a result, the fitted parameters are not the best estimates they are presumed to be. Though not totally spurious the error in the estimation will depend upon relative size of the
x
and
y
errors.
Another example with less real data
[
edit
]
We can use the least square mechanism to figure out the equation of a two body orbit in polar base co-ordinates. The equation typically used is
where
is the radius of how far the object is from one of the bodies. In the equation the parameters
and
are used to determine the path of the orbit. We have measured the following data.
(in degrees)
43
45
52
93
108
116
4.7126
4.5542
4.0419
2.2187
1.8910
1.7599
We need to find the least-squares approximation of
and
for the given data.
First we need to represent e and p in a linear form. So we are going to rewrite the equation
as
.
Furthermore, one could fit for
apsides
by expanding
with an extra parameter as
, which is linear in both
and in the extra basis function
.
We use the original two-parameter form to represent our observational data as:
where:
;
;
contains the coefficients of
in the first column, which are all 1, and the coefficients of
in the second column, given by
; and
, such that:
On solving we get
,
so
and
Bayesian least squares
FamaāMacBeth regression
Nonlinear least squares
Numerical methods for linear least squares
Nonlinear system identification
^
"The Origins of Ordinary Least Squares Assumptions"
.
Feature Column
. 2022-03-01
. Retrieved
2024-05-16
.
^
"What is a complete list of the usual assumptions for linear regression?"
.
Cross Validated
. Retrieved
2022-09-28
.
^
Goldberger, Arthur S.
(1964).
"Classical Linear Regression"
.
Econometric Theory
. New York: John Wiley & Sons. pp.Ā
158
.
ISBN
Ā
0-471-31101-4
.
^
Hayashi, Fumio
(2000).
Econometrics
. Princeton University Press. p.Ā 15.
ISBN
Ā
9780691010182
.
^
Hayashi (2000
, page 18).
^
Ghilani, Charles D.; Wolf, Paul R. (12 June 2006).
Adjustment Computations: Spatial Data Analysis
. John Wiley & Sons.
ISBN
Ā
9780471697282
.
^
Hofmann-Wellenhof, Bernhard; Lichtenegger, Herbert; Wasle, Elmar (20 November 2007).
GNSS ā Global Navigation Satellite Systems: GPS, GLONASS, Galileo, and more
. Springer.
ISBN
Ā
9783211730171
.
^
Xu, Guochang (5 October 2007).
GPS: Theory, Algorithms and Applications
. Springer.
ISBN
Ā
9783540727156
.
^
a
b
Hayashi (2000
, page 19)
^
Hoaglin, David C.; Welsch, Roy E. (1978).
"The Hat Matrix in Regression and ANOVA"
.
The American Statistician
.
32
(1):
17ā
22.
doi
:
10.1080/00031305.1978.10479237
.
hdl
:
1721.1/1920
.
ISSN
Ā
0003-1305
.
^
Julian Faraway (2000),
Practical Regression and Anova using R
^
Kenney, J.; Keeping, E. S. (1963).
Mathematics of Statistics
. van Nostrand. p.Ā 187.
^
Zwillinger, Daniel (1995).
Standard Mathematical Tables and Formulae
. Chapman&Hall/CRC. p.Ā 626.
ISBN
Ā
0-8493-2479-3
.
^
Hayashi (2000
, page 20)
^
Akbarzadeh, Vahab (7 May 2014).
"Line Estimation"
.
^
Hayashi (2000
, page 49)
^
a
b
Hayashi (2000
, page 52)
^
Hayashi (2000
, page 10)
^
Tibshirani, Robert (1996). "Regression Shrinkage and Selection via the Lasso".
Journal of the Royal Statistical Society, Series B
.
58
(1):
267ā
288.
doi
:
10.1111/j.2517-6161.1996.tb02080.x
.
JSTOR
Ā
2346178
.
^
Efron, Bradley; Hastie, Trevor; Johnstone, Iain; Tibshirani, Robert (2004). "Least Angle Regression".
The Annals of Statistics
.
32
(2):
407ā
451.
arXiv
:
math/0406456
.
doi
:
10.1214/009053604000000067
.
JSTOR
Ā
3448465
.
S2CID
Ā
204004121
.
^
Hawkins, Douglas M. (1973). "On the Investigation of Alternative Regressions by Principal Component Analysis".
Journal of the Royal Statistical Society, Series C
.
22
(3):
275ā
286.
doi
:
10.2307/2346776
.
JSTOR
Ā
2346776
.
^
Jolliffe, Ian T. (1982). "A Note on the Use of Principal Components in Regression".
Journal of the Royal Statistical Society, Series C
.
31
(3):
300ā
303.
doi
:
10.2307/2348005
.
JSTOR
Ā
2348005
.
^
Hayashi (2000
, pages 27, 30)
^
a
b
c
Hayashi (2000
, page 27)
^
Amemiya, Takeshi
(1985).
Advanced Econometrics
. Harvard University Press. p.Ā
13
.
ISBN
Ā
9780674005600
.
^
Amemiya (1985
, page 14)
^
Rao, C. R.
(1973).
Linear Statistical Inference and its Applications
(SecondĀ ed.). New York: J. Wiley & Sons. p.Ā 319.
ISBN
Ā
0-471-70823-2
.
^
Amemiya (1985
, page 20)
^
Amemiya (1985
, page 27)
^
a
b
Davidson, Russell;
MacKinnon, James G.
(1993).
Estimation and Inference in Econometrics
. New York: Oxford University Press. p.Ā 33.
ISBN
Ā
0-19-506011-3
.
^
Davidson & MacKinnon (1993
, page 36)
^
Davidson & MacKinnon (1993
, page 20)
^
"Memento on EViews Output"
(PDF)
. Retrieved
28 December
2020
.
^
Amemiya (1985
, page 21)
^
a
b
Amemiya (1985
, page 22)
^
Burnham, Kenneth P.; Anderson, David R. (2002).
Model Selection and Multi-Model Inference
(2ndĀ ed.). Springer.
ISBN
Ā
0-387-95364-7
.
Dougherty, Christopher
(2002).
Introduction to Econometrics
(2ndĀ ed.). New York: Oxford University Press. pp.Ā
48ā
113.
ISBN
Ā
0-19-877643-8
.
Gujarati, Damodar N.
;
Porter, Dawn C.
(2009).
Basic Econometics
(FifthĀ ed.). Boston: McGraw-Hill Irwin. pp.Ā
55ā
96.
ISBN
Ā
978-0-07-337577-9
.
Heij, Christiaan
; Boer, Paul;
Franses, Philip H.
;
Kloek, Teun
;
van Dijk, Herman K.
(2004).
Econometric Methods with Applications in Business and Economics
(1stĀ ed.). Oxford: Oxford University Press. pp.Ā
76ā
115.
ISBN
Ā
978-0-19-926801-6
.
Hill, R. Carter; Griffiths, William E.; Lim, Guay C. (2008).
Principles of Econometrics
(3rdĀ ed.). Hoboken, NJ: John Wiley & Sons. pp.Ā
8ā
47.
ISBN
Ā
978-0-471-72360-8
.
Wooldridge, Jeffrey
(2008).
"The Simple Regression Model"
.
Introductory Econometrics: A Modern Approach
(4thĀ ed.). Mason, OH: Cengage Learning. pp.Ā
22ā
67.
ISBN
Ā
978-0-324-58162-1
. | |||||||||
| Markdown | [Jump to content](https://en.wikipedia.org/wiki/Ordinary_least_squares#bodyContent)
Main menu
Main menu
move to sidebar
hide
Navigation
- [Main page](https://en.wikipedia.org/wiki/Main_Page "Visit the main page [z]")
- [Contents](https://en.wikipedia.org/wiki/Wikipedia:Contents "Guides to browsing Wikipedia")
- [Current events](https://en.wikipedia.org/wiki/Portal:Current_events "Articles related to current events")
- [Random article](https://en.wikipedia.org/wiki/Special:Random "Visit a randomly selected article [x]")
- [About Wikipedia](https://en.wikipedia.org/wiki/Wikipedia:About "Learn about Wikipedia and how it works")
- [Contact us](https://en.wikipedia.org/wiki/Wikipedia:Contact_us "How to contact Wikipedia")
Contribute
- [Help](https://en.wikipedia.org/wiki/Help:Contents "Guidance on how to use and edit Wikipedia")
- [Learn to edit](https://en.wikipedia.org/wiki/Help:Introduction "Learn how to edit Wikipedia")
- [Community portal](https://en.wikipedia.org/wiki/Wikipedia:Community_portal "The hub for editors")
- [Recent changes](https://en.wikipedia.org/wiki/Special:RecentChanges "A list of recent changes to Wikipedia [r]")
- [Upload file](https://en.wikipedia.org/wiki/Wikipedia:File_upload_wizard "Add images or other media for use on Wikipedia")
- [Special pages](https://en.wikipedia.org/wiki/Special:SpecialPages "A list of all special pages [q]")
[  ](https://en.wikipedia.org/wiki/Main_Page)
[Search](https://en.wikipedia.org/wiki/Special:Search "Search Wikipedia [f]")
Appearance
- [Donate](https://donate.wikimedia.org/?wmf_source=donate&wmf_medium=sidebar&wmf_campaign=en.wikipedia.org&uselang=en)
- [Create account](https://en.wikipedia.org/w/index.php?title=Special:CreateAccount&returnto=Ordinary+least+squares "You are encouraged to create an account and log in; however, it is not mandatory")
- [Log in](https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Ordinary+least+squares "You're encouraged to log in; however, it's not mandatory. [o]")
Personal tools
- [Donate](https://donate.wikimedia.org/?wmf_source=donate&wmf_medium=sidebar&wmf_campaign=en.wikipedia.org&uselang=en)
- [Create account](https://en.wikipedia.org/w/index.php?title=Special:CreateAccount&returnto=Ordinary+least+squares "You are encouraged to create an account and log in; however, it is not mandatory")
- [Log in](https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Ordinary+least+squares "You're encouraged to log in; however, it's not mandatory. [o]")
## Contents
move to sidebar
hide
- [(Top)](https://en.wikipedia.org/wiki/Ordinary_least_squares)
- [1 Linear model](https://en.wikipedia.org/wiki/Ordinary_least_squares#Linear_model)
Toggle Linear model subsection
- [1\.1 Matrix/vector formulation](https://en.wikipedia.org/wiki/Ordinary_least_squares#Matrix/vector_formulation)
- [2 Estimation](https://en.wikipedia.org/wiki/Ordinary_least_squares#Estimation)
- [3 Prediction](https://en.wikipedia.org/wiki/Ordinary_least_squares#Prediction)
- [4 Sample statistics](https://en.wikipedia.org/wiki/Ordinary_least_squares#Sample_statistics)
Toggle Sample statistics subsection
- [4\.1 Simple linear regression model](https://en.wikipedia.org/wiki/Ordinary_least_squares#Simple_linear_regression_model)
- [5 Alternative derivations](https://en.wikipedia.org/wiki/Ordinary_least_squares#Alternative_derivations)
Toggle Alternative derivations subsection
- [5\.1 Projection](https://en.wikipedia.org/wiki/Ordinary_least_squares#Projection)
- [5\.2 Maximum likelihood](https://en.wikipedia.org/wiki/Ordinary_least_squares#Maximum_likelihood)
- [5\.3 Generalized method of moments](https://en.wikipedia.org/wiki/Ordinary_least_squares#Generalized_method_of_moments)
- [6 Assumptions](https://en.wikipedia.org/wiki/Ordinary_least_squares#Assumptions)
- [7 Properties](https://en.wikipedia.org/wiki/Ordinary_least_squares#Properties)
Toggle Properties subsection
- [7\.1 Finite sample properties](https://en.wikipedia.org/wiki/Ordinary_least_squares#Finite_sample_properties)
- [7\.1.1 Assuming normality](https://en.wikipedia.org/wiki/Ordinary_least_squares#Assuming_normality)
- [7\.1.2 Influential observations](https://en.wikipedia.org/wiki/Ordinary_least_squares#Influential_observations)
- [7\.1.3 Partitioned regression](https://en.wikipedia.org/wiki/Ordinary_least_squares#Partitioned_regression)
- [7\.2 Large sample properties](https://en.wikipedia.org/wiki/Ordinary_least_squares#Large_sample_properties)
- [7\.2.1 Inference](https://en.wikipedia.org/wiki/Ordinary_least_squares#Inference)
- [7\.2.2 Hypothesis testing](https://en.wikipedia.org/wiki/Ordinary_least_squares#Hypothesis_testing)
- [7\.3 Violations of assumptions](https://en.wikipedia.org/wiki/Ordinary_least_squares#Violations_of_assumptions)
- [7\.3.1 Time series model](https://en.wikipedia.org/wiki/Ordinary_least_squares#Time_series_model)
- [7\.3.2 Constrained estimation](https://en.wikipedia.org/wiki/Ordinary_least_squares#Constrained_estimation)
- [8 Example with real data](https://en.wikipedia.org/wiki/Ordinary_least_squares#Example_with_real_data)
Toggle Example with real data subsection
- [8\.1 Sensitivity to rounding](https://en.wikipedia.org/wiki/Ordinary_least_squares#Sensitivity_to_rounding)
- [9 Another example with less real data](https://en.wikipedia.org/wiki/Ordinary_least_squares#Another_example_with_less_real_data)
Toggle Another example with less real data subsection
- [9\.1 Problem statement](https://en.wikipedia.org/wiki/Ordinary_least_squares#Problem_statement)
- [9\.2 Solution](https://en.wikipedia.org/wiki/Ordinary_least_squares#Solution)
- [10 See also](https://en.wikipedia.org/wiki/Ordinary_least_squares#See_also)
- [11 References](https://en.wikipedia.org/wiki/Ordinary_least_squares#References)
- [12 Further reading](https://en.wikipedia.org/wiki/Ordinary_least_squares#Further_reading)
Toggle the table of contents
# Ordinary least squares
12 languages
- [Ų§ŁŲ¹Ų±ŲØŁŲ©](https://ar.wikipedia.org/wiki/%D9%85%D8%B1%D8%A8%D8%B9%D8%A7%D8%AA_%D8%B5%D8%BA%D8%B1%D9%89_%D8%B9%D8%A7%D8%AF%D9%8A%D8%A9 "Ł
Ų±ŲØŲ¹Ų§ŲŖ ŲµŲŗŲ±Ł Ų¹Ų§ŲÆŁŲ© ā Arabic")
- [Asturianu](https://ast.wikipedia.org/wiki/M%C3%ADnimos_cuadraos_ordinarios "MĆnimos cuadraos ordinarios ā Asturian")
- [CatalĆ ](https://ca.wikipedia.org/wiki/M%C3%A8tode_dels_m%C3%ADnims_quadrats_ordinaris "MĆØtode dels mĆnims quadrats ordinaris ā Catalan")
- [ĪĪ»Ī»Ī·Ī½Ī¹ĪŗĪ¬](https://el.wikipedia.org/wiki/%CE%9C%CE%AD%CE%B8%CE%BF%CE%B4%CE%BF%CF%82_%CE%B5%CE%BB%CE%B1%CF%87%CE%AF%CF%83%CF%84%CF%89%CE%BD_%CF%84%CE%B5%CF%84%CF%81%CE%B1%CE%B3%CF%8E%CE%BD%CF%89%CE%BD "ĪĪĪøĪæĪ“ĪæĻ ĪµĪ»Ī±ĻĪÆĻĻĻĪ½ ĻĪµĻĻĪ±Ī³ĻĪ½ĻĪ½ ā Greek")
- [EspaƱol](https://es.wikipedia.org/wiki/M%C3%ADnimos_cuadrados_ordinarios "MĆnimos cuadrados ordinarios ā Spanish")
- [ŁŲ§Ų±Ų³Ū](https://fa.wikipedia.org/wiki/%DA%A9%D9%85%D8%AA%D8%B1%DB%8C%D9%86_%D9%85%D8%B1%D8%A8%D8%B9%D8%A7%D8%AA_%D9%85%D8%B9%D9%85%D9%88%D9%84%DB%8C "Ś©Ł
ŲŖŲ±ŪŁ Ł
Ų±ŲØŲ¹Ų§ŲŖ Ł
Ų¹Ł
ŁŁŪ ā Persian")
- [FranƧais](https://fr.wikipedia.org/wiki/M%C3%A9thode_des_moindres_carr%C3%A9s_ordinaire "MĆ©thode des moindres carrĆ©s ordinaire ā French")
- [ķźµģ“](https://ko.wikipedia.org/wiki/%EC%A0%95%EA%B7%9C%EB%B0%A9%EC%A0%95%EC%8B%9D "ģ ź·ė°©ģ ģ ā Korean")
- [Simple English](https://simple.wikipedia.org/wiki/Ordinary_least_squares "Ordinary least squares ā Simple English")
- [SlovenÅ”Äina](https://sl.wikipedia.org/wiki/Navadni_najmanj%C5%A1i_kvadrati "Navadni najmanjÅ”i kvadrati ā Slovenian")
- [ē²µčŖ](https://zh-yue.wikipedia.org/wiki/%E6%99%AE%E9%80%9A%E6%9C%80%E5%B0%8F%E4%BA%8C%E4%B9%98%E6%B3%95 "ę®éęå°äŗä¹ę³ ā Cantonese")
- [äøę](https://zh.wikipedia.org/wiki/%E6%99%AE%E9%80%9A%E6%9C%80%E5%B0%8F%E4%BA%8C%E4%B9%98%E6%B3%95 "ę®éęå°äŗä¹ę³ ā Chinese")
[Edit links](https://www.wikidata.org/wiki/Special:EntityPage/Q2912993#sitelinks-wikipedia "Edit interlanguage links")
- [Article](https://en.wikipedia.org/wiki/Ordinary_least_squares "View the content page [c]")
- [Talk](https://en.wikipedia.org/wiki/Talk:Ordinary_least_squares "Discuss improvements to the content page [t]")
English
- [Read](https://en.wikipedia.org/wiki/Ordinary_least_squares)
- [Edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit "Edit this page [e]")
- [View history](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=history "Past revisions of this page [h]")
Tools
Tools
move to sidebar
hide
Actions
- [Read](https://en.wikipedia.org/wiki/Ordinary_least_squares)
- [Edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit "Edit this page [e]")
- [View history](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=history)
General
- [What links here](https://en.wikipedia.org/wiki/Special:WhatLinksHere/Ordinary_least_squares "List of all English Wikipedia pages containing links to this page [j]")
- [Related changes](https://en.wikipedia.org/wiki/Special:RecentChangesLinked/Ordinary_least_squares "Recent changes in pages linked from this page [k]")
- [Upload file](https://en.wikipedia.org/wiki/Wikipedia:File_Upload_Wizard "Upload files [u]")
- [Permanent link](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&oldid=1345164414 "Permanent link to this revision of this page")
- [Page information](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=info "More information about this page")
- [Cite this page](https://en.wikipedia.org/w/index.php?title=Special:CiteThisPage&page=Ordinary_least_squares&id=1345164414&wpFormIdentifier=titleform "Information on how to cite this page")
- [Get shortened URL](https://en.wikipedia.org/w/index.php?title=Special:UrlShortener&url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FOrdinary_least_squares)
Print/export
- [Download as PDF](https://en.wikipedia.org/w/index.php?title=Special:DownloadAsPdf&page=Ordinary_least_squares&action=show-download-screen "Download this page as a PDF file")
- [Printable version](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&printable=yes "Printable version of this page [p]")
In other projects
- [Wikidata item](https://www.wikidata.org/wiki/Special:EntityPage/Q2912993 "Structured data on this page hosted by Wikidata [g]")
Appearance
move to sidebar
hide
From Wikipedia, the free encyclopedia
Method for estimating the unknown parameters in a linear regression model
[](https://en.wikipedia.org/wiki/File:Okuns_law_quarterly_differences.svg)
[Okun's law](https://en.wikipedia.org/wiki/Okun%27s_law "Okun's law") in [macroeconomics](https://en.wikipedia.org/wiki/Macroeconomics "Macroeconomics") states that in an economy the [GDP](https://en.wikipedia.org/wiki/GDP "GDP") growth should depend linearly on the changes in the unemployment rate. Here the ordinary least squares method is used to construct the regression line describing this law.
| |
|---|
| Part of a series on |
| [Regression analysis](https://en.wikipedia.org/wiki/Regression_analysis "Regression analysis") |
| Models |
| [Linear regression](https://en.wikipedia.org/wiki/Linear_regression "Linear regression") [Simple regression](https://en.wikipedia.org/wiki/Simple_linear_regression "Simple linear regression") [Polynomial regression](https://en.wikipedia.org/wiki/Polynomial_regression "Polynomial regression") [General linear model](https://en.wikipedia.org/wiki/General_linear_model "General linear model") |
| [Generalized linear model](https://en.wikipedia.org/wiki/Generalized_linear_model "Generalized linear model") [Vector generalized linear model](https://en.wikipedia.org/wiki/Vector_generalized_linear_model "Vector generalized linear model") [Discrete choice](https://en.wikipedia.org/wiki/Discrete_choice "Discrete choice") [Binomial regression](https://en.wikipedia.org/wiki/Binomial_regression "Binomial regression") [Binary regression](https://en.wikipedia.org/wiki/Binary_regression "Binary regression") [Logistic regression](https://en.wikipedia.org/wiki/Logistic_regression "Logistic regression") [Multinomial logistic regression](https://en.wikipedia.org/wiki/Multinomial_logistic_regression "Multinomial logistic regression") [Mixed logit](https://en.wikipedia.org/wiki/Mixed_logit "Mixed logit") [Probit](https://en.wikipedia.org/wiki/Probit_model "Probit model") [Multinomial probit](https://en.wikipedia.org/wiki/Multinomial_probit "Multinomial probit") [Ordered logit](https://en.wikipedia.org/wiki/Ordered_logit "Ordered logit") [Ordered probit](https://en.wikipedia.org/wiki/Ordered_probit "Ordered probit") [Poisson](https://en.wikipedia.org/wiki/Poisson_regression "Poisson regression") |
| [Multilevel model](https://en.wikipedia.org/wiki/Multilevel_model "Multilevel model") [Fixed effects](https://en.wikipedia.org/wiki/Fixed_effects_model "Fixed effects model") [Random effects](https://en.wikipedia.org/wiki/Random_effects_model "Random effects model") [Linear mixed-effects model](https://en.wikipedia.org/wiki/Mixed_model "Mixed model") [Nonlinear mixed-effects model](https://en.wikipedia.org/wiki/Nonlinear_mixed-effects_model "Nonlinear mixed-effects model") |
| [Nonlinear regression](https://en.wikipedia.org/wiki/Nonlinear_regression "Nonlinear regression") [Nonparametric](https://en.wikipedia.org/wiki/Nonparametric_regression "Nonparametric regression") [Semiparametric](https://en.wikipedia.org/wiki/Semiparametric_regression "Semiparametric regression") [Robust](https://en.wikipedia.org/wiki/Robust_regression "Robust regression") [Quantile](https://en.wikipedia.org/wiki/Quantile_regression "Quantile regression") [Isotonic](https://en.wikipedia.org/wiki/Isotonic_regression "Isotonic regression") [Principal components](https://en.wikipedia.org/wiki/Principal_component_regression "Principal component regression") [Least angle](https://en.wikipedia.org/wiki/Least-angle_regression "Least-angle regression") [Local](https://en.wikipedia.org/wiki/Local_regression "Local regression") [Segmented](https://en.wikipedia.org/wiki/Segmented_regression "Segmented regression") |
| [Errors-in-variables](https://en.wikipedia.org/wiki/Errors-in-variables_models "Errors-in-variables models") |
| Estimation |
| [Least squares](https://en.wikipedia.org/wiki/Least_squares "Least squares") [Linear](https://en.wikipedia.org/wiki/Linear_least_squares "Linear least squares") [Non-linear](https://en.wikipedia.org/wiki/Non-linear_least_squares "Non-linear least squares") |
| [Ordinary]() [Weighted](https://en.wikipedia.org/wiki/Weighted_least_squares "Weighted least squares") [Generalized](https://en.wikipedia.org/wiki/Generalized_least_squares "Generalized least squares") [Generalized estimating equation](https://en.wikipedia.org/wiki/Generalized_estimating_equation "Generalized estimating equation") |
| [Partial](https://en.wikipedia.org/wiki/Partial_least_squares_regression "Partial least squares regression") [Total](https://en.wikipedia.org/wiki/Total_least_squares "Total least squares") [Non-negative](https://en.wikipedia.org/wiki/Non-negative_least_squares "Non-negative least squares") [Ridge regression](https://en.wikipedia.org/wiki/Tikhonov_regularization "Tikhonov regularization") [Regularized](https://en.wikipedia.org/wiki/Regularized_least_squares "Regularized least squares") |
| [Least absolute deviations](https://en.wikipedia.org/wiki/Least_absolute_deviations "Least absolute deviations") [Iteratively reweighted](https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares "Iteratively reweighted least squares") [Bayesian](https://en.wikipedia.org/wiki/Bayesian_linear_regression "Bayesian linear regression") [Bayesian multivariate](https://en.wikipedia.org/wiki/Bayesian_multivariate_linear_regression "Bayesian multivariate linear regression") [Least-squares spectral analysis](https://en.wikipedia.org/wiki/Least-squares_spectral_analysis "Least-squares spectral analysis") |
| Background |
| [Regression validation](https://en.wikipedia.org/wiki/Regression_validation "Regression validation") [Mean and predicted response](https://en.wikipedia.org/wiki/Mean_and_predicted_response "Mean and predicted response") [Errors and residuals](https://en.wikipedia.org/wiki/Errors_and_residuals "Errors and residuals") [Goodness of fit](https://en.wikipedia.org/wiki/Goodness_of_fit "Goodness of fit") [Studentized residual](https://en.wikipedia.org/wiki/Studentized_residual "Studentized residual") [GaussāMarkov theorem](https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem "GaussāMarkov theorem") |
| [](https://en.wikipedia.org/wiki/File:Nuvola_apps_edu_mathematics_blue-p.svg) [Mathematics portal](https://en.wikipedia.org/wiki/Portal:Mathematics "Portal:Mathematics") |
| [v](https://en.wikipedia.org/wiki/Template:Regression_bar "Template:Regression bar") [t](https://en.wikipedia.org/wiki/Template_talk:Regression_bar "Template talk:Regression bar") [e](https://en.wikipedia.org/wiki/Special:EditPage/Template:Regression_bar "Special:EditPage/Template:Regression bar") |
In [statistics](https://en.wikipedia.org/wiki/Statistics "Statistics"), **ordinary least squares** (**OLS**) is a type of [linear least squares](https://en.wikipedia.org/wiki/Linear_least_squares "Linear least squares") method for choosing the unknown [parameters](https://en.wikipedia.org/wiki/Statistical_parameter "Statistical parameter") in a [linear regression](https://en.wikipedia.org/wiki/Linear_regression "Linear regression") model (with fixed level-one\[*[clarification needed](https://en.wikipedia.org/wiki/Wikipedia:Please_clarify "Wikipedia:Please clarify")*\] effects of a [linear function](https://en.wikipedia.org/wiki/Linear_function "Linear function") of a set of [explanatory variables](https://en.wikipedia.org/wiki/Explanatory_variable "Explanatory variable")) by the principle of [least squares](https://en.wikipedia.org/wiki/Least_squares "Least squares"): minimizing the sum of the squares of the differences between the observed [dependent variable](https://en.wikipedia.org/wiki/Dependent_variable "Dependent variable") (values of the variable being observed) in the input [dataset](https://en.wikipedia.org/wiki/Dataset "Dataset") and the output of the (linear) function of the [independent variable](https://en.wikipedia.org/wiki/Independent_variable "Independent variable"). Some sources consider OLS to be linear regression.[\[1\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-1)
Geometrically, this is seen as the sum of the squared distances, parallel to the axis of the dependent variable, between each data point in the set and the corresponding point on the regression surfaceāthe smaller the differences, the better the model fits the data. The resulting [estimator](https://en.wikipedia.org/wiki/Statistical_estimation "Statistical estimation") can be expressed by a simple formula, especially in the case of a [simple linear regression](https://en.wikipedia.org/wiki/Simple_linear_regression "Simple linear regression"), in which there is a single [regressor](https://en.wikipedia.org/wiki/Regressor "Regressor") on the right side of the regression equation.
The OLS estimator is [consistent](https://en.wikipedia.org/wiki/Consistent_estimator "Consistent estimator") for the level-one fixed effects when the regressors are [exogenous](https://en.wikipedia.org/wiki/Exogenous "Exogenous") and forms perfect [colinearity](https://en.wikipedia.org/wiki/Collinearity "Collinearity") (rank condition), consistent for the variance estimate of the residuals when regressors have finite fourth moments[\[2\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-2) andāby the [GaussāMarkov theorem](https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem "GaussāMarkov theorem")ā[optimal in the class of linear unbiased estimators](https://en.wikipedia.org/wiki/Best_linear_unbiased_estimator "Best linear unbiased estimator") when the [errors](https://en.wikipedia.org/wiki/Statistical_error "Statistical error") are [homoscedastic](https://en.wikipedia.org/wiki/Homoscedastic "Homoscedastic") and [serially uncorrelated](https://en.wikipedia.org/wiki/Autocorrelation "Autocorrelation"). Under these conditions, the method of OLS provides [minimum-variance mean-unbiased](https://en.wikipedia.org/wiki/UMVU "UMVU") estimation when the errors have finite [variances](https://en.wikipedia.org/wiki/Variance "Variance"). Under the additional assumption that the errors are [normally distributed](https://en.wikipedia.org/wiki/Normal_distribution "Normal distribution") with zero mean, OLS is the [maximum likelihood estimator](https://en.wikipedia.org/wiki/Maximum_likelihood_estimator "Maximum likelihood estimator") that outperforms any non-linear unbiased estimator.
## Linear model
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=1 "Edit section: Linear model")\]
Main article: [Linear regression model](https://en.wikipedia.org/wiki/Linear_regression_model "Linear regression model")
Suppose the data consists of n {\\displaystyle n}  [observations](https://en.wikipedia.org/wiki/Statistical_unit "Statistical unit") { x i , y i } i \= 1 n {\\displaystyle \\left\\{\\mathbf {x} \_{i},y\_{i}\\right\\}\_{i=1}^{n}} . Each observation i {\\displaystyle i}  includes a scalar response y i {\\displaystyle y\_{i}}  and a column vector x i {\\displaystyle \\mathbf {x} \_{i}}  of p {\\displaystyle p}  parameters (regressors), i.e., x i \= \[ x i 1 , x i 2 , ā¦ , x i p \] T {\\displaystyle \\mathbf {x} \_{i}=\\left\[x\_{i1},x\_{i2},\\dots ,x\_{ip}\\right\]^{\\operatorname {T} }} ![{\\displaystyle \\mathbf {x} \_{i}=\\left\[x\_{i1},x\_{i2},\\dots ,x\_{ip}\\right\]^{\\operatorname {T} }}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3278872b5bdb53e6af3474d92e9926c0238e8935). In a [linear regression model](https://en.wikipedia.org/wiki/Linear_regression_model "Linear regression model"), the response variable, y i {\\displaystyle y\_{i}} , is a linear function of the regressors:
y
i
\=
Ī²
1
x
i
1
\+
Ī²
2
x
i
2
\+
āÆ
\+
Ī²
p
x
i
p
\+
Īµ
i
,
{\\displaystyle y\_{i}=\\beta \_{1}\\ x\_{i1}+\\beta \_{2}\\ x\_{i2}+\\cdots +\\beta \_{p}\\ x\_{ip}+\\varepsilon \_{i},}
or in [vector](https://en.wikipedia.org/wiki/Row_and_column_vectors "Row and column vectors") form,
y
i
\=
x
i
T
Ī²
\+
Īµ
i
,
{\\displaystyle y\_{i}=\\mathbf {x} \_{i}^{\\operatorname {T} }{\\boldsymbol {\\beta }}+\\varepsilon \_{i},\\,}
where x i {\\displaystyle \\mathbf {x} \_{i}} , as introduced previously, is a column vector of the i {\\displaystyle i} \-th observation of all the explanatory variables; Ī² {\\displaystyle {\\boldsymbol {\\beta }}}  is a p Ć 1 {\\displaystyle p\\times 1}  vector of unknown parameters; and the scalar Īµ i {\\displaystyle \\varepsilon \_{i}}  represents unobserved random variables ([errors](https://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics "Errors and residuals in statistics")) of the i {\\displaystyle i} \-th observation. Īµ i {\\displaystyle \\varepsilon \_{i}}  accounts for the influences upon the responses y i {\\displaystyle y\_{i}}  from sources other than the explanatory variables x i {\\displaystyle \\mathbf {x} \_{i}} . This model can also be written in matrix notation as
y
\=
X
Ī²
\+
Īµ
,
{\\displaystyle \\mathbf {y} =\\mathbf {X} {\\boldsymbol {\\beta }}+{\\boldsymbol {\\varepsilon }},\\,}
where y {\\displaystyle \\mathbf {y} }  and Īµ {\\displaystyle {\\boldsymbol {\\varepsilon }}}  are n Ć 1 {\\displaystyle n\\times 1}  vectors of the response variables and the errors of the n {\\displaystyle n}  observations, and X {\\displaystyle \\mathbf {X} }  is an n Ć p {\\displaystyle n\\times p}  matrix of regressors, also sometimes called the [design matrix](https://en.wikipedia.org/wiki/Design_matrix "Design matrix"), whose row i {\\displaystyle i}  is x i T {\\displaystyle \\mathbf {x} \_{i}^{\\operatorname {T} }}  and contains the i {\\displaystyle i} \-th observations on all the explanatory variables.
Typically, a constant term is included in the set of regressors X {\\displaystyle \\mathbf {X} } , say, by taking x i 1 \= 1 {\\displaystyle x\_{i1}=1}  for all i \= 1 , ā¦ , n {\\displaystyle i=1,\\dots ,n} . The coefficient Ī² 1 {\\displaystyle \\beta \_{1}}  corresponding to this regressor is called the *intercept*. Without the intercept, the fitted line is forced to cross the origin when x i \= 0 ā {\\displaystyle x\_{i}={\\vec {0}}} .
Regressors do not have to be independent for estimation to be consistent e.g. they may be non-linearly dependent. Short of perfect multicollinearity, parameter estimates may still be consistent; however, as multicollinearity rises the standard error around such estimates increases and reduces the precision of such estimates. When there is perfect multicollinearity, it is no longer possible to obtain unique estimates for the coefficients to the related regressors; estimation for these parameters cannot converge (thus, it cannot be consistent).
As a concrete example where regressors are non-linearly dependent yet estimation may still be consistent, we might suspect the response depends linearly both on a value and its square; in which case we would include one regressor whose value is just the square of another regressor. In that case, the model would be *quadratic* in the second regressor, but none-the-less is still considered a *linear* model because the model *is* still linear in the parameters (Ī² {\\displaystyle {\\boldsymbol {\\beta }}} ).
### Matrix/vector formulation
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=2 "Edit section: Matrix/vector formulation")\]
Consider an [overdetermined system](https://en.wikipedia.org/wiki/Overdetermined_system "Overdetermined system")
ā
j
\=
1
p
x
i
j
Ī²
j
\=
y
i
,
(
i
\=
1
,
2
,
ā¦
,
n
)
,
{\\displaystyle \\sum \_{j=1}^{p}x\_{ij}\\beta \_{j}=y\_{i},\\ (i=1,2,\\dots ,n),}
of n {\\displaystyle n}  [linear equations](https://en.wikipedia.org/wiki/Linear_equation "Linear equation") in p {\\displaystyle p}  unknown [coefficients](https://en.wikipedia.org/wiki/Coefficients "Coefficients"), Ī² 1 , Ī² 2 , ā¦ , Ī² p {\\displaystyle \\beta \_{1},\\beta \_{2},\\dots ,\\beta \_{p}} , with n \> p {\\displaystyle n\>p} . This can be written in [matrix](https://en.wikipedia.org/wiki/Matrix_\(mathematics\) "Matrix (mathematics)") form as
X
Ī²
\=
y
,
{\\displaystyle \\mathbf {X} {\\boldsymbol {\\beta }}=\\mathbf {y} ,}
where
X
\=
\[
X
11
X
12
āÆ
X
1
p
X
21
X
22
āÆ
X
2
p
ā®
ā®
ā±
ā®
X
n
1
X
n
2
āÆ
X
n
p
\]
,
Ī²
\=
\[
Ī²
1
Ī²
2
ā®
Ī²
p
\]
,
y
\=
\[
y
1
y
2
ā®
y
n
\]
.
{\\displaystyle \\mathbf {X} ={\\begin{bmatrix}X\_{11}\&X\_{12}&\\cdots \&X\_{1p}\\\\X\_{21}\&X\_{22}&\\cdots \&X\_{2p}\\\\\\vdots &\\vdots &\\ddots &\\vdots \\\\X\_{n1}\&X\_{n2}&\\cdots \&X\_{np}\\end{bmatrix}},\\qquad {\\boldsymbol {\\beta }}={\\begin{bmatrix}\\beta \_{1}\\\\\\beta \_{2}\\\\\\vdots \\\\\\beta \_{p}\\end{bmatrix}},\\qquad \\mathbf {y} ={\\begin{bmatrix}y\_{1}\\\\y\_{2}\\\\\\vdots \\\\y\_{n}\\end{bmatrix}}.}
(Note: for a linear model as above, not all elements in X {\\displaystyle \\mathbf {X} }  contains information on the data points. The first column is populated with ones, X i 1 \= 1 {\\displaystyle X\_{i1}=1} . Only the other columns contain actual data. So here p {\\displaystyle p}  is equal to the number of regressors plus one).
Such a system usually has no exact solution, so the goal is instead to find the coefficients Ī² {\\displaystyle {\\boldsymbol {\\beta }}}  which fit the equations "best", in the sense of solving the [quadratic](https://en.wikipedia.org/wiki/Quadratic_form_\(statistics\) "Quadratic form (statistics)") [minimization](https://en.wikipedia.org/wiki/Mathematical_optimization "Mathematical optimization") problem
Ī²
^
\=
a
r
g
m
i
n
Ī²
S
(
Ī²
)
,
{\\displaystyle {\\hat {\\boldsymbol {\\beta }}}={\\underset {\\boldsymbol {\\beta }}{\\operatorname {arg\\,min} }}\\,S({\\boldsymbol {\\beta }}),}
where the objective function S {\\displaystyle S}  is given by
S
(
Ī²
)
\=
ā
i
\=
1
n
\|
y
i
ā
ā
j
\=
1
p
X
i
j
Ī²
j
\|
2
\=
ā
y
ā
X
Ī²
ā
2
.
{\\displaystyle S({\\boldsymbol {\\beta }})=\\sum \_{i=1}^{n}\\left\|y\_{i}-\\sum \_{j=1}^{p}X\_{ij}\\beta \_{j}\\right\|^{2}=\\left\\\|\\mathbf {y} -\\mathbf {X} {\\boldsymbol {\\beta }}\\right\\\|^{2}.}
A justification for choosing this criterion is given in [Properties](https://en.wikipedia.org/wiki/Ordinary_least_squares#Properties) below. This minimization problem has a unique solution, provided that the p {\\displaystyle p}  columns of the matrix X {\\displaystyle \\mathbf {X} }  are [linearly independent](https://en.wikipedia.org/wiki/Linearly_independent "Linearly independent"), given by solving the so-called *normal equations*:
(
X
T
X
)
Ī²
^
\=
X
T
y
.
{\\displaystyle \\left(\\mathbf {X} ^{\\operatorname {T} }\\mathbf {X} \\right){\\hat {\\boldsymbol {\\beta }}}=\\mathbf {X} ^{\\operatorname {T} }\\mathbf {y} \\ .}
The matrix X T X {\\displaystyle \\mathbf {X} ^{\\operatorname {T} }\\mathbf {X} }  is known as the *normal matrix* or [Gram matrix](https://en.wikipedia.org/wiki/Gram_matrix "Gram matrix") and the matrix X T y {\\displaystyle \\mathbf {X} ^{\\operatorname {T} }\\mathbf {y} }  is known as the [moment matrix](https://en.wikipedia.org/wiki/Moment_matrix "Moment matrix") of regressand by regressors.[\[3\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-3) Finally, Ī² ^ {\\displaystyle {\\hat {\\boldsymbol {\\beta }}}}  is the coefficient vector of the least-squares [hyperplane](https://en.wikipedia.org/wiki/Hyperplane "Hyperplane"), expressed as
Ī²
^
\=
(
X
ā¤
X
)
ā
1
X
ā¤
y
.
{\\displaystyle {\\hat {\\boldsymbol {\\beta }}}=\\left(\\mathbf {X} ^{\\top }\\mathbf {X} \\right)^{-1}\\mathbf {X} ^{\\top }\\mathbf {y} .}
or
Ī²
^
\=
Ī²
\+
(
X
ā¤
X
)
ā
1
X
ā¤
Īµ
.
{\\displaystyle {\\hat {\\boldsymbol {\\beta }}}={\\boldsymbol {\\beta }}+\\left(\\mathbf {X} ^{\\top }\\mathbf {X} \\right)^{-1}\\mathbf {X} ^{\\top }{\\boldsymbol {\\varepsilon }}.}
## Estimation
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=3 "Edit section: Estimation")\]
Suppose *b* is a "candidate" value for the parameter vector *Ī²*. The quantity *yi* ā *xi*T*b*, called the *[residual](https://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics "Errors and residuals in statistics")* for the *i*\-th observation, measures the vertical distance between the data point (*xi*, *yi*) and the hyperplane *y* = *x*T*b*, and thus assesses the degree of fit between the actual data and the model. The *[sum of squared residuals](https://en.wikipedia.org/wiki/Sum_of_squared_residuals "Sum of squared residuals")* (*SSR*) (also called the *error sum of squares* (*ESS*) or *residual sum of squares* (*RSS*))[\[4\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-4) is a measure of the overall model fit:
S
(
b
)
\=
ā
i
\=
1
n
(
y
i
ā
x
i
T
b
)
2
\=
(
y
ā
X
b
)
T
(
y
ā
X
b
)
,
{\\displaystyle S(b)=\\sum \_{i=1}^{n}(y\_{i}-x\_{i}^{\\operatorname {T} }b)^{2}=(y-Xb)^{\\operatorname {T} }(y-Xb),}
where *T* denotes the matrix [transpose](https://en.wikipedia.org/wiki/Transpose "Transpose"), and the rows of *X*, denoting the values of all the independent variables associated with a particular value of the dependent variable, are *Xi = xi*T. The value of *b* which minimizes this sum is called the **OLS estimator for *Ī²***. The function *S*(*b*) is quadratic in *b* with positive-definite [Hessian](https://en.wikipedia.org/wiki/Hessian_matrix "Hessian matrix"), and therefore this function possesses a unique global minimum at b \= Ī² ^ {\\displaystyle b={\\hat {\\beta }}} , which can be given by the explicit formula[\[5\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-5)[\[proof\]](https://en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares#Least_squares_estimator_for_.CE.B2 "Proofs involving ordinary least squares")
Ī²
^
\=
argmin
b
ā
R
p
ā”
S
(
b
)
\=
(
X
T
X
)
ā
1
X
T
y
.
{\\displaystyle {\\hat {\\beta }}=\\operatorname {argmin} \_{b\\in \\mathbb {R} ^{p}}S(b)=(X^{\\operatorname {T} }X)^{-1}X^{\\operatorname {T} }y\\ .}
The product *N* = *X*T *X* is a [Gram matrix](https://en.wikipedia.org/wiki/Gram_matrix "Gram matrix"), and its inverse, *Q* = *N*ā1, is the *cofactor matrix* of *Ī²*,[\[6\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-6)[\[7\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-7)[\[8\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-8) closely related to its [covariance matrix](https://en.wikipedia.org/wiki/Ordinary_least_squares#Covariance_matrix), *C**Ī²*. The matrix (*X*T *X*)ā1 *X*T = *Q* *X*T is called the [MooreāPenrose pseudoinverse](https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_pseudoinverse "MooreāPenrose pseudoinverse") matrix of *X*. This formulation highlights the point that estimation can be carried out if, and only if, there is no perfect [multicollinearity](https://en.wikipedia.org/wiki/Multicollinearity "Multicollinearity") between the explanatory variables (which would cause the Gram matrix to have no inverse).
## Prediction
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=4 "Edit section: Prediction")\]
After we have estimated *Ī²*, the *[fitted values](https://en.wikipedia.org/wiki/Fitted_value "Fitted value")* (or *predicted values*) from the regression will be
y
^
\=
X
Ī²
^
\=
P
y
,
{\\displaystyle {\\hat {y}}=X{\\hat {\\beta }}=Py,}
where *P* = *X*(*X*T*X*)ā1*X*T is the *[projection matrix](https://en.wikipedia.org/wiki/Projection_matrix "Projection matrix")* onto the space *V* spanned by the columns of *X*. This matrix *P* is also sometimes called the *[hat matrix](https://en.wikipedia.org/wiki/Hat_matrix "Hat matrix")* because it "puts a hat" onto the variable *y*. Another matrix, closely related to *P* is the *annihilator* matrix *M* = *In* ā *P*; this is a projection matrix onto the space orthogonal to *V*. Both matrices *P* and *M* are [symmetric](https://en.wikipedia.org/wiki/Symmetric_matrix "Symmetric matrix") and [idempotent](https://en.wikipedia.org/wiki/Idempotent_matrix "Idempotent matrix") (meaning that *P*2 = *P* and *M*2 = *M*), and relate to the data matrix *X* via identities *PX* = *X* and *MX* = 0.[\[9\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hayashi_2000_loc=page_19-9) Matrix *M* creates the *residuals* from the regression:
Īµ
^
\=
y
ā
y
^
\=
y
ā
X
Ī²
^
\=
M
y
\=
M
(
X
Ī²
\+
Īµ
)
\=
(
M
X
)
Ī²
\+
M
Īµ
\=
M
Īµ
.
{\\displaystyle {\\hat {\\varepsilon }}=y-{\\hat {y}}=y-X{\\hat {\\beta }}=My=M(X\\beta +\\varepsilon )=(MX)\\beta +M\\varepsilon =M\\varepsilon .}
The variances of the predicted values s y ^ i 2 {\\displaystyle s\_{{\\hat {y}}\_{i}}^{2}}  are found in the main diagonal of the [variance-covariance matrix](https://en.wikipedia.org/wiki/Variance-covariance_matrix "Variance-covariance matrix") of predicted values:
C
y
^
\=
s
2
P
,
{\\displaystyle C\_{\\hat {y}}=s^{2}P,}
where *P* is the projection matrix and *s*2 is the sample variance.[\[10\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-q011-10) The full matrix is very large; its diagonal elements can be calculated individually as:
s
y
^
i
2
\=
s
2
X
i
(
X
T
X
)
ā
1
X
i
T
,
{\\displaystyle s\_{{\\hat {y}}\_{i}}^{2}=s^{2}X\_{i}(X^{T}X)^{-1}X\_{i}^{T},}
where *X*i is the *i*\-th row of matrix *X*.
## Sample statistics
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=5 "Edit section: Sample statistics")\]
Using these residuals we can estimate the sample variance *s*2 using the *[reduced chi-squared](https://en.wikipedia.org/wiki/Reduced_chi-squared "Reduced chi-squared")* statistic:
s
2
\=
Īµ
^
T
Īµ
^
n
ā
p
\=
(
M
y
)
T
M
y
n
ā
p
\=
y
T
M
T
M
y
n
ā
p
\=
y
T
M
y
n
ā
p
\=
S
(
Ī²
^
)
n
ā
p
,
Ļ
^
2
\=
n
ā
p
n
s
2
{\\displaystyle s^{2}={\\frac {{\\hat {\\varepsilon }}^{\\mathrm {T} }{\\hat {\\varepsilon }}}{n-p}}={\\frac {(My)^{\\mathrm {T} }My}{n-p}}={\\frac {y^{\\mathrm {T} }M^{\\mathrm {T} }My}{n-p}}={\\frac {y^{\\mathrm {T} }My}{n-p}}={\\frac {S({\\hat {\\beta }})}{n-p}},\\qquad {\\hat {\\sigma }}^{2}={\\frac {n-p}{n}}\\;s^{2}}
The denominator, *n*ā*p*, is the [statistical degrees of freedom](https://en.wikipedia.org/wiki/Degrees_of_freedom_\(statistics\) "Degrees of freedom (statistics)"). The first quantity, *s*2, is the OLS estimate for *Ļ*2, whereas the second, Ļ ^ 2 {\\displaystyle \\scriptstyle {\\hat {\\sigma }}^{2}} , is the MLE estimate for *Ļ*2. The two estimators are quite similar in large samples; the first estimator is always [unbiased](https://en.wikipedia.org/wiki/Estimator_bias "Estimator bias"), while the second estimator is biased but has a smaller [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error "Mean squared error"). In practice *s*2 is used more often, since it is more convenient for the hypothesis testing. The square root of *s*2 is called the *[regression standard error](https://en.wikipedia.org/wiki/Regression_standard_error "Regression standard error")*,[\[11\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-11) *standard error of the regression*,[\[12\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-12)[\[13\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-13) or *standard error of the equation*.[\[9\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hayashi_2000_loc=page_19-9)
It is common to assess the goodness-of-fit of the OLS regression by comparing how much the initial variation in the sample can be reduced by regressing onto *X*. The *[coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination "Coefficient of determination")* *R*2 is defined as a ratio of "explained" variance to the "total" variance of the dependent variable *y*, in the cases where the regression sum of squares equals the sum of squares of residuals:[\[14\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-14)
R
2
\=
ā
(
y
^
i
ā
y
ĀÆ
)
2
ā
(
y
i
ā
y
ĀÆ
)
2
\=
y
T
P
T
L
P
y
y
T
L
y
\=
1
ā
y
T
M
y
y
T
L
y
\=
1
ā
R
S
S
T
S
S
{\\displaystyle R^{2}={\\frac {\\sum ({\\hat {y}}\_{i}-{\\overline {y}})^{2}}{\\sum (y\_{i}-{\\overline {y}})^{2}}}={\\frac {y^{\\mathrm {T} }P^{\\mathrm {T} }LPy}{y^{\\mathrm {T} }Ly}}=1-{\\frac {y^{\\mathrm {T} }My}{y^{\\mathrm {T} }Ly}}=1-{\\frac {\\rm {RSS}}{\\rm {TSS}}}}
where TSS is the *[total sum of squares](https://en.wikipedia.org/wiki/Total_sum_of_squares "Total sum of squares")* for the dependent variable, L \= I n ā 1 n J n {\\textstyle L=I\_{n}-{\\frac {1}{n}}J\_{n}} , and J n {\\textstyle J\_{n}}  is an *n*Ć*n* matrix of ones. (L {\\displaystyle L}  is a [centering matrix](https://en.wikipedia.org/wiki/Centering_matrix "Centering matrix") which is equivalent to regression on a constant; it simply subtracts the mean from a variable.) In order for *R*2 to be meaningful, the matrix *X* of data on regressors must contain a column vector of ones to represent the constant whose coefficient is the regression intercept. In that case, *R*2 will always be a number between 0 and 1, with values close to 1 indicating a good degree of fit.
### Simple linear regression model
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=6 "Edit section: Simple linear regression model")\]
Main article: [Simple linear regression](https://en.wikipedia.org/wiki/Simple_linear_regression "Simple linear regression")
If the data matrix *X* contains only two variables, a constant and a scalar regressor *xi*, then this is called the "simple regression model". This case is often considered in the beginner statistics classes, as it provides much simpler formulas even suitable for manual calculation. The parameters are commonly denoted as (*Ī±*, *Ī²*):
y
i
\=
Ī±
\+
Ī²
x
i
\+
Īµ
i
.
{\\displaystyle y\_{i}=\\alpha +\\beta x\_{i}+\\varepsilon \_{i}.}
The least squares estimates in this case are given by simple formulas
Ī²
^
\=
ā
i
\=
1
n
(
x
i
ā
x
ĀÆ
)
(
y
i
ā
y
ĀÆ
)
ā
i
\=
1
n
(
x
i
ā
x
ĀÆ
)
2
Ī±
^
\=
y
ĀÆ
ā
Ī²
^
x
ĀÆ
,
{\\displaystyle {\\begin{aligned}{\\widehat {\\beta }}&={\\frac {\\sum \_{i=1}^{n}{(x\_{i}-{\\bar {x}})(y\_{i}-{\\bar {y}})}}{\\sum \_{i=1}^{n}{(x\_{i}-{\\bar {x}})^{2}}}}\\\\\[2pt\]{\\widehat {\\alpha }}&={\\bar {y}}-{\\widehat {\\beta }}\\,{\\bar {x}}\\ ,\\end{aligned}}}
![{\\displaystyle {\\begin{aligned}{\\widehat {\\beta }}&={\\frac {\\sum \_{i=1}^{n}{(x\_{i}-{\\bar {x}})(y\_{i}-{\\bar {y}})}}{\\sum \_{i=1}^{n}{(x\_{i}-{\\bar {x}})^{2}}}}\\\\\[2pt\]{\\widehat {\\alpha }}&={\\bar {y}}-{\\widehat {\\beta }}\\,{\\bar {x}}\\ ,\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/932c6407f7ceba533fef69961fe504fc3b565e1e)
## Alternative derivations
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=7 "Edit section: Alternative derivations")\]
In the previous section the least squares estimator Ī² ^ {\\displaystyle {\\hat {\\beta }}}  was obtained as a value that minimizes the sum of squared residuals of the model. However it is also possible to derive the same estimator from other approaches. In all cases the formula for OLS estimator remains the same: ^*Ī²* = (*X*T*X*)ā1*X*T*y*; the only difference is in how we interpret this result.
### Projection
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=8 "Edit section: Projection")\]
[](https://en.wikipedia.org/wiki/File:OLS_geometric_interpretation.svg)
OLS estimation can be viewed as a projection onto the linear space spanned by the regressors. (Here each of
X
1
{\\displaystyle X\_{1}}
and
X
2
{\\displaystyle X\_{2}}
refers to a column of the data matrix.)
[](https://en.wikipedia.org/wiki/File:Geometric_interpretation_of_least_squares_\(three_observations\).png)
Least squares as projection of y onto col(X) for three observations; Å· = XĪ² gives fitted values and y ā Å· is the residual.
| | |
|---|---|
|  | This section **may need to be cleaned up.** It has been [merged](https://en.wikipedia.org/wiki/H:M "H:M") from *[Linear least squares](https://en.wikipedia.org/wiki/Linear_least_squares "Linear least squares")*. |
For mathematicians, OLS is an approximate solution to an overdetermined system of linear equations *XĪ²* ā *y*, where *Ī²* is the unknown. Assuming the system cannot be solved exactly (the number of equations *n* is much larger than the number of unknowns *p*), we are looking for a solution that could provide the smallest discrepancy between the right- and left- hand sides. In other words, we are looking for the solution that satisfies
Ī²
^
\=
a
r
g
min
Ī²
ā
y
ā
X
Ī²
ā
2
,
{\\displaystyle {\\hat {\\beta }}={\\rm {arg}}\\min \_{\\beta }\\,\\lVert \\mathbf {y} -\\mathbf {X} {\\boldsymbol {\\beta }}\\rVert ^{2},}
where āĀ·ā is the standard [*L*2 norm](https://en.wikipedia.org/wiki/Norm_\(mathematics\)#Euclidean_norm "Norm (mathematics)") in the *n*\-dimensional [Euclidean space](https://en.wikipedia.org/wiki/Euclidean_space "Euclidean space") **R***n*. The predicted quantity *XĪ²* is just a certain linear combination of the vectors of regressors. Thus, the residual vector *y* ā *XĪ²* will have the smallest length when *y* is [projected orthogonally](https://en.wikipedia.org/wiki/Projection_\(linear_algebra\) "Projection (linear algebra)") onto the [linear subspace](https://en.wikipedia.org/wiki/Linear_subspace "Linear subspace") [spanned](https://en.wikipedia.org/wiki/Linear_span "Linear span") by the columns of *X*. The OLS estimator Ī² ^ {\\displaystyle {\\hat {\\beta }}}  in this case can be interpreted as the coefficients of [vector decomposition](https://en.wikipedia.org/wiki/Vector_decomposition "Vector decomposition") of ^*y* = *Py* along the basis of *X*.
In other words, the gradient equations at the minimum can be written as:
(
y
ā
X
Ī²
^
)
ā¤
X
\=
0\.
{\\displaystyle (\\mathbf {y} -\\mathbf {X} {\\hat {\\boldsymbol {\\beta }}})^{\\top }\\mathbf {X} =0.}
A geometrical interpretation of these equations is that the vector of residuals, y ā X Ī² ^ {\\displaystyle \\mathbf {y} -X{\\hat {\\boldsymbol {\\beta }}}}  is orthogonal to the [column space](https://en.wikipedia.org/wiki/Column_space "Column space") of *X*, since the [dot product](https://en.wikipedia.org/wiki/Dot_product "Dot product") ( y ā X Ī² ^ ) ā
X v {\\displaystyle (\\mathbf {y} -\\mathbf {X} {\\hat {\\boldsymbol {\\beta }}})\\cdot \\mathbf {X} \\mathbf {v} }  is equal to zero for *any* conformal vector, **v**. This means that y ā X Ī² ^ {\\displaystyle \\mathbf {y} -\\mathbf {X} {\\boldsymbol {\\hat {\\beta }}}}  is the shortest of all possible vectors y ā X Ī² {\\displaystyle \\mathbf {y} -\\mathbf {X} {\\boldsymbol {\\beta }}} , that is, the variance of the residuals is the minimum possible. This is illustrated at the right.
Introducing Ī³ ^ {\\displaystyle {\\hat {\\boldsymbol {\\gamma }}}}  and a matrix *K* with the assumption that a matrix \[ X K \] {\\displaystyle \[\\mathbf {X} \\ \\mathbf {K} \]} ![{\\displaystyle \[\\mathbf {X} \\ \\mathbf {K} \]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2ac770308f79814997ffbdfd971621c67b76aef6) is non-singular and *K*T *X* = 0 (cf. [Orthogonal projections](https://en.wikipedia.org/wiki/Linear_projection#Orthogonal_projections "Linear projection")), the residual vector should satisfy the following equation:
r
^
:=
y
ā
X
Ī²
^
\=
K
Ī³
^
.
{\\displaystyle {\\hat {\\mathbf {r} }}:=\\mathbf {y} -\\mathbf {X} {\\hat {\\boldsymbol {\\beta }}}=\\mathbf {K} {\\hat {\\boldsymbol {\\gamma }}}.}
The equation and solution of linear least squares are thus described as follows:
y
\=
\[
X
K
\]
\[
Ī²
^
Ī³
^
\]
,
ā
\[
Ī²
^
Ī³
^
\]
\=
\[
X
K
\]
ā
1
y
\=
\[
(
X
ā¤
X
)
ā
1
X
ā¤
(
K
ā¤
K
)
ā
1
K
ā¤
\]
y
.
{\\displaystyle {\\begin{aligned}\\mathbf {y} &={\\begin{bmatrix}\\mathbf {X} &\\mathbf {K} \\end{bmatrix}}{\\begin{bmatrix}{\\hat {\\boldsymbol {\\beta }}}\\\\{\\hat {\\boldsymbol {\\gamma }}}\\end{bmatrix}},\\\\{}\\Rightarrow {\\begin{bmatrix}{\\hat {\\boldsymbol {\\beta }}}\\\\{\\hat {\\boldsymbol {\\gamma }}}\\end{bmatrix}}&={\\begin{bmatrix}\\mathbf {X} &\\mathbf {K} \\end{bmatrix}}^{-1}\\mathbf {y} ={\\begin{bmatrix}\\left(\\mathbf {X} ^{\\top }\\mathbf {X} \\right)^{-1}\\mathbf {X} ^{\\top }\\\\\\left(\\mathbf {K} ^{\\top }\\mathbf {K} \\right)^{-1}\\mathbf {K} ^{\\top }\\end{bmatrix}}\\mathbf {y} .\\end{aligned}}}
Another way of looking at it is to consider the regression line to be a weighted average of the lines passing through the combination of any two points in the dataset.[\[15\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-15) Although this way of calculation is more computationally expensive, it provides a better intuition on OLS.
### Maximum likelihood
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=9 "Edit section: Maximum likelihood")\]
The OLS estimator is identical to the [maximum likelihood estimator](https://en.wikipedia.org/wiki/Maximum_likelihood_estimator "Maximum likelihood estimator") (MLE) under the normality assumption for the error terms.[\[16\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-16)[\[proof\]](https://en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares#Maximum_likelihood_approach "Proofs involving ordinary least squares") This normality assumption has historical importance, as it provided the basis for the early work in linear regression analysis by [Yule](https://en.wikipedia.org/wiki/Udny_Yule "Udny Yule") and [Pearson](https://en.wikipedia.org/wiki/Karl_Pearson "Karl Pearson").\[*[citation needed](https://en.wikipedia.org/wiki/Wikipedia:Citation_needed "Wikipedia:Citation needed")*\] From the properties of MLE, we can infer that the OLS estimator is asymptotically efficient (in the sense of attaining the [CramĆ©rāRao bound](https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_bound "CramĆ©rāRao bound") for variance) if the normality assumption is satisfied.[\[17\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hayashi_2000_loc=page_52-17)
### Generalized method of moments
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=10 "Edit section: Generalized method of moments")\]
In [iid](https://en.wikipedia.org/wiki/Iid "Iid") case the OLS estimator can also be viewed as a [GMM](https://en.wikipedia.org/wiki/Generalized_method_of_moments "Generalized method of moments") estimator arising from the moment conditions
E
\[
x
i
(
y
i
ā
x
i
T
Ī²
)
\]
\=
0\.
{\\displaystyle \\mathrm {E} {\\big \[}\\,x\_{i}\\left(y\_{i}-x\_{i}^{\\operatorname {T} }\\beta \\right)\\,{\\big \]}=0.}
![{\\displaystyle \\mathrm {E} {\\big \[}\\,x\_{i}\\left(y\_{i}-x\_{i}^{\\operatorname {T} }\\beta \\right)\\,{\\big \]}=0.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1d7894c141dad7e6dae3aed8bb708aada174daf2)
These moment conditions state that the regressors should be uncorrelated with the errors. Since *xi* is a *p*\-vector, the number of moment conditions is equal to the dimension of the parameter vector *Ī²*, and thus the system is exactly identified. This is the so-called classical GMM case, when the estimator does not depend on the choice of the weighting matrix.
Note that the original strict exogeneity assumption E\[*Īµi* \| *xi*\] = 0 implies a far richer set of moment conditions than stated above. In particular, this assumption implies that for any vector-function Ę, the moment condition E\[*Ę*(*xi*)Ā·*Īµi*\] = 0 will hold. However it can be shown using the [GaussāMarkov theorem](https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem "GaussāMarkov theorem") that the optimal choice of function Ę is to take *Ę*(*x*) = *x*, which results in the moment equation posted above.
## Assumptions
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=11 "Edit section: Assumptions")\]
See also: [Linear regression Ā§ Assumptions](https://en.wikipedia.org/wiki/Linear_regression#Assumptions "Linear regression")
There are several different frameworks in which the [linear regression model](https://en.wikipedia.org/wiki/Linear_regression_model "Linear regression model") can be cast in order to make the OLS technique applicable. Each of these settings produces the same formulas and same results. The only difference is the interpretation and the assumptions which have to be imposed in order for the method to give meaningful results. The choice of the applicable framework depends mostly on the nature of data in hand, and on the inference task which has to be performed.
One of the lines of difference in interpretation is whether to treat the regressors as random variables, or as predefined constants. In the first case (**random design**) the regressors *xi* are random and sampled together with the *yi*'s from some [population](https://en.wikipedia.org/wiki/Statistical_population "Statistical population"), as in an [observational study](https://en.wikipedia.org/wiki/Observational_study "Observational study"). This approach allows for more natural study of the [asymptotic properties](https://en.wikipedia.org/wiki/Asymptotic_theory_\(statistics\) "Asymptotic theory (statistics)") of the estimators. In the other interpretation (**fixed design**), the regressors *X* are treated as known constants set by a [design](https://en.wikipedia.org/wiki/Design_of_experiments "Design of experiments"), and *y* is sampled conditionally on the values of *X* as in an [experiment](https://en.wikipedia.org/wiki/Experiment "Experiment"). For practical purposes, this distinction is often unimportant, since estimation and inference is carried out while conditioning on *X*. All results stated in this article are within the random design framework.
The classical model focuses on the "finite sample" estimation and inference, meaning that the number of observations *n* is fixed. This contrasts with the other approaches, which study the [asymptotic behavior](https://en.wikipedia.org/wiki/Asymptotic_theory_\(statistics\) "Asymptotic theory (statistics)") of OLS, and in which the behavior at a large number of samples is studied. To prove finite sample unbiasedness of the OLS estimator, we require the following assumptions.
[](https://en.wikipedia.org/wiki/File:Polyreg_scheffe.svg)
Example of a cubic polynomial regression, which is a type of linear regression. Although *polynomial regression* fits a curve model to the data, as a [statistical estimation](https://en.wikipedia.org/wiki/Estimation_theory "Estimation theory") problem it is linear, in the sense that the conditional expectation function
E
\[
y
\|
x
\]
{\\displaystyle \\mathbb {E} \[y\|x\]}
![{\\displaystyle \\mathbb {E} \[y\|x\]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/391e556522e79e05efc7dd6a5248cc0e5b0c4651)
is linear in the unknown [parameters](https://en.wikipedia.org/wiki/Parameter "Parameter") that are estimated from the [data](https://en.wikipedia.org/wiki/Data "Data"). For this reason, polynomial regression is considered to be a special case of [multiple linear regression](https://en.wikipedia.org/wiki/Multiple_linear_regression "Multiple linear regression").
- **Exogeneity**. The regressors do not [covary](https://en.wikipedia.org/wiki/Covariance "Covariance") with the error term:
E
\[
Īµ
i
x
i
\]
\=
0\.
{\\displaystyle \\mathbb {E} \[\\varepsilon \_{i}x\_{i}\]=0.}
![{\\displaystyle \\mathbb {E} \[\\varepsilon \_{i}x\_{i}\]=0.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/865dc6e3427f19d12afd3bed41c45bb7661ef289)
This requires, for example, that there are no [omitted variables](https://en.wikipedia.org/wiki/Omitted_variable_bias "Omitted variable bias") that covary with observed variables and affect the response variable. An alternative (but stronger) statement that is often required when explaining linear regression in [mathematical statistics](https://en.wikipedia.org/wiki/Mathematical_statistics "Mathematical statistics") is that the predictor variables *x* can be treated as fixed values, rather than [random variables](https://en.wikipedia.org/wiki/Random_variable "Random variable"). This stronger form means, for example, that the predictor variables are assumed to be error-free, that is, not contaminated with measurement error. Although this assumption is not realistic in many settings, dropping it leads to more complex [errors-in-variables models](https://en.wikipedia.org/wiki/Errors-in-variables_models "Errors-in-variables models"), [instrumental variable models](https://en.wikipedia.org/wiki/Instrumental_variable "Instrumental variable") and the like.
- **Linearity**, or **correct specification**. This means that the mean of the response variable is a [linear combination](https://en.wikipedia.org/wiki/Linear_combination "Linear combination") of the parameters (regression coefficients) and the predictor variables. Note that this assumption is much less restrictive than it may at first seem. Because the predictor variables are treated as fixed values (see above), linearity is really only a restriction on the parameters. The predictor variables themselves can be arbitrarily transformed, and in fact multiple copies of the same underlying predictor variable can be added, each one transformed differently. This technique is used, for example, in [polynomial regression](https://en.wikipedia.org/wiki/Polynomial_regression "Polynomial regression"), which uses linear regression to fit the response variable as an arbitrary [polynomial](https://en.wikipedia.org/wiki/Polynomial "Polynomial") function (up to a given degree) of a predictor variable. With this much flexibility, models such as polynomial regression often have "too much power", in that they tend to [overfit](https://en.wikipedia.org/wiki/Overfit "Overfit") the data. As a result, some kind of [regularization](https://en.wikipedia.org/wiki/Regularization_\(mathematics\) "Regularization (mathematics)") must typically be used to prevent unreasonable solutions coming out of the estimation process. Common examples are [ridge regression](https://en.wikipedia.org/wiki/Ridge_regression "Ridge regression") and [lasso regression](https://en.wikipedia.org/wiki/Lasso_regression "Lasso regression"). [Bayesian linear regression](https://en.wikipedia.org/wiki/Bayesian_linear_regression "Bayesian linear regression") can also be used, which by its nature is more or less immune to the problem of overfitting. (In fact, [ridge regression](https://en.wikipedia.org/wiki/Ridge_regression "Ridge regression") and [lasso regression](https://en.wikipedia.org/wiki/Lasso_regression "Lasso regression") can both be viewed as special cases of Bayesian linear regression, with particular types of [prior distributions](https://en.wikipedia.org/wiki/Prior_distribution "Prior distribution") placed on the regression coefficients.)
- [](https://en.wikipedia.org/wiki/File:Heteroscedasticity_in_Linear_Regression.png)
Visualization of heteroscedasticity in a scatter plot against 100 random fitted values using Matlab
**Constant variance** or **[homoscedasticity](https://en.wikipedia.org/wiki/Homoscedasticity "Homoscedasticity")**. This means that the variance of the errors does not depend on the values of the predictor variables:
E
\[
Īµ
i
2
\|
x
i
\]
\=
Ļ
2
.
{\\displaystyle \\mathbb {E} \[\\varepsilon \_{i}^{2}\|x\_{i}\]=\\sigma ^{2}.}
![{\\displaystyle \\mathbb {E} \[\\varepsilon \_{i}^{2}\|x\_{i}\]=\\sigma ^{2}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/93072a3d09cd2b89e6a544dfd4c7cbb017200cbd)
Thus the variability of the responses for given fixed values of the predictors is the same regardless of how large or small the responses are. This is often not the case, as a variable whose mean is large will typically have a greater variance than one whose mean is small. For example, a person whose income is predicted to be \$100,000 may easily have an actual income of \$80,000 or \$120,000āi.e., a [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation "Standard deviation") of around \$20,000āwhile another person with a predicted income of \$10,000 is unlikely to have the same \$20,000 standard deviation, since that would imply their actual income could vary anywhere between ā\$10,000 and \$30,000. (In fact, as this shows, in many casesāoften the same cases where the assumption of normally distributed errors failsāthe variance or standard deviation should be predicted to be proportional to the mean, rather than constant.) The absence of homoscedasticity is called [heteroscedasticity](https://en.wikipedia.org/wiki/Heteroscedasticity "Heteroscedasticity"). In order to check this assumption, a plot of residuals versus predicted values (or the values of each individual predictor) can be examined for a "fanning effect" (i.e., increasing or decreasing vertical spread as one moves left to right on the plot). A plot of the absolute or squared residuals versus the predicted values (or each predictor) can also be examined for a trend or curvature. Formal tests can also be used; see [Heteroscedasticity](https://en.wikipedia.org/wiki/Heteroscedasticity "Heteroscedasticity"). The presence of heteroscedasticity will result in an overall "average" estimate of variance being used instead of one that takes into account the true variance structure. This leads to less precise (but in the case of [ordinary least squares](), not biased) parameter estimates and biased standard errors, resulting in misleading tests and interval estimates. The [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error "Mean squared error") for the model will also be wrong. Various estimation techniques including [weighted least squares](https://en.wikipedia.org/wiki/Weighted_least_squares "Weighted least squares") and the use of [heteroscedasticity-consistent standard errors](https://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors "Heteroscedasticity-consistent standard errors") can handle heteroscedasticity in a quite general way. [Bayesian linear regression](https://en.wikipedia.org/wiki/Bayesian_linear_regression "Bayesian linear regression") techniques can also be used when the variance is assumed to be a function of the mean. It is also possible in some cases to fix the problem by applying a transformation to the response variable (e.g., fitting the [logarithm](https://en.wikipedia.org/wiki/Logarithm "Logarithm") of the response variable using a linear regression model, which implies that the response variable itself has a [log-normal distribution](https://en.wikipedia.org/wiki/Log-normal_distribution "Log-normal distribution") rather than a [normal distribution](https://en.wikipedia.org/wiki/Normal_distribution "Normal distribution")).
[](https://en.wikipedia.org/wiki/File:Independence_of_Errors_Assumption_for_Linear_Regressions.png)
To check for violations of the assumptions of linearity, constant variance, and independence of errors within a linear regression model, the residuals are typically plotted against the predicted values (or each of the individual predictors). An apparently random scatter of points about the horizontal midline at 0 is ideal, but cannot rule out certain kinds of violations such as [autocorrelation](https://en.wikipedia.org/wiki/Autocorrelation "Autocorrelation") in the errors or their correlation with one or more covariates.
- **Uncorrelatedness of errors**. This assumes that the errors of the response variables are uncorrelated with each other:
E
\[
Īµ
i
Īµ
j
\|
x
i
,
x
j
\]
\=
0\.
{\\displaystyle \\mathbb {E} \[\\varepsilon \_{i}\\varepsilon \_{j}\|x\_{i},x\_{j}\]=0.}
![{\\displaystyle \\mathbb {E} \[\\varepsilon \_{i}\\varepsilon \_{j}\|x\_{i},x\_{j}\]=0.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d2049f9a6b8ce702724b8271a5912158bf3f0fc3)
Some methods such as [generalized least squares](https://en.wikipedia.org/wiki/Generalized_least_squares "Generalized least squares") are capable of handling correlated errors, although they typically require significantly more data unless some sort of [regularization](https://en.wikipedia.org/wiki/Regularization_\(mathematics\) "Regularization (mathematics)") is used to bias the model towards assuming uncorrelated errors. [Bayesian linear regression](https://en.wikipedia.org/wiki/Bayesian_linear_regression "Bayesian linear regression") is a general way of handling this issue. Full [statistical independence](https://en.wikipedia.org/wiki/Statistical_independence "Statistical independence") is a stronger condition than mere lack of correlation and is often not needed, although it implies mean-independence.
- **Lack of perfect multicollinearity** in the predictors. For standard [least squares](https://en.wikipedia.org/wiki/Least_squares "Least squares") estimation methods, the design matrix *X* must have full [column rank](https://en.wikipedia.org/wiki/Column_rank "Column rank") *p*: [\[18\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hayashi_2000_loc=page_10-18)
Pr
\[
rank
ā”
(
X
)
\=
p
\]
\=
1\.
{\\displaystyle \\Pr \\!{\\big \[}\\,\\operatorname {rank} (X)=p\\,{\\big \]}=1.}
![{\\displaystyle \\Pr \\!{\\big \[}\\,\\operatorname {rank} (X)=p\\,{\\big \]}=1.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6a11be3b89ce51c6441155fddbe512a991132fbf)
If this assumption is violated, perfect [multicollinearity](https://en.wikipedia.org/wiki/Multicollinearity "Multicollinearity") exists in the predictor variables, meaning a linear relationship exists between two or more predictor variables. Multicollinearity can be caused by accidentally duplicating a variable in the data, using a linear transformation of a variable along with the original (e.g., the same temperature measurements expressed in Fahrenheit and Celsius), or including a linear combination of multiple variables in the model, such as their mean. It can also happen if there is too little data available compared to the number of parameters to be estimated (e.g., fewer data points than regression coefficients). Near violations of this assumption, where predictors are highly but not perfectly correlated, can reduce the precision of parameter estimates (see [Variance inflation factor](https://en.wikipedia.org/wiki/Variance_inflation_factor "Variance inflation factor")). In the case of perfect multicollinearity, the parameter vector ***Ī²*** will be [non-identifiable](https://en.wikipedia.org/wiki/Non-identifiable "Non-identifiable")āit has no unique solution. In such a case, only some of the parameters can be identified (i.e., their values can only be estimated within some linear subspace of the full parameter space **R***p*). See [partial least squares regression](https://en.wikipedia.org/wiki/Partial_least_squares_regression "Partial least squares regression"). Methods for fitting linear models with multicollinearity have been developed,[\[19\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Tibshirani-1996-19)[\[20\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Efron-2004-20)[\[21\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hawkins-1973-21)[\[22\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Jolliffe-1982-22) some of which require additional assumptions such as "effect sparsity"āthat a large fraction of the effects are exactly zero. Note that the more computationally expensive iterated algorithms for parameter estimation, such as those used in [generalized linear models](https://en.wikipedia.org/wiki/Generalized_linear_model "Generalized linear model"), do not suffer from this problem.
Violations of these assumptions can result in biased estimations of ***Ī²***, biased standard errors, untrustworthy confidence intervals and significance tests. Beyond these assumptions, several other statistical properties of the data strongly influence the performance of different estimation methods:
- The statistical relationship between the error terms and the regressors plays an important role in determining whether an estimation procedure has desirable sampling properties such as being unbiased and consistent.
- The arrangement, or [probability distribution](https://en.wikipedia.org/wiki/Probability_distribution "Probability distribution") of the predictor variables **x** has a major influence on the precision of estimates of ***Ī²***. [Sampling](https://en.wikipedia.org/wiki/Sampling_\(statistics\) "Sampling (statistics)") and [design of experiments](https://en.wikipedia.org/wiki/Design_of_experiments "Design of experiments") are highly developed subfields of statistics that provide guidance for collecting data in such a way to achieve a precise estimate of ***Ī²***.
## Properties
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=12 "Edit section: Properties")\]
### Finite sample properties
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=13 "Edit section: Finite sample properties")\]
First of all, under the *strict exogeneity* assumption the OLS estimators Ī² ^ {\\displaystyle \\scriptstyle {\\hat {\\beta }}}  and *s*2 are [unbiased](https://en.wikipedia.org/wiki/Bias_of_an_estimator "Bias of an estimator"), meaning that their expected values coincide with the true values of the parameters:[\[23\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-23)[\[proof\]](https://en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares#Unbiasedness_of_.CE.B2.CC.82 "Proofs involving ordinary least squares")
E
ā”
\[
Ī²
^
ā£
X
\]
\=
Ī²
,
E
ā”
\[
s
2
ā£
X
\]
\=
Ļ
2
.
{\\displaystyle \\operatorname {E} \[\\,{\\hat {\\beta }}\\mid X\\,\]=\\beta ,\\quad \\operatorname {E} \[\\,s^{2}\\mid X\\,\]=\\sigma ^{2}.}
![{\\displaystyle \\operatorname {E} \[\\,{\\hat {\\beta }}\\mid X\\,\]=\\beta ,\\quad \\operatorname {E} \[\\,s^{2}\\mid X\\,\]=\\sigma ^{2}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/67bc2fd0f90c46da207712893fdcea01e729026c)
If the strict exogeneity does not hold (as is the case with many [time series](https://en.wikipedia.org/wiki/Time_series "Time series") models, where exogeneity is assumed only with respect to the past shocks but not the future ones), then these estimators will be biased in finite samples.
The *[variance-covariance matrix](https://en.wikipedia.org/wiki/Variance-covariance_matrix "Variance-covariance matrix")* (or simply *covariance matrix*) of Ī² ^ {\\displaystyle \\scriptstyle {\\hat {\\beta }}}  is equal to[\[24\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-HayashiFSP-24)
Var
ā”
\[
Ī²
^
ā£
X
\]
\=
Ļ
2
(
X
T
X
)
ā
1
\=
Ļ
2
Q
.
{\\displaystyle \\operatorname {Var} \[\\,{\\hat {\\beta }}\\mid X\\,\]=\\sigma ^{2}\\left(X^{\\operatorname {T} }X\\right)^{-1}=\\sigma ^{2}Q.}
![{\\displaystyle \\operatorname {Var} \[\\,{\\hat {\\beta }}\\mid X\\,\]=\\sigma ^{2}\\left(X^{\\operatorname {T} }X\\right)^{-1}=\\sigma ^{2}Q.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/08f6cb596d94073731ee47f4a2571dbbfc1d214a)
In particular, the standard error of each coefficient Ī² ^ j {\\displaystyle \\scriptstyle {\\hat {\\beta }}\_{j}}  is equal to square root of the *j*\-th diagonal element of this matrix. The estimate of this standard error is obtained by replacing the unknown quantity *Ļ*2 with its estimate *s*2. Thus,
s
.
e
.
^
(
Ī²
^
j
)
\=
s
2
(
X
T
X
)
j
j
ā
1
{\\displaystyle {\\widehat {\\operatorname {s.\\!e.} }}({\\hat {\\beta }}\_{j})={\\sqrt {s^{2}\\left(X^{\\operatorname {T} }X\\right)\_{jj}^{-1}}}}
It can also be easily shown that the estimator Ī² ^ {\\displaystyle \\scriptstyle {\\hat {\\beta }}}  is uncorrelated with the residuals from the model:[\[24\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-HayashiFSP-24)
Cov
ā”
\[
Ī²
^
,
Īµ
^
ā£
X
\]
\=
0\.
{\\displaystyle \\operatorname {Cov} \[\\,{\\hat {\\beta }},{\\hat {\\varepsilon }}\\mid X\\,\]=0.}
![{\\displaystyle \\operatorname {Cov} \[\\,{\\hat {\\beta }},{\\hat {\\varepsilon }}\\mid X\\,\]=0.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/664c1a5e37957a1aa2ae381b9bcb07350c2c816c)
The *[GaussāMarkov theorem](https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem "GaussāMarkov theorem")* states that under the *spherical errors* assumption (that is, the errors should be [uncorrelated](https://en.wikipedia.org/wiki/Uncorrelated "Uncorrelated") and [homoscedastic](https://en.wikipedia.org/wiki/Homoscedastic "Homoscedastic")) the estimator Ī² ^ {\\displaystyle \\scriptstyle {\\hat {\\beta }}}  is efficient in the class of linear unbiased estimators. This is called the *best linear unbiased estimator* (BLUE). Efficiency should be understood as if we were to find some other estimator Ī² ~ {\\displaystyle \\scriptstyle {\\tilde {\\beta }}}  which would be linear in *y* and unbiased, then [\[24\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-HayashiFSP-24)
Var
ā”
\[
Ī²
~
ā£
X
\]
ā
Var
ā”
\[
Ī²
^
ā£
X
\]
ā„
0
{\\displaystyle \\operatorname {Var} \[\\,{\\tilde {\\beta }}\\mid X\\,\]-\\operatorname {Var} \[\\,{\\hat {\\beta }}\\mid X\\,\]\\geq 0}
![{\\displaystyle \\operatorname {Var} \[\\,{\\tilde {\\beta }}\\mid X\\,\]-\\operatorname {Var} \[\\,{\\hat {\\beta }}\\mid X\\,\]\\geq 0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/53796c9205889cc4d675b9749a58eb97fcd998f1)
in the sense that this is a [nonnegative-definite matrix](https://en.wikipedia.org/wiki/Nonnegative-definite_matrix "Nonnegative-definite matrix"). This theorem establishes optimality only in the class of linear unbiased estimators, which is quite restrictive. Depending on the distribution of the error terms *Īµ*, other, non-linear estimators may provide better results than OLS.
#### Assuming normality
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=14 "Edit section: Assuming normality")\]
The properties listed so far are all valid regardless of the underlying distribution of the error terms. However, if you are willing to assume that the *normality assumption* holds (that is, that *Īµ* ~ *N*(0, *Ļ*2*In*)), then additional properties of the OLS estimators can be stated.
The estimator Ī² ^ {\\displaystyle \\scriptstyle {\\hat {\\beta }}}  is normally distributed, with mean and variance as given before:[\[25\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-25)
Ī²
^
ā¼
N
(
Ī²
,
Ļ
2
(
X
T
X
)
ā
1
)
.
{\\displaystyle {\\hat {\\beta }}\\ \\sim \\ {\\mathcal {N}}{\\big (}\\beta ,\\ \\sigma ^{2}(X^{\\mathrm {T} }X)^{-1}{\\big )}.}
This estimator reaches the [CramĆ©rāRao bound](https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_bound "CramĆ©rāRao bound") for the model, and thus is optimal in the class of all unbiased estimators.[\[17\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hayashi_2000_loc=page_52-17) Note that unlike the [GaussāMarkov theorem](https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem "GaussāMarkov theorem"), this result establishes optimality among both linear and non-linear estimators, but only in the case of normally distributed error terms.
The estimator *s*2 will be proportional to the [chi-squared distribution](https://en.wikipedia.org/wiki/Chi-squared_distribution "Chi-squared distribution"):[\[26\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-26)
s
2
ā¼
Ļ
2
n
ā
p
ā
Ļ
n
ā
p
2
{\\displaystyle s^{2}\\ \\sim \\ {\\frac {\\sigma ^{2}}{n-p}}\\cdot \\chi \_{n-p}^{2}}
The variance of this estimator is equal to 2*Ļ*4/(*n* ā *p*), which does not attain the [CramĆ©rāRao bound](https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_bound "CramĆ©rāRao bound") of 2*Ļ*4/*n*. However it was shown that there are no unbiased estimators of *Ļ*2 with variance smaller than that of the estimator *s*2.[\[27\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-27) If we are willing to allow biased estimators, and consider the class of estimators that are proportional to the sum of squared residuals (SSR) of the model, then the best (in the sense of the [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error "Mean squared error")) estimator in this class will be ~*Ļ*2 = SSR */* (*n* ā *p* + 2), which even beats the CramĆ©rāRao bound in case when there is only one regressor (*p* = 1).[\[28\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-28)
Moreover, the estimators Ī² ^ {\\displaystyle \\scriptstyle {\\hat {\\beta }}}  and *s*2 are [independent](https://en.wikipedia.org/wiki/Independent_random_variables "Independent random variables"),[\[29\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-29) the fact which comes in useful when constructing the t- and F-tests for the regression.
#### Influential observations
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=15 "Edit section: Influential observations")\]
Main article: [Influential observation](https://en.wikipedia.org/wiki/Influential_observation "Influential observation")
See also: [Leverage (statistics)](https://en.wikipedia.org/wiki/Leverage_\(statistics\) "Leverage (statistics)")
As was mentioned before, the estimator Ī² ^ {\\displaystyle {\\hat {\\beta }}}  is linear in *y*, meaning that it represents a linear combination of the dependent variables *yi*. The weights in this linear combination are functions of the regressors *X*, and generally are unequal. The observations with high weights are called **influential** because they have a more pronounced effect on the value of the estimator.
To analyze which observations are influential we remove a specific *j*\-th observation and consider how much the estimated quantities are going to change (similarly to the [jackknife method](https://en.wikipedia.org/wiki/Jackknife_method "Jackknife method")). It can be shown that the change in the OLS estimator for *Ī²* will be equal to [\[30\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-DvdMck33-30)
Ī²
^
(
j
)
ā
Ī²
^
\=
ā
1
1
ā
h
j
(
X
T
X
)
ā
1
x
j
T
Īµ
^
j
,
{\\displaystyle {\\hat {\\beta }}^{(j)}-{\\hat {\\beta }}=-{\\frac {1}{1-h\_{j}}}(X^{\\mathrm {T} }X)^{-1}x\_{j}^{\\mathrm {T} }{\\hat {\\varepsilon }}\_{j}\\,,}
where *hj* = *xj*T (*X*T*X*)ā1*xj* is the *j*\-th diagonal element of the hat matrix *P*, and *xj* is the vector of regressors corresponding to the *j*\-th observation. Similarly, the change in the predicted value for *j*\-th observation resulting from omitting that observation from the dataset will be equal to [\[30\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-DvdMck33-30)
y
^
j
(
j
)
ā
y
^
j
\=
x
j
T
Ī²
^
(
j
)
ā
x
j
T
Ī²
^
\=
ā
h
j
1
ā
h
j
Īµ
^
j
{\\displaystyle {\\hat {y}}\_{j}^{(j)}-{\\hat {y}}\_{j}=x\_{j}^{\\mathrm {T} }{\\hat {\\beta }}^{(j)}-x\_{j}^{\\operatorname {T} }{\\hat {\\beta }}=-{\\frac {h\_{j}}{1-h\_{j}}}\\,{\\hat {\\varepsilon }}\_{j}}
From the properties of the hat matrix, 0 ā¤ *hj* ā¤ 1, and they sum up to *p*, so that on average *hj* ā *p/n*. These quantities *hj* are called the **leverages**, and observations with high *hj* are called **leverage points**.[\[31\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-31) Usually the observations with high leverage ought to be scrutinized more carefully, in case they are erroneous, or outliers, or in some other way atypical of the rest of the dataset.
#### Partitioned regression
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=16 "Edit section: Partitioned regression")\]
Sometimes the variables and corresponding parameters in the regression can be logically split into two groups, so that the regression takes form
y
\=
X
1
Ī²
1
\+
X
2
Ī²
2
\+
Īµ
,
{\\displaystyle y=X\_{1}\\beta \_{1}+X\_{2}\\beta \_{2}+\\varepsilon ,}
where *X*1 and *X*2 have dimensions *n*Ć*p*1, *n*Ć*p*2, and *Ī²*1, *Ī²*2 are *p*1Ć1 and *p*2Ć1 vectors, with *p*1 + *p*2 = *p*.
The **[FrischāWaughāLovell theorem](https://en.wikipedia.org/wiki/Frisch%E2%80%93Waugh%E2%80%93Lovell_theorem "FrischāWaughāLovell theorem")** states that in this regression the residuals Īµ ^ {\\displaystyle {\\hat {\\varepsilon }}}  and the OLS estimate Ī² ^ 2 {\\displaystyle \\scriptstyle {\\hat {\\beta }}\_{2}}  will be numerically identical to the residuals and the OLS estimate for *Ī²*2 in the following regression:[\[32\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-32)
M
1
y
\=
M
1
X
2
Ī²
2
\+
Ī·
,
{\\displaystyle M\_{1}y=M\_{1}X\_{2}\\beta \_{2}+\\eta \\,,}
where *M*1 is the [annihilator matrix](https://en.wikipedia.org/wiki/Annihilator_matrix "Annihilator matrix") for regressors *X*1.
The theorem can be used to establish a number of theoretical results. For example, having a regression with a constant and another regressor is equivalent to subtracting the means from the dependent variable and the regressor and then running the regression for the de-meaned variables but without the constant term.
### Large sample properties
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=17 "Edit section: Large sample properties")\]
The least squares estimators are [point estimates](https://en.wikipedia.org/wiki/Point_estimate "Point estimate") of the linear regression model parameters *Ī²*. However, generally we also want to know how close those estimates might be to the true values of parameters. In other words, we want to construct the [interval estimates](https://en.wikipedia.org/wiki/Interval_estimate "Interval estimate").
Since we have not made any assumption about the distribution of error term *Īµi*, it is impossible to infer the distribution of the estimators Ī² ^ {\\displaystyle {\\hat {\\beta }}}  and Ļ ^ 2 {\\displaystyle {\\hat {\\sigma }}^{2}} . Nevertheless, we can apply the [central limit theorem](https://en.wikipedia.org/wiki/Central_limit_theorem "Central limit theorem") to derive their *asymptotic* properties as sample size *n* goes to infinity. While the sample size is necessarily finite, it is customary to assume that *n* is "large enough" so that the true distribution of the OLS estimator is close to its asymptotic limit.
We can show that under the model assumptions, the least squares estimator for *Ī²* is [consistent](https://en.wikipedia.org/wiki/Consistent_estimator "Consistent estimator") (that is Ī² ^ {\\displaystyle {\\hat {\\beta }}}  [converges in probability](https://en.wikipedia.org/wiki/Convergence_of_random_variables#Convergence_in_probability "Convergence of random variables") to *Ī²*) and asymptotically normal:[\[proof\]](https://en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares#Consistency_and_asymptotic_normality_of_.CE.B2.CC.82 "Proofs involving ordinary least squares")
(
Ī²
^
ā
Ī²
)
ā
d
N
(
0
,
Ļ
2
Q
x
x
ā
1
)
,
{\\displaystyle ({\\hat {\\beta }}-\\beta )\\ {\\xrightarrow {d}}\\ {\\mathcal {N}}{\\big (}0,\\;\\sigma ^{2}Q\_{xx}^{-1}{\\big )},}
where Q x x \= X T X . {\\displaystyle Q\_{xx}=X^{\\operatorname {T} }X.} 
#### Inference
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=18 "Edit section: Inference")\]
Main articles: [Confidence interval](https://en.wikipedia.org/wiki/Confidence_interval "Confidence interval") and [Prediction interval](https://en.wikipedia.org/wiki/Prediction_interval "Prediction interval")
Using this asymptotic distribution, approximate two-sided confidence intervals for the *j*\-th component of the vector Ī² ^ {\\displaystyle {\\hat {\\beta }}}  can be constructed as
Ī²
j
ā
\[
Ī²
^
j
Ā±
q
1
ā
Ī±
2
N
(
0
,
1
)
Ļ
^
2
\[
Q
x
x
ā
1
\]
j
j
\]
{\\displaystyle \\beta \_{j}\\in {\\bigg \[}\\ {\\hat {\\beta }}\_{j}\\pm q\_{1-{\\frac {\\alpha }{2}}}^{{\\mathcal {N}}(0,1)}\\!{\\sqrt {{\\hat {\\sigma }}^{2}\\left\[Q\_{xx}^{-1}\\right\]\_{jj}}}\\ {\\bigg \]}}
![{\\displaystyle \\beta \_{j}\\in {\\bigg \[}\\ {\\hat {\\beta }}\_{j}\\pm q\_{1-{\\frac {\\alpha }{2}}}^{{\\mathcal {N}}(0,1)}\\!{\\sqrt {{\\hat {\\sigma }}^{2}\\left\[Q\_{xx}^{-1}\\right\]\_{jj}}}\\ {\\bigg \]}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cf79688aac9f662ff39253fbfb0d234246d370e5)
at the 1 ā *Ī±* confidence level,
where *q* denotes the [quantile function](https://en.wikipedia.org/wiki/Quantile_function "Quantile function") of standard normal distribution, and \[Ā·\]*jj* is the *j*\-th diagonal element of a matrix.
Similarly, the least squares estimator for *Ļ*2 is also consistent and asymptotically normal (provided that the fourth moment of *Īµi* exists) with limiting distribution
(
Ļ
^
2
ā
Ļ
2
)
ā
d
N
(
0
,
E
ā”
\[
Īµ
i
4
\]
ā
Ļ
4
)
.
{\\displaystyle ({\\hat {\\sigma }}^{2}-\\sigma ^{2})\\ {\\xrightarrow {d}}\\ {\\mathcal {N}}\\left(0,\\;\\operatorname {E} \\left\[\\varepsilon \_{i}^{4}\\right\]-\\sigma ^{4}\\right).}
![{\\displaystyle ({\\hat {\\sigma }}^{2}-\\sigma ^{2})\\ \\xrightarrow {d} \\ {\\mathcal {N}}\\left(0,\\;\\operatorname {E} \\left\[\\varepsilon \_{i}^{4}\\right\]-\\sigma ^{4}\\right).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7c909dea2a4f0bf40e253680b953d1bfbb66298f)
These asymptotic distributions can be used for prediction, testing hypotheses, constructing other estimators, etc.. As an example consider the problem of prediction. Suppose x 0 {\\displaystyle x\_{0}}  is some point within the domain of distribution of the regressors, and one wants to know what the response variable would have been at that point. The [mean response](https://en.wikipedia.org/wiki/Mean_response "Mean response") is the quantity y 0 \= x 0 T Ī² {\\displaystyle y\_{0}=x\_{0}^{\\mathrm {T} }\\beta } , whereas the [predicted response](https://en.wikipedia.org/wiki/Predicted_response "Predicted response") is y ^ 0 \= x 0 T Ī² ^ {\\displaystyle {\\hat {y}}\_{0}=x\_{0}^{\\mathrm {T} }{\\hat {\\beta }}} . Clearly the predicted response is a random variable, its distribution can be derived from that of Ī² ^ {\\displaystyle {\\hat {\\beta }}} :
(
y
^
0
ā
y
0
)
ā
d
N
(
0
,
Ļ
2
x
0
T
Q
x
x
ā
1
x
0
)
,
{\\displaystyle \\left({\\hat {y}}\_{0}-y\_{0}\\right)\\ {\\xrightarrow {d}}\\ {\\mathcal {N}}\\left(0,\\;\\sigma ^{2}x\_{0}^{\\mathrm {T} }Q\_{xx}^{-1}x\_{0}\\right),}
which allows construct confidence intervals for mean response y 0 {\\displaystyle y\_{0}}  to be constructed:
y
0
ā
\[
x
0
T
Ī²
^
Ā±
q
1
ā
Ī±
2
N
(
0
,
1
)
Ļ
^
2
x
0
T
Q
x
x
ā
1
x
0
\]
{\\displaystyle y\_{0}\\in \\left\[\\ x\_{0}^{\\mathrm {T} }{\\hat {\\beta }}\\pm q\_{1-{\\frac {\\alpha }{2}}}^{{\\mathcal {N}}(0,1)}\\!{\\sqrt {{\\hat {\\sigma }}^{2}x\_{0}^{\\mathrm {T} }Q\_{xx}^{-1}x\_{0}}}\\ \\right\]}
![{\\displaystyle y\_{0}\\in \\left\[\\ x\_{0}^{\\mathrm {T} }{\\hat {\\beta }}\\pm q\_{1-{\\frac {\\alpha }{2}}}^{{\\mathcal {N}}(0,1)}\\!{\\sqrt {{\\hat {\\sigma }}^{2}x\_{0}^{\\mathrm {T} }Q\_{xx}^{-1}x\_{0}}}\\ \\right\]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cf86d7a311c97d35fb6e039c3cd74bc9f3e752bf)
at the 1 ā *Ī±* confidence level.
#### Hypothesis testing
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=19 "Edit section: Hypothesis testing")\]
Main article: [Hypothesis testing](https://en.wikipedia.org/wiki/Hypothesis_testing "Hypothesis testing")
| | |
|---|---|
| [![\[icon\]](https://upload.wikimedia.org/wikipedia/commons/thumb/1/1c/Wiki_letter_w_cropped.svg/20px-Wiki_letter_w_cropped.svg.png)](https://en.wikipedia.org/wiki/File:Wiki_letter_w_cropped.svg) | This section **needs expansion**. You can help by [adding missing information](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=). *(February 2017)* |
Two hypothesis tests are particularly widely used. First, one wants to know if the estimated regression equation is any better than simply predicting that all values of the response variable equal its sample mean (if not, it is said to have no explanatory power). The [null hypothesis](https://en.wikipedia.org/wiki/Null_hypothesis "Null hypothesis") of no explanatory value of the estimated regression is tested using an [F-test](https://en.wikipedia.org/wiki/F-test "F-test"). If the calculated F-value is found to be large enough to exceed its critical value for the pre-chosen level of significance, the null hypothesis is rejected and the [alternative hypothesis](https://en.wikipedia.org/wiki/Alternative_hypothesis "Alternative hypothesis"), that the regression has explanatory power, is accepted. Otherwise, the null hypothesis of no explanatory power is accepted.
Second, for each explanatory variable of interest, one wants to know whether its estimated coefficient differs significantly from zeroāthat is, whether this particular explanatory variable in fact has explanatory power in predicting the response variable. Here the null hypothesis is that the true coefficient is zero. This hypothesis is tested by computing the coefficient's [t-statistic](https://en.wikipedia.org/wiki/T-statistic "T-statistic"), as the ratio of the coefficient estimate to its [standard error](https://en.wikipedia.org/wiki/Standard_error "Standard error"). If the t-statistic is larger than a predetermined value, the null hypothesis is rejected and the variable is found to have explanatory power, with its coefficient significantly different from zero. Otherwise, the null hypothesis of a zero value of the true coefficient is accepted.
In addition, the [Chow test](https://en.wikipedia.org/wiki/Chow_test "Chow test") is used to test whether two subsamples both have the same underlying true coefficient values. The sum of squared residuals of regressions on each of the subsets and on the combined data set are compared by computing an F-statistic; if this exceeds a critical value, the null hypothesis of no difference between the two subsets is rejected; otherwise, it is accepted.
### Violations of assumptions
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=20 "Edit section: Violations of assumptions")\]
#### Time series model
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=21 "Edit section: Time series model")\]
In a [time series](https://en.wikipedia.org/wiki/Time_series "Time series") model, we require the [stochastic process](https://en.wikipedia.org/wiki/Stochastic_process "Stochastic process") {*xi*, *yi*} to be [stationary](https://en.wikipedia.org/wiki/Stationary_process "Stationary process") and [ergodic](https://en.wikipedia.org/wiki/Ergodic_process "Ergodic process"); if {*xi*, *yi*} is nonstationary, OLS results are often biased unless {*xi*, *yi*} is [co-integrating](https://en.wikipedia.org/wiki/Cointegration "Cointegration").[\[33\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-33)
We still require the regressors to be *strictly exogenous*: E\[*xiĪµi*\] = 0 for all *i* = 1, ..., *n*. If they are only [predetermined](https://en.wikipedia.org/wiki/Weak_exogeneity "Weak exogeneity"), OLS is biased in finite sample;
Finally, the assumptions on the variance take the form of requiring that {*xiĪµi*} is a [martingale difference sequence](https://en.wikipedia.org/wiki/Martingale_difference_sequence "Martingale difference sequence"), with a finite matrix of second moments *Q**xxĪµ*Ā² = E\[ *Īµi*2*xi xi*T \].
#### Constrained estimation
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=22 "Edit section: Constrained estimation")\]
Main article: [Ridge regression](https://en.wikipedia.org/wiki/Ridge_regression "Ridge regression")
Suppose it is known that the coefficients in the regression satisfy a system of linear equations
A
:
Q
T
Ī²
\=
c
,
{\\displaystyle A\\colon \\quad Q^{\\operatorname {T} }\\beta =c,\\,}
where *Q* is a *p*Ć*q* matrix of full rank, and *c* is a *q*Ć1 vector of known constants, where *q \< p*. In this case least squares estimation is equivalent to minimizing the sum of squared residuals of the model subject to the constraint *A*. The **constrained least squares (CLS)** estimator can be given by an explicit formula:[\[34\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-34)
Ī²
^
c
\=
Ī²
^
ā
(
X
T
X
)
ā
1
Q
(
Q
T
(
X
T
X
)
ā
1
Q
)
ā
1
(
Q
T
Ī²
^
ā
c
)
.
{\\displaystyle {\\hat {\\beta }}^{c}={\\hat {\\beta }}-(X^{\\operatorname {T} }X)^{-1}Q{\\Big (}Q^{\\operatorname {T} }(X^{\\operatorname {T} }X)^{-1}Q{\\Big )}^{-1}(Q^{\\operatorname {T} }{\\hat {\\beta }}-c).}
This expression for the constrained estimator is valid as long as the matrix *XTX* is invertible. It was assumed from the beginning of this article that this matrix is of full rank, and it was noted that when the rank condition fails, *Ī²* will not be identifiable. However it may happen that adding the restriction *A* makes *Ī²* identifiable, in which case one would like to find the formula for the estimator. The estimator is equal to [\[35\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Amemiya22-35)
Ī²
^
c
\=
R
(
R
T
X
T
X
R
)
ā
1
R
T
X
T
y
\+
(
I
p
ā
R
(
R
T
X
T
X
R
)
ā
1
R
T
X
T
X
)
Q
(
Q
T
Q
)
ā
1
c
,
{\\displaystyle {\\hat {\\beta }}^{c}=R(R^{\\operatorname {T} }X^{\\operatorname {T} }XR)^{-1}R^{\\operatorname {T} }X^{\\operatorname {T} }y+{\\Big (}I\_{p}-R(R^{\\operatorname {T} }X^{\\operatorname {T} }XR)^{-1}R^{\\operatorname {T} }X^{\\operatorname {T} }X{\\Big )}Q(Q^{\\operatorname {T} }Q)^{-1}c,}
where *R* is a *p*Ć(*p* ā *q*) matrix such that the matrix \[*Q R*\] is non-singular, and *RTQ* = 0. Such a matrix can always be found, although generally it is not unique. The second formula coincides with the first in case when *XTX* is invertible.[\[35\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Amemiya22-35)
## Example with real data
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=23 "Edit section: Example with real data")\]
See also: [Simple linear regression Ā§ Example](https://en.wikipedia.org/wiki/Simple_linear_regression#Example "Simple linear regression"), and [Linear least squares Ā§ Example](https://en.wikipedia.org/wiki/Linear_least_squares#Example "Linear least squares")
The following data set gives average heights and weights for American women aged 30ā39 (source: *The World Almanac and Book of Facts, 1975*).
| | | | | | | |
|---|---|---|---|---|---|---|
| Height (m) | 1\.47 | 1\.50 | 1\.52 | 1\.55 | 1\.57 | [](https://en.wikipedia.org/wiki/File:OLS_example_weight_vs_height_scatterplot.svg) [Scatterplot](https://en.wikipedia.org/wiki/Scatterplot "Scatterplot") of the data, the relationship is slightly curved but close to linear |
| Weight (kg) | 52\.21 | 53\.12 | 54\.48 | 55\.84 | 57\.20 | |
| Height (m) | 1\.60 | 1\.63 | 1\.65 | 1\.68 | 1\.70 | |
| Weight (kg) | 58\.57 | 59\.93 | 61\.29 | 63\.11 | 64\.47 | |
| Height (m) | 1\.73 | 1\.75 | 1\.78 | 1\.80 | 1\.83 | |
| Weight (kg) | 66\.28 | 68\.10 | 69\.92 | 72\.19 | 74\.46 | |
When only one dependent variable is being modeled, a [scatterplot](https://en.wikipedia.org/wiki/Scatterplot "Scatterplot") will suggest the form and strength of the relationship between the dependent variable and regressors. It might also reveal outliers, heteroscedasticity, and other aspects of the data that may complicate the interpretation of a fitted regression model. The scatterplot suggests that the relationship is strong and can be approximated as a quadratic function. OLS can handle non-linear relationships by introducing the regressor HEIGHT2. The regression model then becomes a multiple linear model:
w
i
\=
Ī²
1
\+
Ī²
2
h
i
\+
Ī²
3
h
i
2
\+
Īµ
i
.
{\\displaystyle w\_{i}=\\beta \_{1}+\\beta \_{2}h\_{i}+\\beta \_{3}h\_{i}^{2}+\\varepsilon \_{i}.}
[](https://en.wikipedia.org/wiki/File:OLS_example_weight_vs_height_fitted_line.svg)
Fitted regression
The output from most popular [statistical packages](https://en.wikipedia.org/wiki/List_of_statistical_packages "List of statistical packages") will look similar to this:
| | | | | |
|---|---|---|---|---|
| Method | Least squares | | | |
| Dependent variable | WEIGHT | | | |
| Observations | 15 | | | |
| Parameter | Value | [Std error](https://en.wikipedia.org/wiki/Standard_error "Standard error") | [t-statistic](https://en.wikipedia.org/wiki/T-statistic "T-statistic") | [p-value](https://en.wikipedia.org/wiki/P-value "P-value") |
| Ī² 1 {\\displaystyle \\beta \_{1}}  | | | | |
In this table:
- The *Value* column gives the least squares estimates of parameters *Ī²j*
- The *Std error* column shows [standard errors](https://en.wikipedia.org/wiki/Standard_error_\(statistics\) "Standard error (statistics)") of each coefficient estimate:
Ļ
^
j
\=
(
Ļ
^
2
\[
Q
x
x
ā
1
\]
j
j
)
1
2
{\\displaystyle {\\hat {\\sigma }}\_{j}=\\left({\\hat {\\sigma }}^{2}\\left\[Q\_{xx}^{-1}\\right\]\_{jj}\\right)^{\\frac {1}{2}}}
![{\\displaystyle {\\hat {\\sigma }}\_{j}=\\left({\\hat {\\sigma }}^{2}\\left\[Q\_{xx}^{-1}\\right\]\_{jj}\\right)^{\\frac {1}{2}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/5087e66171bf3ef9ad3ac75decdd715274919669)
- The *[t-statistic](https://en.wikipedia.org/wiki/T-statistic "T-statistic")* and *p-value* columns are testing whether any of the coefficients might be equal to zero. The *t*\-statistic is calculated simply as
t
\=
Ī²
^
j
/
Ļ
^
j
{\\displaystyle t={\\hat {\\beta }}\_{j}/{\\hat {\\sigma }}\_{j}}
. If the errors Īµ follow a normal distribution, *t* follows a Student-t distribution. Under weaker conditions, *t* is asymptotically normal. Large values of *t* indicate that the null hypothesis can be rejected and that the corresponding coefficient is not zero. The second column, [*p*\-value](https://en.wikipedia.org/wiki/P-value "P-value"), expresses the results of the hypothesis test as a [significance level](https://en.wikipedia.org/wiki/Statistical_significance "Statistical significance"). Conventionally, *p*\-values smaller than 0.05 are taken as evidence that the population coefficient is nonzero.
- *R-squared* is the [coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination "Coefficient of determination") indicating goodness-of-fit of the regression. This statistic will be equal to one if fit is perfect, and to zero when regressors *X* have no explanatory power whatsoever. This is a biased estimate of the population *R-squared*, and will never decrease if additional regressors are added, even if they are irrelevant.
- *Adjusted R-squared* is a slightly modified version of
R
2
{\\displaystyle R^{2}}
, designed to penalize for the excess number of regressors which do not add to the explanatory power of the regression. This statistic is always smaller than
R
2
{\\displaystyle R^{2}}
, can decrease as new regressors are added, and even be negative for poorly fitting models:
R
ĀÆ
2
\=
1
ā
n
ā
1
n
ā
p
(
1
ā
R
2
)
{\\displaystyle {\\overline {R}}^{2}=1-{\\frac {n-1}{n-p}}(1-R^{2})}
- *Log-likelihood* is calculated under the assumption that errors follow normal distribution. Even though the assumption is not very reasonable, this statistic may still find its use in conducting LR tests.
- *[DurbināWatson statistic](https://en.wikipedia.org/wiki/Durbin%E2%80%93Watson_statistic "DurbināWatson statistic")* tests whether there is any evidence of serial correlation between the residuals. As a rule of thumb, the value smaller than 2 will be an evidence of positive correlation.
- *[Akaike information criterion](https://en.wikipedia.org/wiki/Akaike_information_criterion "Akaike information criterion")* and *[Schwarz criterion](https://en.wikipedia.org/wiki/Schwarz_criterion "Schwarz criterion")* are both used for model selection. Generally when comparing two alternative models, smaller values of one of these criteria will indicate a better model.[\[36\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-36)
- *Standard error of regression* is an estimate of *Ļ*, standard error of the error term.
- *Total sum of squares*, *model sum of squared*, and *residual sum of squares* tell us how much of the initial variation in the sample were explained by the regression.
- *F-statistic* tries to test the hypothesis that all coefficients (except the intercept) are equal to zero. This statistic has *F*(*pā1*,*nāp*) distribution under the null hypothesis and normality assumption, and its *p-value* indicates probability that the hypothesis is indeed true. Note that when errors are not normal this statistic becomes invalid, and other tests such as [Wald test](https://en.wikipedia.org/wiki/Wald_test "Wald test") or [LR test](https://en.wikipedia.org/wiki/Likelihood_ratio_test "Likelihood ratio test") should be used.
[](https://en.wikipedia.org/wiki/File:OLS_example_weight_vs_height_residuals.svg)
Residuals plot
Ordinary least squares analysis often includes the use of diagnostic plots designed to detect departures of the data from the assumed form of the model. These are some of the common diagnostic plots:
- Residuals against the explanatory variables in the model. A non-linear relation between these variables suggests that the linearity of the conditional mean function may not hold. Different levels of variability in the residuals for different levels of the explanatory variables suggests possible heteroscedasticity.
- Residuals against explanatory variables not in the model. Any relation of the residuals to these variables would suggest considering these variables for inclusion in the model.
- Residuals against the fitted values,
y
^
{\\displaystyle {\\hat {y}}}
.
- Residuals against the preceding residual. This plot may identify serial correlations in the residuals.
An important consideration when carrying out statistical inference using regression models is how the data were sampled. In this example, the data are averages rather than measurements on individual women. The fit of the model is very good, but this does not imply that the weight of an individual woman can be predicted with high accuracy based only on her height.
### Sensitivity to rounding
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=24 "Edit section: Sensitivity to rounding")\]
Main article: [Errors-in-variables models](https://en.wikipedia.org/wiki/Errors-in-variables_models "Errors-in-variables models")
See also: [Quantization error model](https://en.wikipedia.org/wiki/Quantization_error_model "Quantization error model")
This example also demonstrates that coefficients determined by these calculations are sensitive to how the data is prepared. The heights were originally given rounded to the nearest inch and have been converted and rounded to the nearest centimetre. Since the conversion factor is one inch to 2.54 cm this is *not* an exact conversion. The original inches can be recovered by Round(x/0.0254) and then re-converted to metric without rounding. If this is done the results become:
| | Const | Height | Height2 |
|---|---|---|---|
| Converted to metric with rounding. | 128\.8128 | ā143.162 | 61\.96033 |
| Converted to metric without rounding. | 119\.0205 | ā131.5076 | 58\.5046 |
[](https://en.wikipedia.org/wiki/File:HeightWeightResiduals.jpg)
Residuals to a quadratic fit for correctly and incorrectly converted data.
Using either of these equations to predict the weight of a 5' 6" (1.6764 m) woman gives similar values: 62.94 kg with rounding vs. 62.98 kg without rounding. Thus a seemingly small variation in the data has a real effect on the coefficients but a small effect on the results of the equation.
While this may look innocuous in the middle of the data range it could become significant at the extremes or in the case where the fitted model is used to project outside the data range ([extrapolation](https://en.wikipedia.org/wiki/Extrapolation "Extrapolation")).
This highlights a common error: this example is an abuse of OLS which inherently requires that the errors in the independent variable (in this case height) are zero or at least negligible. The initial rounding to nearest inch plus any actual measurement errors constitute a finite and non-negligible error. As a result, the fitted parameters are not the best estimates they are presumed to be. Though not totally spurious the error in the estimation will depend upon relative size of the *x* and *y* errors.
## Another example with less real data
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=25 "Edit section: Another example with less real data")\]
### Problem statement
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=26 "Edit section: Problem statement")\]
We can use the least square mechanism to figure out the equation of a two body orbit in polar base co-ordinates. The equation typically used is r ( Īø ) \= p 1 ā e cos ā” ( Īø ) {\\displaystyle r(\\theta )={\\frac {p}{1-e\\cos(\\theta )}}}  where r ( Īø ) {\\displaystyle r(\\theta )}  is the radius of how far the object is from one of the bodies. In the equation the parameters p {\\displaystyle p}  and e {\\displaystyle e}  are used to determine the path of the orbit. We have measured the following data.
| Īø {\\displaystyle \\theta }  (in degrees) |
|---|
We need to find the least-squares approximation of e {\\displaystyle e}  and p {\\displaystyle p}  for the given data.
### Solution
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=27 "Edit section: Solution")\]
First we need to represent e and p in a linear form. So we are going to rewrite the equation r ( Īø ) {\\displaystyle r(\\theta )}  as 1 r ( Īø ) \= 1 p ā e p cos ā” ( Īø ) {\\displaystyle {\\frac {1}{r(\\theta )}}={\\frac {1}{p}}-{\\frac {e}{p}}\\cos(\\theta )} .
Furthermore, one could fit for [apsides](https://en.wikipedia.org/wiki/Apsides "Apsides") by expanding cos ā” ( Īø ) {\\displaystyle \\cos(\\theta )}  with an extra parameter as cos ā” ( Īø ā Īø 0 ) \= cos ā” ( Īø ) cos ā” ( Īø 0 ) \+ sin ā” ( Īø ) sin ā” ( Īø 0 ) {\\displaystyle \\cos(\\theta -\\theta \_{0})=\\cos(\\theta )\\cos(\\theta \_{0})+\\sin(\\theta )\\sin(\\theta \_{0})} , which is linear in both cos ā” ( Īø ) {\\displaystyle \\cos(\\theta )}  and in the extra basis function sin ā” ( Īø ) {\\displaystyle \\sin(\\theta )} .
We use the original two-parameter form to represent our observational data as:
A T A ( x y ) \= A T b , {\\displaystyle A^{T}A{\\binom {x}{y}}=A^{T}b,} 
where:
x \= 1 / p {\\displaystyle x=1/p\\,} ; y \= e / p {\\displaystyle y=e/p\\,} ; A {\\displaystyle A}  contains the coefficients of 1 / p {\\displaystyle 1/p}  in the first column, which are all 1, and the coefficients of e / p {\\displaystyle e/p}  in the second column, given by cos ā” ( Īø ) {\\displaystyle \\cos(\\theta )\\,} ; and b \= 1 / r ( Īø ) {\\displaystyle b=1/r(\\theta )} , such that:
A \= \[ 1 ā 0\.731354 1 ā 0\.707107 1 ā 0\.615661 1 0\.052336 1 0\.309017 1 0\.438371 \] , b \= \[ 0\.21220 0\.21958 0\.24741 0\.45071 0\.52883 0\.56820 \] . {\\displaystyle A={\\begin{bmatrix}1&-0.731354\\\\1&-0.707107\\\\1&-0.615661\\\\1&\\ 0.052336\\\\1&0.309017\\\\1&0.438371\\end{bmatrix}},\\quad b={\\begin{bmatrix}0.21220\\\\0.21958\\\\0.24741\\\\0.45071\\\\0.52883\\\\0.56820\\end{bmatrix}}.} 
On solving we get ( x y ) \= ( 0\.43478 0\.30435 ) {\\displaystyle {\\binom {x}{y}}={\\binom {0.43478}{0.30435}}\\,} ,
so p \= 1 x \= 2\.3000 {\\displaystyle p={\\frac {1}{x}}=2.3000}  and e \= p ā
y \= 0\.70001 {\\displaystyle e=p\\cdot y=0.70001} 
## See also
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=28 "Edit section: See also")\]
- [Bayesian least squares](https://en.wikipedia.org/wiki/Minimum_mean_square_error "Minimum mean square error")
- [FamaāMacBeth regression](https://en.wikipedia.org/wiki/Fama%E2%80%93MacBeth_regression "FamaāMacBeth regression")
- [Nonlinear least squares](https://en.wikipedia.org/wiki/Non-linear_least_squares "Non-linear least squares")
- [Numerical methods for linear least squares](https://en.wikipedia.org/wiki/Numerical_methods_for_linear_least_squares "Numerical methods for linear least squares")
- [Nonlinear system identification](https://en.wikipedia.org/wiki/Nonlinear_system_identification "Nonlinear system identification")
## References
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=29 "Edit section: References")\]
1. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-1)**
["The Origins of Ordinary Least Squares Assumptions"](https://mathvoices.ams.org/featurecolumn/2022/03/01/ordinary-least-squares/). *Feature Column*. 2022-03-01. Retrieved 2024-05-16.
2. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-2)**
["What is a complete list of the usual assumptions for linear regression?"](https://stats.stackexchange.com/q/16381). *Cross Validated*. Retrieved 2022-09-28.
3. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-3)**
[Goldberger, Arthur S.](https://en.wikipedia.org/wiki/Arthur_Goldberger "Arthur Goldberger") (1964). ["Classical Linear Regression"](https://books.google.com/books?id=KZq5AAAAIAAJ&pg=PA156). [*Econometric Theory*](https://archive.org/details/econometrictheor0000gold/page/158). New York: John Wiley & Sons. pp. [158](https://archive.org/details/econometrictheor0000gold/page/158). [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-471-31101-4](https://en.wikipedia.org/wiki/Special:BookSources/0-471-31101-4 "Special:BookSources/0-471-31101-4")
.
`{{cite book}}`: ISBN / Date incompatibility ([help](https://en.wikipedia.org/wiki/Help:CS1_errors#invalid_isbn_date "Help:CS1 errors"))
4. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-4)**
[Hayashi, Fumio](https://en.wikipedia.org/wiki/Fumio_Hayashi "Fumio Hayashi") (2000). *Econometrics*. Princeton University Press. p. 15. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[9780691010182](https://en.wikipedia.org/wiki/Special:BookSources/9780691010182 "Special:BookSources/9780691010182")
.
5. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-5)** [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 18).
6. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-6)**
Ghilani, Charles D.; Wolf, Paul R. (12 June 2006). [*Adjustment Computations: Spatial Data Analysis*](https://books.google.com/books?id=hZ4mAOXVowoC&pg=PA160). John Wiley & Sons. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[9780471697282](https://en.wikipedia.org/wiki/Special:BookSources/9780471697282 "Special:BookSources/9780471697282")
.
7. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-7)**
Hofmann-Wellenhof, Bernhard; Lichtenegger, Herbert; Wasle, Elmar (20 November 2007). [*GNSS ā Global Navigation Satellite Systems: GPS, GLONASS, Galileo, and more*](https://books.google.com/books?id=Np7y43HU_m8C&pg=PA263). Springer. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[9783211730171](https://en.wikipedia.org/wiki/Special:BookSources/9783211730171 "Special:BookSources/9783211730171")
.
8. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-8)**
Xu, Guochang (5 October 2007). [*GPS: Theory, Algorithms and Applications*](https://books.google.com/books?id=peYFZ69HqEsC&pg=PA134). Springer. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[9783540727156](https://en.wikipedia.org/wiki/Special:BookSources/9783540727156 "Special:BookSources/9783540727156")
.
9. ^ [***a***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hayashi_2000_loc=page_19_9-0) [***b***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hayashi_2000_loc=page_19_9-1) [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 19)
10. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-q011_10-0)**
Hoaglin, David C.; Welsch, Roy E. (1978). ["The Hat Matrix in Regression and ANOVA"](https://doi.org/10.1080%2F00031305.1978.10479237). *The American Statistician*. **32** (1): 17ā22\. [doi](https://en.wikipedia.org/wiki/Doi_\(identifier\) "Doi (identifier)"):[10\.1080/00031305.1978.10479237](https://doi.org/10.1080%2F00031305.1978.10479237). [hdl](https://en.wikipedia.org/wiki/Hdl_\(identifier\) "Hdl (identifier)"):[1721\.1/1920](https://hdl.handle.net/1721.1%2F1920). [ISSN](https://en.wikipedia.org/wiki/ISSN_\(identifier\) "ISSN (identifier)") [0003-1305](https://search.worldcat.org/issn/0003-1305).
11. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-11)** [Julian Faraway (2000), *Practical Regression and Anova using R*](https://cran.r-project.org/doc/contrib/Faraway-PRA.pdf)
12. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-12)**
Kenney, J.; Keeping, E. S. (1963). *Mathematics of Statistics*. van Nostrand. p. 187.
13. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-13)**
Zwillinger, Daniel (1995). [*Standard Mathematical Tables and Formulae*](https://en.wikipedia.org/wiki/CRC_Standard_Mathematical_Tables "CRC Standard Mathematical Tables"). Chapman\&Hall/CRC. p. 626. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-8493-2479-3](https://en.wikipedia.org/wiki/Special:BookSources/0-8493-2479-3 "Special:BookSources/0-8493-2479-3")
.
14. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-14)** [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 20)
15. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-15)**
Akbarzadeh, Vahab (7 May 2014). ["Line Estimation"](https://mlmadesimple.wordpress.com/2014/05/07/line-estimation/).
16. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-16)** [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 49)
17. ^ [***a***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hayashi_2000_loc=page_52_17-0) [***b***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hayashi_2000_loc=page_52_17-1) [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 52)
18. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hayashi_2000_loc=page_10_18-0)** [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 10)
19. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Tibshirani-1996_19-0)**
Tibshirani, Robert (1996). "Regression Shrinkage and Selection via the Lasso". *Journal of the Royal Statistical Society, Series B*. **58** (1): 267ā288\. [doi](https://en.wikipedia.org/wiki/Doi_\(identifier\) "Doi (identifier)"):[10\.1111/j.2517-6161.1996.tb02080.x](https://doi.org/10.1111%2Fj.2517-6161.1996.tb02080.x). [JSTOR](https://en.wikipedia.org/wiki/JSTOR_\(identifier\) "JSTOR (identifier)") [2346178](https://www.jstor.org/stable/2346178).
20. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Efron-2004_20-0)**
Efron, Bradley; Hastie, Trevor; Johnstone, Iain; Tibshirani, Robert (2004). "Least Angle Regression". *The Annals of Statistics*. **32** (2): 407ā451\. [arXiv](https://en.wikipedia.org/wiki/ArXiv_\(identifier\) "ArXiv (identifier)"):[math/0406456](https://arxiv.org/abs/math/0406456). [doi](https://en.wikipedia.org/wiki/Doi_\(identifier\) "Doi (identifier)"):[10\.1214/009053604000000067](https://doi.org/10.1214%2F009053604000000067). [JSTOR](https://en.wikipedia.org/wiki/JSTOR_\(identifier\) "JSTOR (identifier)") [3448465](https://www.jstor.org/stable/3448465). [S2CID](https://en.wikipedia.org/wiki/S2CID_\(identifier\) "S2CID (identifier)") [204004121](https://api.semanticscholar.org/CorpusID:204004121).
21. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hawkins-1973_21-0)**
Hawkins, Douglas M. (1973). "On the Investigation of Alternative Regressions by Principal Component Analysis". *Journal of the Royal Statistical Society, Series C*. **22** (3): 275ā286\. [doi](https://en.wikipedia.org/wiki/Doi_\(identifier\) "Doi (identifier)"):[10\.2307/2346776](https://doi.org/10.2307%2F2346776). [JSTOR](https://en.wikipedia.org/wiki/JSTOR_\(identifier\) "JSTOR (identifier)") [2346776](https://www.jstor.org/stable/2346776).
22. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Jolliffe-1982_22-0)**
Jolliffe, Ian T. (1982). "A Note on the Use of Principal Components in Regression". *Journal of the Royal Statistical Society, Series C*. **31** (3): 300ā303\. [doi](https://en.wikipedia.org/wiki/Doi_\(identifier\) "Doi (identifier)"):[10\.2307/2348005](https://doi.org/10.2307%2F2348005). [JSTOR](https://en.wikipedia.org/wiki/JSTOR_\(identifier\) "JSTOR (identifier)") [2348005](https://www.jstor.org/stable/2348005).
23. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-23)** [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), pages 27, 30)
24. ^ [***a***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-HayashiFSP_24-0) [***b***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-HayashiFSP_24-1) [***c***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-HayashiFSP_24-2) [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 27)
25. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-25)**
[Amemiya, Takeshi](https://en.wikipedia.org/wiki/Takeshi_Amemiya "Takeshi Amemiya") (1985). [*Advanced Econometrics*](https://archive.org/details/advancedeconomet00amem). Harvard University Press. p. [13](https://archive.org/details/advancedeconomet00amem/page/13). [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[9780674005600](https://en.wikipedia.org/wiki/Special:BookSources/9780674005600 "Special:BookSources/9780674005600")
.
26. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-26)** [Amemiya (1985](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFAmemiya1985), page 14)
27. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-27)**
[Rao, C. R.](https://en.wikipedia.org/wiki/C._R._Rao "C. R. Rao") (1973). *Linear Statistical Inference and its Applications* (Second ed.). New York: J. Wiley & Sons. p. 319. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-471-70823-2](https://en.wikipedia.org/wiki/Special:BookSources/0-471-70823-2 "Special:BookSources/0-471-70823-2")
.
28. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-28)** [Amemiya (1985](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFAmemiya1985), page 20)
29. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-29)** [Amemiya (1985](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFAmemiya1985), page 27)
30. ^ [***a***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-DvdMck33_30-0) [***b***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-DvdMck33_30-1)
Davidson, Russell; [MacKinnon, James G.](https://en.wikipedia.org/wiki/James_G._MacKinnon "James G. MacKinnon") (1993). *Estimation and Inference in Econometrics*. New York: Oxford University Press. p. 33. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-19-506011-3](https://en.wikipedia.org/wiki/Special:BookSources/0-19-506011-3 "Special:BookSources/0-19-506011-3")
.
31. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-31)** [Davidson & MacKinnon (1993](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFDavidsonMacKinnon1993), page 36)
32. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-32)** [Davidson & MacKinnon (1993](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFDavidsonMacKinnon1993), page 20)
33. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-33)**
["Memento on EViews Output"](https://scholar.harvard.edu/files/jbenchimol/files/memento-eviews.pdf) (PDF). Retrieved 28 December 2020.
34. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-34)** [Amemiya (1985](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFAmemiya1985), page 21)
35. ^ [***a***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Amemiya22_35-0) [***b***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Amemiya22_35-1) [Amemiya (1985](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFAmemiya1985), page 22)
36. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-36)**
Burnham, Kenneth P.; Anderson, David R. (2002). [*Model Selection and Multi-Model Inference*](https://archive.org/details/modelselectionmu0000burn) (2nd ed.). Springer. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-387-95364-7](https://en.wikipedia.org/wiki/Special:BookSources/0-387-95364-7 "Special:BookSources/0-387-95364-7")
.
## Further reading
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=30 "Edit section: Further reading")\]
- [Dougherty, Christopher](https://en.wikipedia.org/wiki/Christopher_Dougherty "Christopher Dougherty") (2002). *Introduction to Econometrics* (2nd ed.). New York: Oxford University Press. pp. 48ā113\. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-19-877643-8](https://en.wikipedia.org/wiki/Special:BookSources/0-19-877643-8 "Special:BookSources/0-19-877643-8")
.
- [Gujarati, Damodar N.](https://en.wikipedia.org/wiki/Damodar_N._Gujarati "Damodar N. Gujarati"); [Porter, Dawn C.](https://en.wikipedia.org/wiki/Dawn_C._Porter "Dawn C. Porter") (2009). *Basic Econometics* (Fifth ed.). Boston: McGraw-Hill Irwin. pp. 55ā96\. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[978-0-07-337577-9](https://en.wikipedia.org/wiki/Special:BookSources/978-0-07-337577-9 "Special:BookSources/978-0-07-337577-9")
.
- [Heij, Christiaan](https://en.wikipedia.org/wiki/Christiaan_Heij "Christiaan Heij"); Boer, Paul; [Franses, Philip H.](https://en.wikipedia.org/wiki/Philip_Hans_Franses "Philip Hans Franses"); [Kloek, Teun](https://en.wikipedia.org/wiki/Teun_Kloek "Teun Kloek"); [van Dijk, Herman K.](https://en.wikipedia.org/wiki/Herman_K._van_Dijk "Herman K. van Dijk") (2004). *Econometric Methods with Applications in Business and Economics* (1st ed.). Oxford: Oxford University Press. pp. 76ā115\. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[978-0-19-926801-6](https://en.wikipedia.org/wiki/Special:BookSources/978-0-19-926801-6 "Special:BookSources/978-0-19-926801-6")
.
- Hill, R. Carter; Griffiths, William E.; Lim, Guay C. (2008). *Principles of Econometrics* (3rd ed.). Hoboken, NJ: John Wiley & Sons. pp. 8ā47\. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[978-0-471-72360-8](https://en.wikipedia.org/wiki/Special:BookSources/978-0-471-72360-8 "Special:BookSources/978-0-471-72360-8")
.
- [Wooldridge, Jeffrey](https://en.wikipedia.org/wiki/Jeffrey_Wooldridge "Jeffrey Wooldridge") (2008). ["The Simple Regression Model"](https://books.google.com/books?id=64vt5TDBNLwC&pg=PA22). *Introductory Econometrics: A Modern Approach* (4th ed.). Mason, OH: Cengage Learning. pp. 22ā67\. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[978-0-324-58162-1](https://en.wikipedia.org/wiki/Special:BookSources/978-0-324-58162-1 "Special:BookSources/978-0-324-58162-1")
.
| [v](https://en.wikipedia.org/wiki/Template:Least_squares_and_regression_analysis "Template:Least squares and regression analysis") [t](https://en.wikipedia.org/wiki/Template_talk:Least_squares_and_regression_analysis "Template talk:Least squares and regression analysis") [e](https://en.wikipedia.org/wiki/Special:EditPage/Template:Least_squares_and_regression_analysis "Special:EditPage/Template:Least squares and regression analysis")[Least squares](https://en.wikipedia.org/wiki/Least_squares "Least squares") and [regression analysis](https://en.wikipedia.org/wiki/Regression_analysis "Regression analysis") | |
|---|---|
| [Computational statistics](https://en.wikipedia.org/wiki/Computational_statistics "Computational statistics") | [Least squares](https://en.wikipedia.org/wiki/Least_squares "Least squares") [Linear least squares](https://en.wikipedia.org/wiki/Linear_least_squares_\(mathematics\) "Linear least squares (mathematics)") [Non-linear least squares](https://en.wikipedia.org/wiki/Non-linear_least_squares "Non-linear least squares") [Iteratively reweighted least squares](https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares "Iteratively reweighted least squares") |
| [Correlation and dependence](https://en.wikipedia.org/wiki/Correlation_and_dependence "Correlation and dependence") | [Pearson product-moment correlation](https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient "Pearson product-moment correlation coefficient") [Rank correlation](https://en.wikipedia.org/wiki/Rank_correlation "Rank correlation") ([Spearman's rho](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient "Spearman's rank correlation coefficient") [Kendall's tau](https://en.wikipedia.org/wiki/Kendall_tau_rank_correlation_coefficient "Kendall tau rank correlation coefficient")) [Partial correlation](https://en.wikipedia.org/wiki/Partial_correlation "Partial correlation") [Confounding variable](https://en.wikipedia.org/wiki/Confounding "Confounding") |
| [Regression analysis](https://en.wikipedia.org/wiki/Regression_analysis "Regression analysis") | [Ordinary least squares]() [Partial least squares](https://en.wikipedia.org/wiki/Partial_least_squares_regression "Partial least squares regression") [Total least squares](https://en.wikipedia.org/wiki/Total_least_squares "Total least squares") [Ridge regression](https://en.wikipedia.org/wiki/Tikhonov_regularization "Tikhonov regularization") |
| Regression as a [statistical model](https://en.wikipedia.org/wiki/Statistical_model "Statistical model") | |
| | |
| [Linear regression](https://en.wikipedia.org/wiki/Linear_regression "Linear regression") | [Simple linear regression](https://en.wikipedia.org/wiki/Simple_linear_regression "Simple linear regression") [Ordinary least squares]() [Generalized least squares](https://en.wikipedia.org/wiki/Generalized_least_squares "Generalized least squares") [Weighted least squares](https://en.wikipedia.org/wiki/Weighted_least_squares "Weighted least squares") [General linear model](https://en.wikipedia.org/wiki/General_linear_model "General linear model") |
| Predictor structure | [Polynomial regression](https://en.wikipedia.org/wiki/Polynomial_regression "Polynomial regression") [Growth curve (statistics)](https://en.wikipedia.org/wiki/Growth_curve_\(statistics\) "Growth curve (statistics)") [Segmented regression](https://en.wikipedia.org/wiki/Segmented_regression "Segmented regression") [Local regression](https://en.wikipedia.org/wiki/Local_regression "Local regression") |
| Non-standard | [Nonlinear regression](https://en.wikipedia.org/wiki/Nonlinear_regression "Nonlinear regression") [Nonparametric](https://en.wikipedia.org/wiki/Nonparametric_regression "Nonparametric regression") [Semiparametric](https://en.wikipedia.org/wiki/Semiparametric_regression "Semiparametric regression") [Robust](https://en.wikipedia.org/wiki/Robust_regression "Robust regression") [Quantile](https://en.wikipedia.org/wiki/Quantile_regression "Quantile regression") [Isotonic](https://en.wikipedia.org/wiki/Isotonic_regression "Isotonic regression") |
| Non-normal errors | [Generalized linear model](https://en.wikipedia.org/wiki/Generalized_linear_model "Generalized linear model") [Binomial](https://en.wikipedia.org/wiki/Binomial_regression "Binomial regression") [Poisson](https://en.wikipedia.org/wiki/Poisson_regression "Poisson regression") [Logistic](https://en.wikipedia.org/wiki/Logistic_regression "Logistic regression") |
| [Decomposition of variance](https://en.wikipedia.org/wiki/Partition_of_sums_of_squares "Partition of sums of squares") | [Analysis of variance](https://en.wikipedia.org/wiki/Analysis_of_variance "Analysis of variance") [Analysis of covariance](https://en.wikipedia.org/wiki/Analysis_of_covariance "Analysis of covariance") [Multivariate AOV](https://en.wikipedia.org/wiki/Multivariate_analysis_of_variance "Multivariate analysis of variance") |
| Model exploration | [Stepwise regression](https://en.wikipedia.org/wiki/Stepwise_regression "Stepwise regression") [Model selection](https://en.wikipedia.org/wiki/Model_selection "Model selection") [Mallows's *Cp*](https://en.wikipedia.org/wiki/Mallows%27s_Cp "Mallows's Cp") [AIC](https://en.wikipedia.org/wiki/Akaike_information_criterion "Akaike information criterion") [BIC](https://en.wikipedia.org/wiki/Bayesian_information_criterion "Bayesian information criterion") [Model specification](https://en.wikipedia.org/wiki/Model_specification "Model specification") [Regression validation](https://en.wikipedia.org/wiki/Regression_validation "Regression validation") |
| Background | [Mean and predicted response](https://en.wikipedia.org/wiki/Mean_and_predicted_response "Mean and predicted response") [GaussāMarkov theorem](https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem "GaussāMarkov theorem") [Errors and residuals](https://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics "Errors and residuals in statistics") [Goodness of fit](https://en.wikipedia.org/wiki/Goodness_of_fit "Goodness of fit") [Studentized residual](https://en.wikipedia.org/wiki/Studentized_residual "Studentized residual") [Minimum mean-square error](https://en.wikipedia.org/wiki/Minimum_mean-square_error "Minimum mean-square error") [FrischāWaughāLovell theorem](https://en.wikipedia.org/wiki/Frisch%E2%80%93Waugh%E2%80%93Lovell_theorem "FrischāWaughāLovell theorem") |
| [Design of experiments](https://en.wikipedia.org/wiki/Design_of_experiments "Design of experiments") | [Response surface methodology](https://en.wikipedia.org/wiki/Response_surface_methodology "Response surface methodology") [Optimal design](https://en.wikipedia.org/wiki/Optimal_design "Optimal design") [Bayesian design](https://en.wikipedia.org/wiki/Bayesian_experimental_design "Bayesian experimental design") |
| [Numerical](https://en.wikipedia.org/wiki/Numerical_analysis "Numerical analysis") [approximation](https://en.wikipedia.org/wiki/Approximation_theory "Approximation theory") | [Numerical analysis](https://en.wikipedia.org/wiki/Numerical_analysis "Numerical analysis") [Approximation theory](https://en.wikipedia.org/wiki/Approximation_theory "Approximation theory") [Numerical integration](https://en.wikipedia.org/wiki/Numerical_integration "Numerical integration") [Gaussian quadrature](https://en.wikipedia.org/wiki/Gaussian_quadrature "Gaussian quadrature") [Orthogonal polynomials](https://en.wikipedia.org/wiki/Orthogonal_polynomials "Orthogonal polynomials") [Chebyshev polynomials](https://en.wikipedia.org/wiki/Chebyshev_polynomials "Chebyshev polynomials") [Chebyshev nodes](https://en.wikipedia.org/wiki/Chebyshev_nodes "Chebyshev nodes") |
| Applications | [Curve fitting](https://en.wikipedia.org/wiki/Curve_fitting "Curve fitting") [Calibration curve](https://en.wikipedia.org/wiki/Calibration_curve "Calibration curve") [Numerical smoothing and differentiation](https://en.wikipedia.org/wiki/Numerical_smoothing_and_differentiation "Numerical smoothing and differentiation") [System identification](https://en.wikipedia.org/wiki/System_identification "System identification") [Moving least squares](https://en.wikipedia.org/wiki/Moving_least_squares "Moving least squares") |
| [Regression analysis category](https://en.wikipedia.org/wiki/Category:Regression_analysis "Category:Regression analysis") [Statistics category](https://en.wikipedia.org/wiki/Category:Statistics "Category:Statistics") [](https://en.wikipedia.org/wiki/File:Nuvola_apps_edu_mathematics_blue-p.svg) [Mathematics portal](https://en.wikipedia.org/wiki/Portal:Mathematics "Portal:Mathematics") [Statistics outline](https://en.wikipedia.org/wiki/Outline_of_statistics "Outline of statistics") [Statistics topics](https://en.wikipedia.org/wiki/List_of_statistics_articles "List of statistics articles") | |
Retrieved from "<https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&oldid=1345164414>"
[Categories](https://en.wikipedia.org/wiki/Help:Category "Help:Category"):
- [Parametric statistics](https://en.wikipedia.org/wiki/Category:Parametric_statistics "Category:Parametric statistics")
- [Least squares](https://en.wikipedia.org/wiki/Category:Least_squares "Category:Least squares")
Hidden categories:
- [CS1 errors: ISBN date](https://en.wikipedia.org/wiki/Category:CS1_errors:_ISBN_date "Category:CS1 errors: ISBN date")
- [Articles with short description](https://en.wikipedia.org/wiki/Category:Articles_with_short_description "Category:Articles with short description")
- [Short description matches Wikidata](https://en.wikipedia.org/wiki/Category:Short_description_matches_Wikidata "Category:Short description matches Wikidata")
- [Wikipedia articles needing clarification from December 2023](https://en.wikipedia.org/wiki/Category:Wikipedia_articles_needing_clarification_from_December_2023 "Category:Wikipedia articles needing clarification from December 2023")
- [All articles with unsourced statements](https://en.wikipedia.org/wiki/Category:All_articles_with_unsourced_statements "Category:All articles with unsourced statements")
- [Articles with unsourced statements from February 2010](https://en.wikipedia.org/wiki/Category:Articles_with_unsourced_statements_from_February_2010 "Category:Articles with unsourced statements from February 2010")
- [Articles to be expanded from February 2017](https://en.wikipedia.org/wiki/Category:Articles_to_be_expanded_from_February_2017 "Category:Articles to be expanded from February 2017")
- [All articles to be expanded](https://en.wikipedia.org/wiki/Category:All_articles_to_be_expanded "Category:All articles to be expanded")
- This page was last edited on 24 March 2026, at 17:11 (UTC).
- Text is available under the [Creative Commons Attribution-ShareAlike 4.0 License](https://en.wikipedia.org/wiki/Wikipedia:Text_of_the_Creative_Commons_Attribution-ShareAlike_4.0_International_License "Wikipedia:Text of the Creative Commons Attribution-ShareAlike 4.0 International License"); additional terms may apply. By using this site, you agree to the [Terms of Use](https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Terms_of_Use "foundation:Special:MyLanguage/Policy:Terms of Use") and [Privacy Policy](https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Privacy_policy "foundation:Special:MyLanguage/Policy:Privacy policy"). WikipediaĀ® is a registered trademark of the [Wikimedia Foundation, Inc.](https://wikimediafoundation.org/), a non-profit organization.
- [Privacy policy](https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Privacy_policy)
- [About Wikipedia](https://en.wikipedia.org/wiki/Wikipedia:About)
- [Disclaimers](https://en.wikipedia.org/wiki/Wikipedia:General_disclaimer)
- [Contact Wikipedia](https://en.wikipedia.org/wiki/Wikipedia:Contact_us)
- [Legal & safety contacts](https://foundation.wikimedia.org/wiki/Special:MyLanguage/Legal:Wikimedia_Foundation_Legal_and_Safety_Contact_Information)
- [Code of Conduct](https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Universal_Code_of_Conduct)
- [Developers](https://developer.wikimedia.org/)
- [Statistics](https://stats.wikimedia.org/#/en.wikipedia.org)
- [Cookie statement](https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Cookie_statement)
- [Mobile view](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&mobileaction=toggle_view_mobile)
- [](https://www.wikimedia.org/)
- [](https://www.mediawiki.org/)
Search
Toggle the table of contents
Ordinary least squares
12 languages
[Add topic](https://en.wikipedia.org/wiki/Ordinary_least_squares) | |||||||||
| Readable Markdown | [](https://en.wikipedia.org/wiki/File:Okuns_law_quarterly_differences.svg)
[Okun's law](https://en.wikipedia.org/wiki/Okun%27s_law "Okun's law") in [macroeconomics](https://en.wikipedia.org/wiki/Macroeconomics "Macroeconomics") states that in an economy the [GDP](https://en.wikipedia.org/wiki/GDP "GDP") growth should depend linearly on the changes in the unemployment rate. Here the ordinary least squares method is used to construct the regression line describing this law.
In [statistics](https://en.wikipedia.org/wiki/Statistics "Statistics"), **ordinary least squares** (**OLS**) is a type of [linear least squares](https://en.wikipedia.org/wiki/Linear_least_squares "Linear least squares") method for choosing the unknown [parameters](https://en.wikipedia.org/wiki/Statistical_parameter "Statistical parameter") in a [linear regression](https://en.wikipedia.org/wiki/Linear_regression "Linear regression") model (with fixed level-one\[*[clarification needed](https://en.wikipedia.org/wiki/Wikipedia:Please_clarify "Wikipedia:Please clarify")*\] effects of a [linear function](https://en.wikipedia.org/wiki/Linear_function "Linear function") of a set of [explanatory variables](https://en.wikipedia.org/wiki/Explanatory_variable "Explanatory variable")) by the principle of [least squares](https://en.wikipedia.org/wiki/Least_squares "Least squares"): minimizing the sum of the squares of the differences between the observed [dependent variable](https://en.wikipedia.org/wiki/Dependent_variable "Dependent variable") (values of the variable being observed) in the input [dataset](https://en.wikipedia.org/wiki/Dataset "Dataset") and the output of the (linear) function of the [independent variable](https://en.wikipedia.org/wiki/Independent_variable "Independent variable"). Some sources consider OLS to be linear regression.[\[1\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-1)
Geometrically, this is seen as the sum of the squared distances, parallel to the axis of the dependent variable, between each data point in the set and the corresponding point on the regression surfaceāthe smaller the differences, the better the model fits the data. The resulting [estimator](https://en.wikipedia.org/wiki/Statistical_estimation "Statistical estimation") can be expressed by a simple formula, especially in the case of a [simple linear regression](https://en.wikipedia.org/wiki/Simple_linear_regression "Simple linear regression"), in which there is a single [regressor](https://en.wikipedia.org/wiki/Regressor "Regressor") on the right side of the regression equation.
The OLS estimator is [consistent](https://en.wikipedia.org/wiki/Consistent_estimator "Consistent estimator") for the level-one fixed effects when the regressors are [exogenous](https://en.wikipedia.org/wiki/Exogenous "Exogenous") and forms perfect [colinearity](https://en.wikipedia.org/wiki/Collinearity "Collinearity") (rank condition), consistent for the variance estimate of the residuals when regressors have finite fourth moments[\[2\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-2) andāby the [GaussāMarkov theorem](https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem "GaussāMarkov theorem")ā[optimal in the class of linear unbiased estimators](https://en.wikipedia.org/wiki/Best_linear_unbiased_estimator "Best linear unbiased estimator") when the [errors](https://en.wikipedia.org/wiki/Statistical_error "Statistical error") are [homoscedastic](https://en.wikipedia.org/wiki/Homoscedastic "Homoscedastic") and [serially uncorrelated](https://en.wikipedia.org/wiki/Autocorrelation "Autocorrelation"). Under these conditions, the method of OLS provides [minimum-variance mean-unbiased](https://en.wikipedia.org/wiki/UMVU "UMVU") estimation when the errors have finite [variances](https://en.wikipedia.org/wiki/Variance "Variance"). Under the additional assumption that the errors are [normally distributed](https://en.wikipedia.org/wiki/Normal_distribution "Normal distribution") with zero mean, OLS is the [maximum likelihood estimator](https://en.wikipedia.org/wiki/Maximum_likelihood_estimator "Maximum likelihood estimator") that outperforms any non-linear unbiased estimator.
Suppose the data consists of  [observations](https://en.wikipedia.org/wiki/Statistical_unit "Statistical unit") . Each observation  includes a scalar response  and a column vector  of  parameters (regressors), i.e., ![{\\displaystyle \\mathbf {x} \_{i}=\\left\[x\_{i1},x\_{i2},\\dots ,x\_{ip}\\right\]^{\\operatorname {T} }}](https://wikimedia.org/api/rest_v1/media/math/render/svg/3278872b5bdb53e6af3474d92e9926c0238e8935). In a [linear regression model](https://en.wikipedia.org/wiki/Linear_regression_model "Linear regression model"), the response variable, , is a linear function of the regressors:
or in [vector](https://en.wikipedia.org/wiki/Row_and_column_vectors "Row and column vectors") form,
where , as introduced previously, is a column vector of the \-th observation of all the explanatory variables;  is a  vector of unknown parameters; and the scalar  represents unobserved random variables ([errors](https://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics "Errors and residuals in statistics")) of the \-th observation.  accounts for the influences upon the responses  from sources other than the explanatory variables . This model can also be written in matrix notation as
where  and  are  vectors of the response variables and the errors of the  observations, and  is an  matrix of regressors, also sometimes called the [design matrix](https://en.wikipedia.org/wiki/Design_matrix "Design matrix"), whose row  is  and contains the \-th observations on all the explanatory variables.
Typically, a constant term is included in the set of regressors , say, by taking  for all . The coefficient  corresponding to this regressor is called the *intercept*. Without the intercept, the fitted line is forced to cross the origin when .
Regressors do not have to be independent for estimation to be consistent e.g. they may be non-linearly dependent. Short of perfect multicollinearity, parameter estimates may still be consistent; however, as multicollinearity rises the standard error around such estimates increases and reduces the precision of such estimates. When there is perfect multicollinearity, it is no longer possible to obtain unique estimates for the coefficients to the related regressors; estimation for these parameters cannot converge (thus, it cannot be consistent).
As a concrete example where regressors are non-linearly dependent yet estimation may still be consistent, we might suspect the response depends linearly both on a value and its square; in which case we would include one regressor whose value is just the square of another regressor. In that case, the model would be *quadratic* in the second regressor, but none-the-less is still considered a *linear* model because the model *is* still linear in the parameters ().
### Matrix/vector formulation
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=2 "Edit section: Matrix/vector formulation")\]
Consider an [overdetermined system](https://en.wikipedia.org/wiki/Overdetermined_system "Overdetermined system")
of  [linear equations](https://en.wikipedia.org/wiki/Linear_equation "Linear equation") in  unknown [coefficients](https://en.wikipedia.org/wiki/Coefficients "Coefficients"), , with . This can be written in [matrix](https://en.wikipedia.org/wiki/Matrix_\(mathematics\) "Matrix (mathematics)") form as
where
(Note: for a linear model as above, not all elements in  contains information on the data points. The first column is populated with ones, . Only the other columns contain actual data. So here  is equal to the number of regressors plus one).
Such a system usually has no exact solution, so the goal is instead to find the coefficients  which fit the equations "best", in the sense of solving the [quadratic](https://en.wikipedia.org/wiki/Quadratic_form_\(statistics\) "Quadratic form (statistics)") [minimization](https://en.wikipedia.org/wiki/Mathematical_optimization "Mathematical optimization") problem
where the objective function  is given by
A justification for choosing this criterion is given in [Properties](https://en.wikipedia.org/wiki/Ordinary_least_squares#Properties) below. This minimization problem has a unique solution, provided that the  columns of the matrix  are [linearly independent](https://en.wikipedia.org/wiki/Linearly_independent "Linearly independent"), given by solving the so-called *normal equations*:
The matrix  is known as the *normal matrix* or [Gram matrix](https://en.wikipedia.org/wiki/Gram_matrix "Gram matrix") and the matrix  is known as the [moment matrix](https://en.wikipedia.org/wiki/Moment_matrix "Moment matrix") of regressand by regressors.[\[3\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-3) Finally,  is the coefficient vector of the least-squares [hyperplane](https://en.wikipedia.org/wiki/Hyperplane "Hyperplane"), expressed as
or
Suppose *b* is a "candidate" value for the parameter vector *Ī²*. The quantity *yi* ā *xi*T*b*, called the *[residual](https://en.wikipedia.org/wiki/Errors_and_residuals_in_statistics "Errors and residuals in statistics")* for the *i*\-th observation, measures the vertical distance between the data point (*xi*, *yi*) and the hyperplane *y* = *x*T*b*, and thus assesses the degree of fit between the actual data and the model. The *[sum of squared residuals](https://en.wikipedia.org/wiki/Sum_of_squared_residuals "Sum of squared residuals")* (*SSR*) (also called the *error sum of squares* (*ESS*) or *residual sum of squares* (*RSS*))[\[4\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-4) is a measure of the overall model fit:
where *T* denotes the matrix [transpose](https://en.wikipedia.org/wiki/Transpose "Transpose"), and the rows of *X*, denoting the values of all the independent variables associated with a particular value of the dependent variable, are *Xi = xi*T. The value of *b* which minimizes this sum is called the **OLS estimator for *Ī²***. The function *S*(*b*) is quadratic in *b* with positive-definite [Hessian](https://en.wikipedia.org/wiki/Hessian_matrix "Hessian matrix"), and therefore this function possesses a unique global minimum at , which can be given by the explicit formula[\[5\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-5)[\[proof\]](https://en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares#Least_squares_estimator_for_.CE.B2 "Proofs involving ordinary least squares")
The product *N* = *X*T *X* is a [Gram matrix](https://en.wikipedia.org/wiki/Gram_matrix "Gram matrix"), and its inverse, *Q* = *N*ā1, is the *cofactor matrix* of *Ī²*,[\[6\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-6)[\[7\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-7)[\[8\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-8) closely related to its [covariance matrix](https://en.wikipedia.org/wiki/Ordinary_least_squares#Covariance_matrix), *C**Ī²*. The matrix (*X*T *X*)ā1 *X*T = *Q* *X*T is called the [MooreāPenrose pseudoinverse](https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_pseudoinverse "MooreāPenrose pseudoinverse") matrix of *X*. This formulation highlights the point that estimation can be carried out if, and only if, there is no perfect [multicollinearity](https://en.wikipedia.org/wiki/Multicollinearity "Multicollinearity") between the explanatory variables (which would cause the Gram matrix to have no inverse).
After we have estimated *Ī²*, the *[fitted values](https://en.wikipedia.org/wiki/Fitted_value "Fitted value")* (or *predicted values*) from the regression will be
where *P* = *X*(*X*T*X*)ā1*X*T is the *[projection matrix](https://en.wikipedia.org/wiki/Projection_matrix "Projection matrix")* onto the space *V* spanned by the columns of *X*. This matrix *P* is also sometimes called the *[hat matrix](https://en.wikipedia.org/wiki/Hat_matrix "Hat matrix")* because it "puts a hat" onto the variable *y*. Another matrix, closely related to *P* is the *annihilator* matrix *M* = *In* ā *P*; this is a projection matrix onto the space orthogonal to *V*. Both matrices *P* and *M* are [symmetric](https://en.wikipedia.org/wiki/Symmetric_matrix "Symmetric matrix") and [idempotent](https://en.wikipedia.org/wiki/Idempotent_matrix "Idempotent matrix") (meaning that *P*2 = *P* and *M*2 = *M*), and relate to the data matrix *X* via identities *PX* = *X* and *MX* = 0.[\[9\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hayashi_2000_loc=page_19-9) Matrix *M* creates the *residuals* from the regression:
The variances of the predicted values  are found in the main diagonal of the [variance-covariance matrix](https://en.wikipedia.org/wiki/Variance-covariance_matrix "Variance-covariance matrix") of predicted values:
where *P* is the projection matrix and *s*2 is the sample variance.[\[10\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-q011-10) The full matrix is very large; its diagonal elements can be calculated individually as:
where *X*i is the *i*\-th row of matrix *X*.
Using these residuals we can estimate the sample variance *s*2 using the *[reduced chi-squared](https://en.wikipedia.org/wiki/Reduced_chi-squared "Reduced chi-squared")* statistic:
The denominator, *n*ā*p*, is the [statistical degrees of freedom](https://en.wikipedia.org/wiki/Degrees_of_freedom_\(statistics\) "Degrees of freedom (statistics)"). The first quantity, *s*2, is the OLS estimate for *Ļ*2, whereas the second, , is the MLE estimate for *Ļ*2. The two estimators are quite similar in large samples; the first estimator is always [unbiased](https://en.wikipedia.org/wiki/Estimator_bias "Estimator bias"), while the second estimator is biased but has a smaller [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error "Mean squared error"). In practice *s*2 is used more often, since it is more convenient for the hypothesis testing. The square root of *s*2 is called the *[regression standard error](https://en.wikipedia.org/wiki/Regression_standard_error "Regression standard error")*,[\[11\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-11) *standard error of the regression*,[\[12\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-12)[\[13\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-13) or *standard error of the equation*.[\[9\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hayashi_2000_loc=page_19-9)
It is common to assess the goodness-of-fit of the OLS regression by comparing how much the initial variation in the sample can be reduced by regressing onto *X*. The *[coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination "Coefficient of determination")* *R*2 is defined as a ratio of "explained" variance to the "total" variance of the dependent variable *y*, in the cases where the regression sum of squares equals the sum of squares of residuals:[\[14\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-14)
where TSS is the *[total sum of squares](https://en.wikipedia.org/wiki/Total_sum_of_squares "Total sum of squares")* for the dependent variable, , and  is an *n*Ć*n* matrix of ones. ( is a [centering matrix](https://en.wikipedia.org/wiki/Centering_matrix "Centering matrix") which is equivalent to regression on a constant; it simply subtracts the mean from a variable.) In order for *R*2 to be meaningful, the matrix *X* of data on regressors must contain a column vector of ones to represent the constant whose coefficient is the regression intercept. In that case, *R*2 will always be a number between 0 and 1, with values close to 1 indicating a good degree of fit.
### Simple linear regression model
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=6 "Edit section: Simple linear regression model")\]
If the data matrix *X* contains only two variables, a constant and a scalar regressor *xi*, then this is called the "simple regression model". This case is often considered in the beginner statistics classes, as it provides much simpler formulas even suitable for manual calculation. The parameters are commonly denoted as (*Ī±*, *Ī²*):
The least squares estimates in this case are given by simple formulas
![{\\displaystyle {\\begin{aligned}{\\widehat {\\beta }}&={\\frac {\\sum \_{i=1}^{n}{(x\_{i}-{\\bar {x}})(y\_{i}-{\\bar {y}})}}{\\sum \_{i=1}^{n}{(x\_{i}-{\\bar {x}})^{2}}}}\\\\\[2pt\]{\\widehat {\\alpha }}&={\\bar {y}}-{\\widehat {\\beta }}\\,{\\bar {x}}\\ ,\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/932c6407f7ceba533fef69961fe504fc3b565e1e)
## Alternative derivations
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=7 "Edit section: Alternative derivations")\]
In the previous section the least squares estimator  was obtained as a value that minimizes the sum of squared residuals of the model. However it is also possible to derive the same estimator from other approaches. In all cases the formula for OLS estimator remains the same: ^*Ī²* = (*X*T*X*)ā1*X*T*y*; the only difference is in how we interpret this result.
[](https://en.wikipedia.org/wiki/File:OLS_geometric_interpretation.svg)
OLS estimation can be viewed as a projection onto the linear space spanned by the regressors. (Here each of  and  refers to a column of the data matrix.)
[](https://en.wikipedia.org/wiki/File:Geometric_interpretation_of_least_squares_\(three_observations\).png)
Least squares as projection of y onto col(X) for three observations; Å· = XĪ² gives fitted values and y ā Å· is the residual.
For mathematicians, OLS is an approximate solution to an overdetermined system of linear equations *XĪ²* ā *y*, where *Ī²* is the unknown. Assuming the system cannot be solved exactly (the number of equations *n* is much larger than the number of unknowns *p*), we are looking for a solution that could provide the smallest discrepancy between the right- and left- hand sides. In other words, we are looking for the solution that satisfies
where āĀ·ā is the standard [*L*2 norm](https://en.wikipedia.org/wiki/Norm_\(mathematics\)#Euclidean_norm "Norm (mathematics)") in the *n*\-dimensional [Euclidean space](https://en.wikipedia.org/wiki/Euclidean_space "Euclidean space") **R***n*. The predicted quantity *XĪ²* is just a certain linear combination of the vectors of regressors. Thus, the residual vector *y* ā *XĪ²* will have the smallest length when *y* is [projected orthogonally](https://en.wikipedia.org/wiki/Projection_\(linear_algebra\) "Projection (linear algebra)") onto the [linear subspace](https://en.wikipedia.org/wiki/Linear_subspace "Linear subspace") [spanned](https://en.wikipedia.org/wiki/Linear_span "Linear span") by the columns of *X*. The OLS estimator  in this case can be interpreted as the coefficients of [vector decomposition](https://en.wikipedia.org/wiki/Vector_decomposition "Vector decomposition") of ^*y* = *Py* along the basis of *X*.
In other words, the gradient equations at the minimum can be written as:
A geometrical interpretation of these equations is that the vector of residuals,  is orthogonal to the [column space](https://en.wikipedia.org/wiki/Column_space "Column space") of *X*, since the [dot product](https://en.wikipedia.org/wiki/Dot_product "Dot product")  is equal to zero for *any* conformal vector, **v**. This means that  is the shortest of all possible vectors , that is, the variance of the residuals is the minimum possible. This is illustrated at the right.
Introducing  and a matrix *K* with the assumption that a matrix ![{\\displaystyle \[\\mathbf {X} \\ \\mathbf {K} \]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/2ac770308f79814997ffbdfd971621c67b76aef6) is non-singular and *K*T *X* = 0 (cf. [Orthogonal projections](https://en.wikipedia.org/wiki/Linear_projection#Orthogonal_projections "Linear projection")), the residual vector should satisfy the following equation:
The equation and solution of linear least squares are thus described as follows:
Another way of looking at it is to consider the regression line to be a weighted average of the lines passing through the combination of any two points in the dataset.[\[15\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-15) Although this way of calculation is more computationally expensive, it provides a better intuition on OLS.
The OLS estimator is identical to the [maximum likelihood estimator](https://en.wikipedia.org/wiki/Maximum_likelihood_estimator "Maximum likelihood estimator") (MLE) under the normality assumption for the error terms.[\[16\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-16)[\[proof\]](https://en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares#Maximum_likelihood_approach "Proofs involving ordinary least squares") This normality assumption has historical importance, as it provided the basis for the early work in linear regression analysis by [Yule](https://en.wikipedia.org/wiki/Udny_Yule "Udny Yule") and [Pearson](https://en.wikipedia.org/wiki/Karl_Pearson "Karl Pearson").\[*[citation needed](https://en.wikipedia.org/wiki/Wikipedia:Citation_needed "Wikipedia:Citation needed")*\] From the properties of MLE, we can infer that the OLS estimator is asymptotically efficient (in the sense of attaining the [CramĆ©rāRao bound](https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_bound "CramĆ©rāRao bound") for variance) if the normality assumption is satisfied.[\[17\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hayashi_2000_loc=page_52-17)
### Generalized method of moments
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=10 "Edit section: Generalized method of moments")\]
In [iid](https://en.wikipedia.org/wiki/Iid "Iid") case the OLS estimator can also be viewed as a [GMM](https://en.wikipedia.org/wiki/Generalized_method_of_moments "Generalized method of moments") estimator arising from the moment conditions
![{\\displaystyle \\mathrm {E} {\\big \[}\\,x\_{i}\\left(y\_{i}-x\_{i}^{\\operatorname {T} }\\beta \\right)\\,{\\big \]}=0.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/1d7894c141dad7e6dae3aed8bb708aada174daf2)
These moment conditions state that the regressors should be uncorrelated with the errors. Since *xi* is a *p*\-vector, the number of moment conditions is equal to the dimension of the parameter vector *Ī²*, and thus the system is exactly identified. This is the so-called classical GMM case, when the estimator does not depend on the choice of the weighting matrix.
Note that the original strict exogeneity assumption E\[*Īµi* \| *xi*\] = 0 implies a far richer set of moment conditions than stated above. In particular, this assumption implies that for any vector-function Ę, the moment condition E\[*Ę*(*xi*)Ā·*Īµi*\] = 0 will hold. However it can be shown using the [GaussāMarkov theorem](https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem "GaussāMarkov theorem") that the optimal choice of function Ę is to take *Ę*(*x*) = *x*, which results in the moment equation posted above.
There are several different frameworks in which the [linear regression model](https://en.wikipedia.org/wiki/Linear_regression_model "Linear regression model") can be cast in order to make the OLS technique applicable. Each of these settings produces the same formulas and same results. The only difference is the interpretation and the assumptions which have to be imposed in order for the method to give meaningful results. The choice of the applicable framework depends mostly on the nature of data in hand, and on the inference task which has to be performed.
One of the lines of difference in interpretation is whether to treat the regressors as random variables, or as predefined constants. In the first case (**random design**) the regressors *xi* are random and sampled together with the *yi*'s from some [population](https://en.wikipedia.org/wiki/Statistical_population "Statistical population"), as in an [observational study](https://en.wikipedia.org/wiki/Observational_study "Observational study"). This approach allows for more natural study of the [asymptotic properties](https://en.wikipedia.org/wiki/Asymptotic_theory_\(statistics\) "Asymptotic theory (statistics)") of the estimators. In the other interpretation (**fixed design**), the regressors *X* are treated as known constants set by a [design](https://en.wikipedia.org/wiki/Design_of_experiments "Design of experiments"), and *y* is sampled conditionally on the values of *X* as in an [experiment](https://en.wikipedia.org/wiki/Experiment "Experiment"). For practical purposes, this distinction is often unimportant, since estimation and inference is carried out while conditioning on *X*. All results stated in this article are within the random design framework.
The classical model focuses on the "finite sample" estimation and inference, meaning that the number of observations *n* is fixed. This contrasts with the other approaches, which study the [asymptotic behavior](https://en.wikipedia.org/wiki/Asymptotic_theory_\(statistics\) "Asymptotic theory (statistics)") of OLS, and in which the behavior at a large number of samples is studied. To prove finite sample unbiasedness of the OLS estimator, we require the following assumptions.
[](https://en.wikipedia.org/wiki/File:Polyreg_scheffe.svg)
Example of a cubic polynomial regression, which is a type of linear regression. Although *polynomial regression* fits a curve model to the data, as a [statistical estimation](https://en.wikipedia.org/wiki/Estimation_theory "Estimation theory") problem it is linear, in the sense that the conditional expectation function ![{\\displaystyle \\mathbb {E} \[y\|x\]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/391e556522e79e05efc7dd6a5248cc0e5b0c4651) is linear in the unknown [parameters](https://en.wikipedia.org/wiki/Parameter "Parameter") that are estimated from the [data](https://en.wikipedia.org/wiki/Data "Data"). For this reason, polynomial regression is considered to be a special case of [multiple linear regression](https://en.wikipedia.org/wiki/Multiple_linear_regression "Multiple linear regression").
- **Exogeneity**. The regressors do not [covary](https://en.wikipedia.org/wiki/Covariance "Covariance") with the error term: ![{\\displaystyle \\mathbb {E} \[\\varepsilon \_{i}x\_{i}\]=0.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/865dc6e3427f19d12afd3bed41c45bb7661ef289) This requires, for example, that there are no [omitted variables](https://en.wikipedia.org/wiki/Omitted_variable_bias "Omitted variable bias") that covary with observed variables and affect the response variable. An alternative (but stronger) statement that is often required when explaining linear regression in [mathematical statistics](https://en.wikipedia.org/wiki/Mathematical_statistics "Mathematical statistics") is that the predictor variables *x* can be treated as fixed values, rather than [random variables](https://en.wikipedia.org/wiki/Random_variable "Random variable"). This stronger form means, for example, that the predictor variables are assumed to be error-free, that is, not contaminated with measurement error. Although this assumption is not realistic in many settings, dropping it leads to more complex [errors-in-variables models](https://en.wikipedia.org/wiki/Errors-in-variables_models "Errors-in-variables models"), [instrumental variable models](https://en.wikipedia.org/wiki/Instrumental_variable "Instrumental variable") and the like.
- **Linearity**, or **correct specification**. This means that the mean of the response variable is a [linear combination](https://en.wikipedia.org/wiki/Linear_combination "Linear combination") of the parameters (regression coefficients) and the predictor variables. Note that this assumption is much less restrictive than it may at first seem. Because the predictor variables are treated as fixed values (see above), linearity is really only a restriction on the parameters. The predictor variables themselves can be arbitrarily transformed, and in fact multiple copies of the same underlying predictor variable can be added, each one transformed differently. This technique is used, for example, in [polynomial regression](https://en.wikipedia.org/wiki/Polynomial_regression "Polynomial regression"), which uses linear regression to fit the response variable as an arbitrary [polynomial](https://en.wikipedia.org/wiki/Polynomial "Polynomial") function (up to a given degree) of a predictor variable. With this much flexibility, models such as polynomial regression often have "too much power", in that they tend to [overfit](https://en.wikipedia.org/wiki/Overfit "Overfit") the data. As a result, some kind of [regularization](https://en.wikipedia.org/wiki/Regularization_\(mathematics\) "Regularization (mathematics)") must typically be used to prevent unreasonable solutions coming out of the estimation process. Common examples are [ridge regression](https://en.wikipedia.org/wiki/Ridge_regression "Ridge regression") and [lasso regression](https://en.wikipedia.org/wiki/Lasso_regression "Lasso regression"). [Bayesian linear regression](https://en.wikipedia.org/wiki/Bayesian_linear_regression "Bayesian linear regression") can also be used, which by its nature is more or less immune to the problem of overfitting. (In fact, [ridge regression](https://en.wikipedia.org/wiki/Ridge_regression "Ridge regression") and [lasso regression](https://en.wikipedia.org/wiki/Lasso_regression "Lasso regression") can both be viewed as special cases of Bayesian linear regression, with particular types of [prior distributions](https://en.wikipedia.org/wiki/Prior_distribution "Prior distribution") placed on the regression coefficients.)
- [](https://en.wikipedia.org/wiki/File:Heteroscedasticity_in_Linear_Regression.png)
Visualization of heteroscedasticity in a scatter plot against 100 random fitted values using Matlab
**Constant variance** or **[homoscedasticity](https://en.wikipedia.org/wiki/Homoscedasticity "Homoscedasticity")**. This means that the variance of the errors does not depend on the values of the predictor variables: ![{\\displaystyle \\mathbb {E} \[\\varepsilon \_{i}^{2}\|x\_{i}\]=\\sigma ^{2}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/93072a3d09cd2b89e6a544dfd4c7cbb017200cbd) Thus the variability of the responses for given fixed values of the predictors is the same regardless of how large or small the responses are. This is often not the case, as a variable whose mean is large will typically have a greater variance than one whose mean is small. For example, a person whose income is predicted to be \$100,000 may easily have an actual income of \$80,000 or \$120,000āi.e., a [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation "Standard deviation") of around \$20,000āwhile another person with a predicted income of \$10,000 is unlikely to have the same \$20,000 standard deviation, since that would imply their actual income could vary anywhere between ā\$10,000 and \$30,000. (In fact, as this shows, in many casesāoften the same cases where the assumption of normally distributed errors failsāthe variance or standard deviation should be predicted to be proportional to the mean, rather than constant.) The absence of homoscedasticity is called [heteroscedasticity](https://en.wikipedia.org/wiki/Heteroscedasticity "Heteroscedasticity"). In order to check this assumption, a plot of residuals versus predicted values (or the values of each individual predictor) can be examined for a "fanning effect" (i.e., increasing or decreasing vertical spread as one moves left to right on the plot). A plot of the absolute or squared residuals versus the predicted values (or each predictor) can also be examined for a trend or curvature. Formal tests can also be used; see [Heteroscedasticity](https://en.wikipedia.org/wiki/Heteroscedasticity "Heteroscedasticity"). The presence of heteroscedasticity will result in an overall "average" estimate of variance being used instead of one that takes into account the true variance structure. This leads to less precise (but in the case of [ordinary least squares](), not biased) parameter estimates and biased standard errors, resulting in misleading tests and interval estimates. The [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error "Mean squared error") for the model will also be wrong. Various estimation techniques including [weighted least squares](https://en.wikipedia.org/wiki/Weighted_least_squares "Weighted least squares") and the use of [heteroscedasticity-consistent standard errors](https://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors "Heteroscedasticity-consistent standard errors") can handle heteroscedasticity in a quite general way. [Bayesian linear regression](https://en.wikipedia.org/wiki/Bayesian_linear_regression "Bayesian linear regression") techniques can also be used when the variance is assumed to be a function of the mean. It is also possible in some cases to fix the problem by applying a transformation to the response variable (e.g., fitting the [logarithm](https://en.wikipedia.org/wiki/Logarithm "Logarithm") of the response variable using a linear regression model, which implies that the response variable itself has a [log-normal distribution](https://en.wikipedia.org/wiki/Log-normal_distribution "Log-normal distribution") rather than a [normal distribution](https://en.wikipedia.org/wiki/Normal_distribution "Normal distribution")).
[](https://en.wikipedia.org/wiki/File:Independence_of_Errors_Assumption_for_Linear_Regressions.png)
To check for violations of the assumptions of linearity, constant variance, and independence of errors within a linear regression model, the residuals are typically plotted against the predicted values (or each of the individual predictors). An apparently random scatter of points about the horizontal midline at 0 is ideal, but cannot rule out certain kinds of violations such as [autocorrelation](https://en.wikipedia.org/wiki/Autocorrelation "Autocorrelation") in the errors or their correlation with one or more covariates.
- **Uncorrelatedness of errors**. This assumes that the errors of the response variables are uncorrelated with each other: ![{\\displaystyle \\mathbb {E} \[\\varepsilon \_{i}\\varepsilon \_{j}\|x\_{i},x\_{j}\]=0.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d2049f9a6b8ce702724b8271a5912158bf3f0fc3) Some methods such as [generalized least squares](https://en.wikipedia.org/wiki/Generalized_least_squares "Generalized least squares") are capable of handling correlated errors, although they typically require significantly more data unless some sort of [regularization](https://en.wikipedia.org/wiki/Regularization_\(mathematics\) "Regularization (mathematics)") is used to bias the model towards assuming uncorrelated errors. [Bayesian linear regression](https://en.wikipedia.org/wiki/Bayesian_linear_regression "Bayesian linear regression") is a general way of handling this issue. Full [statistical independence](https://en.wikipedia.org/wiki/Statistical_independence "Statistical independence") is a stronger condition than mere lack of correlation and is often not needed, although it implies mean-independence.
- **Lack of perfect multicollinearity** in the predictors. For standard [least squares](https://en.wikipedia.org/wiki/Least_squares "Least squares") estimation methods, the design matrix *X* must have full [column rank](https://en.wikipedia.org/wiki/Column_rank "Column rank") *p*: [\[18\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hayashi_2000_loc=page_10-18)![{\\displaystyle \\Pr \\!{\\big \[}\\,\\operatorname {rank} (X)=p\\,{\\big \]}=1.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/6a11be3b89ce51c6441155fddbe512a991132fbf) If this assumption is violated, perfect [multicollinearity](https://en.wikipedia.org/wiki/Multicollinearity "Multicollinearity") exists in the predictor variables, meaning a linear relationship exists between two or more predictor variables. Multicollinearity can be caused by accidentally duplicating a variable in the data, using a linear transformation of a variable along with the original (e.g., the same temperature measurements expressed in Fahrenheit and Celsius), or including a linear combination of multiple variables in the model, such as their mean. It can also happen if there is too little data available compared to the number of parameters to be estimated (e.g., fewer data points than regression coefficients). Near violations of this assumption, where predictors are highly but not perfectly correlated, can reduce the precision of parameter estimates (see [Variance inflation factor](https://en.wikipedia.org/wiki/Variance_inflation_factor "Variance inflation factor")). In the case of perfect multicollinearity, the parameter vector ***Ī²*** will be [non-identifiable](https://en.wikipedia.org/wiki/Non-identifiable "Non-identifiable")āit has no unique solution. In such a case, only some of the parameters can be identified (i.e., their values can only be estimated within some linear subspace of the full parameter space **R***p*). See [partial least squares regression](https://en.wikipedia.org/wiki/Partial_least_squares_regression "Partial least squares regression"). Methods for fitting linear models with multicollinearity have been developed,[\[19\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Tibshirani-1996-19)[\[20\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Efron-2004-20)[\[21\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hawkins-1973-21)[\[22\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Jolliffe-1982-22) some of which require additional assumptions such as "effect sparsity"āthat a large fraction of the effects are exactly zero. Note that the more computationally expensive iterated algorithms for parameter estimation, such as those used in [generalized linear models](https://en.wikipedia.org/wiki/Generalized_linear_model "Generalized linear model"), do not suffer from this problem.
Violations of these assumptions can result in biased estimations of ***Ī²***, biased standard errors, untrustworthy confidence intervals and significance tests. Beyond these assumptions, several other statistical properties of the data strongly influence the performance of different estimation methods:
- The statistical relationship between the error terms and the regressors plays an important role in determining whether an estimation procedure has desirable sampling properties such as being unbiased and consistent.
- The arrangement, or [probability distribution](https://en.wikipedia.org/wiki/Probability_distribution "Probability distribution") of the predictor variables **x** has a major influence on the precision of estimates of ***Ī²***. [Sampling](https://en.wikipedia.org/wiki/Sampling_\(statistics\) "Sampling (statistics)") and [design of experiments](https://en.wikipedia.org/wiki/Design_of_experiments "Design of experiments") are highly developed subfields of statistics that provide guidance for collecting data in such a way to achieve a precise estimate of ***Ī²***.
### Finite sample properties
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=13 "Edit section: Finite sample properties")\]
First of all, under the *strict exogeneity* assumption the OLS estimators  and *s*2 are [unbiased](https://en.wikipedia.org/wiki/Bias_of_an_estimator "Bias of an estimator"), meaning that their expected values coincide with the true values of the parameters:[\[23\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-23)[\[proof\]](https://en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares#Unbiasedness_of_.CE.B2.CC.82 "Proofs involving ordinary least squares")
![{\\displaystyle \\operatorname {E} \[\\,{\\hat {\\beta }}\\mid X\\,\]=\\beta ,\\quad \\operatorname {E} \[\\,s^{2}\\mid X\\,\]=\\sigma ^{2}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/67bc2fd0f90c46da207712893fdcea01e729026c)
If the strict exogeneity does not hold (as is the case with many [time series](https://en.wikipedia.org/wiki/Time_series "Time series") models, where exogeneity is assumed only with respect to the past shocks but not the future ones), then these estimators will be biased in finite samples.
The *[variance-covariance matrix](https://en.wikipedia.org/wiki/Variance-covariance_matrix "Variance-covariance matrix")* (or simply *covariance matrix*) of  is equal to[\[24\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-HayashiFSP-24)
![{\\displaystyle \\operatorname {Var} \[\\,{\\hat {\\beta }}\\mid X\\,\]=\\sigma ^{2}\\left(X^{\\operatorname {T} }X\\right)^{-1}=\\sigma ^{2}Q.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/08f6cb596d94073731ee47f4a2571dbbfc1d214a)
In particular, the standard error of each coefficient  is equal to square root of the *j*\-th diagonal element of this matrix. The estimate of this standard error is obtained by replacing the unknown quantity *Ļ*2 with its estimate *s*2. Thus,
It can also be easily shown that the estimator  is uncorrelated with the residuals from the model:[\[24\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-HayashiFSP-24)
![{\\displaystyle \\operatorname {Cov} \[\\,{\\hat {\\beta }},{\\hat {\\varepsilon }}\\mid X\\,\]=0.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/664c1a5e37957a1aa2ae381b9bcb07350c2c816c)
The *[GaussāMarkov theorem](https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem "GaussāMarkov theorem")* states that under the *spherical errors* assumption (that is, the errors should be [uncorrelated](https://en.wikipedia.org/wiki/Uncorrelated "Uncorrelated") and [homoscedastic](https://en.wikipedia.org/wiki/Homoscedastic "Homoscedastic")) the estimator  is efficient in the class of linear unbiased estimators. This is called the *best linear unbiased estimator* (BLUE). Efficiency should be understood as if we were to find some other estimator  which would be linear in *y* and unbiased, then [\[24\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-HayashiFSP-24)
![{\\displaystyle \\operatorname {Var} \[\\,{\\tilde {\\beta }}\\mid X\\,\]-\\operatorname {Var} \[\\,{\\hat {\\beta }}\\mid X\\,\]\\geq 0}](https://wikimedia.org/api/rest_v1/media/math/render/svg/53796c9205889cc4d675b9749a58eb97fcd998f1)
in the sense that this is a [nonnegative-definite matrix](https://en.wikipedia.org/wiki/Nonnegative-definite_matrix "Nonnegative-definite matrix"). This theorem establishes optimality only in the class of linear unbiased estimators, which is quite restrictive. Depending on the distribution of the error terms *Īµ*, other, non-linear estimators may provide better results than OLS.
The properties listed so far are all valid regardless of the underlying distribution of the error terms. However, if you are willing to assume that the *normality assumption* holds (that is, that *Īµ* ~ *N*(0, *Ļ*2*In*)), then additional properties of the OLS estimators can be stated.
The estimator  is normally distributed, with mean and variance as given before:[\[25\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-25)
This estimator reaches the [CramĆ©rāRao bound](https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_bound "CramĆ©rāRao bound") for the model, and thus is optimal in the class of all unbiased estimators.[\[17\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Hayashi_2000_loc=page_52-17) Note that unlike the [GaussāMarkov theorem](https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem "GaussāMarkov theorem"), this result establishes optimality among both linear and non-linear estimators, but only in the case of normally distributed error terms.
The estimator *s*2 will be proportional to the [chi-squared distribution](https://en.wikipedia.org/wiki/Chi-squared_distribution "Chi-squared distribution"):[\[26\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-26)
The variance of this estimator is equal to 2*Ļ*4/(*n* ā *p*), which does not attain the [CramĆ©rāRao bound](https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_bound "CramĆ©rāRao bound") of 2*Ļ*4/*n*. However it was shown that there are no unbiased estimators of *Ļ*2 with variance smaller than that of the estimator *s*2.[\[27\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-27) If we are willing to allow biased estimators, and consider the class of estimators that are proportional to the sum of squared residuals (SSR) of the model, then the best (in the sense of the [mean squared error](https://en.wikipedia.org/wiki/Mean_squared_error "Mean squared error")) estimator in this class will be ~*Ļ*2 = SSR */* (*n* ā *p* + 2), which even beats the CramĆ©rāRao bound in case when there is only one regressor (*p* = 1).[\[28\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-28)
Moreover, the estimators  and *s*2 are [independent](https://en.wikipedia.org/wiki/Independent_random_variables "Independent random variables"),[\[29\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-29) the fact which comes in useful when constructing the t- and F-tests for the regression.
#### Influential observations
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=15 "Edit section: Influential observations")\]
As was mentioned before, the estimator  is linear in *y*, meaning that it represents a linear combination of the dependent variables *yi*. The weights in this linear combination are functions of the regressors *X*, and generally are unequal. The observations with high weights are called **influential** because they have a more pronounced effect on the value of the estimator.
To analyze which observations are influential we remove a specific *j*\-th observation and consider how much the estimated quantities are going to change (similarly to the [jackknife method](https://en.wikipedia.org/wiki/Jackknife_method "Jackknife method")). It can be shown that the change in the OLS estimator for *Ī²* will be equal to [\[30\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-DvdMck33-30)
where *hj* = *xj*T (*X*T*X*)ā1*xj* is the *j*\-th diagonal element of the hat matrix *P*, and *xj* is the vector of regressors corresponding to the *j*\-th observation. Similarly, the change in the predicted value for *j*\-th observation resulting from omitting that observation from the dataset will be equal to [\[30\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-DvdMck33-30)
From the properties of the hat matrix, 0 ā¤ *hj* ā¤ 1, and they sum up to *p*, so that on average *hj* ā *p/n*. These quantities *hj* are called the **leverages**, and observations with high *hj* are called **leverage points**.[\[31\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-31) Usually the observations with high leverage ought to be scrutinized more carefully, in case they are erroneous, or outliers, or in some other way atypical of the rest of the dataset.
#### Partitioned regression
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=16 "Edit section: Partitioned regression")\]
Sometimes the variables and corresponding parameters in the regression can be logically split into two groups, so that the regression takes form
where *X*1 and *X*2 have dimensions *n*Ć*p*1, *n*Ć*p*2, and *Ī²*1, *Ī²*2 are *p*1Ć1 and *p*2Ć1 vectors, with *p*1 + *p*2 = *p*.
The **[FrischāWaughāLovell theorem](https://en.wikipedia.org/wiki/Frisch%E2%80%93Waugh%E2%80%93Lovell_theorem "FrischāWaughāLovell theorem")** states that in this regression the residuals  and the OLS estimate  will be numerically identical to the residuals and the OLS estimate for *Ī²*2 in the following regression:[\[32\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-32)
where *M*1 is the [annihilator matrix](https://en.wikipedia.org/wiki/Annihilator_matrix "Annihilator matrix") for regressors *X*1.
The theorem can be used to establish a number of theoretical results. For example, having a regression with a constant and another regressor is equivalent to subtracting the means from the dependent variable and the regressor and then running the regression for the de-meaned variables but without the constant term.
### Large sample properties
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=17 "Edit section: Large sample properties")\]
The least squares estimators are [point estimates](https://en.wikipedia.org/wiki/Point_estimate "Point estimate") of the linear regression model parameters *Ī²*. However, generally we also want to know how close those estimates might be to the true values of parameters. In other words, we want to construct the [interval estimates](https://en.wikipedia.org/wiki/Interval_estimate "Interval estimate").
Since we have not made any assumption about the distribution of error term *Īµi*, it is impossible to infer the distribution of the estimators  and . Nevertheless, we can apply the [central limit theorem](https://en.wikipedia.org/wiki/Central_limit_theorem "Central limit theorem") to derive their *asymptotic* properties as sample size *n* goes to infinity. While the sample size is necessarily finite, it is customary to assume that *n* is "large enough" so that the true distribution of the OLS estimator is close to its asymptotic limit.
We can show that under the model assumptions, the least squares estimator for *Ī²* is [consistent](https://en.wikipedia.org/wiki/Consistent_estimator "Consistent estimator") (that is  [converges in probability](https://en.wikipedia.org/wiki/Convergence_of_random_variables#Convergence_in_probability "Convergence of random variables") to *Ī²*) and asymptotically normal:[\[proof\]](https://en.wikipedia.org/wiki/Proofs_involving_ordinary_least_squares#Consistency_and_asymptotic_normality_of_.CE.B2.CC.82 "Proofs involving ordinary least squares")
where 
Using this asymptotic distribution, approximate two-sided confidence intervals for the *j*\-th component of the vector  can be constructed as
![{\\displaystyle \\beta \_{j}\\in {\\bigg \[}\\ {\\hat {\\beta }}\_{j}\\pm q\_{1-{\\frac {\\alpha }{2}}}^{{\\mathcal {N}}(0,1)}\\!{\\sqrt {{\\hat {\\sigma }}^{2}\\left\[Q\_{xx}^{-1}\\right\]\_{jj}}}\\ {\\bigg \]}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cf79688aac9f662ff39253fbfb0d234246d370e5) at the 1 ā *Ī±* confidence level,
where *q* denotes the [quantile function](https://en.wikipedia.org/wiki/Quantile_function "Quantile function") of standard normal distribution, and \[Ā·\]*jj* is the *j*\-th diagonal element of a matrix.
Similarly, the least squares estimator for *Ļ*2 is also consistent and asymptotically normal (provided that the fourth moment of *Īµi* exists) with limiting distribution
![{\\displaystyle ({\\hat {\\sigma }}^{2}-\\sigma ^{2})\\ \\xrightarrow {d} \\ {\\mathcal {N}}\\left(0,\\;\\operatorname {E} \\left\[\\varepsilon \_{i}^{4}\\right\]-\\sigma ^{4}\\right).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/7c909dea2a4f0bf40e253680b953d1bfbb66298f)
These asymptotic distributions can be used for prediction, testing hypotheses, constructing other estimators, etc.. As an example consider the problem of prediction. Suppose  is some point within the domain of distribution of the regressors, and one wants to know what the response variable would have been at that point. The [mean response](https://en.wikipedia.org/wiki/Mean_response "Mean response") is the quantity , whereas the [predicted response](https://en.wikipedia.org/wiki/Predicted_response "Predicted response") is . Clearly the predicted response is a random variable, its distribution can be derived from that of :
which allows construct confidence intervals for mean response  to be constructed:
![{\\displaystyle y\_{0}\\in \\left\[\\ x\_{0}^{\\mathrm {T} }{\\hat {\\beta }}\\pm q\_{1-{\\frac {\\alpha }{2}}}^{{\\mathcal {N}}(0,1)}\\!{\\sqrt {{\\hat {\\sigma }}^{2}x\_{0}^{\\mathrm {T} }Q\_{xx}^{-1}x\_{0}}}\\ \\right\]}](https://wikimedia.org/api/rest_v1/media/math/render/svg/cf86d7a311c97d35fb6e039c3cd74bc9f3e752bf) at the 1 ā *Ī±* confidence level.
Two hypothesis tests are particularly widely used. First, one wants to know if the estimated regression equation is any better than simply predicting that all values of the response variable equal its sample mean (if not, it is said to have no explanatory power). The [null hypothesis](https://en.wikipedia.org/wiki/Null_hypothesis "Null hypothesis") of no explanatory value of the estimated regression is tested using an [F-test](https://en.wikipedia.org/wiki/F-test "F-test"). If the calculated F-value is found to be large enough to exceed its critical value for the pre-chosen level of significance, the null hypothesis is rejected and the [alternative hypothesis](https://en.wikipedia.org/wiki/Alternative_hypothesis "Alternative hypothesis"), that the regression has explanatory power, is accepted. Otherwise, the null hypothesis of no explanatory power is accepted.
Second, for each explanatory variable of interest, one wants to know whether its estimated coefficient differs significantly from zeroāthat is, whether this particular explanatory variable in fact has explanatory power in predicting the response variable. Here the null hypothesis is that the true coefficient is zero. This hypothesis is tested by computing the coefficient's [t-statistic](https://en.wikipedia.org/wiki/T-statistic "T-statistic"), as the ratio of the coefficient estimate to its [standard error](https://en.wikipedia.org/wiki/Standard_error "Standard error"). If the t-statistic is larger than a predetermined value, the null hypothesis is rejected and the variable is found to have explanatory power, with its coefficient significantly different from zero. Otherwise, the null hypothesis of a zero value of the true coefficient is accepted.
In addition, the [Chow test](https://en.wikipedia.org/wiki/Chow_test "Chow test") is used to test whether two subsamples both have the same underlying true coefficient values. The sum of squared residuals of regressions on each of the subsets and on the combined data set are compared by computing an F-statistic; if this exceeds a critical value, the null hypothesis of no difference between the two subsets is rejected; otherwise, it is accepted.
### Violations of assumptions
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=20 "Edit section: Violations of assumptions")\]
In a [time series](https://en.wikipedia.org/wiki/Time_series "Time series") model, we require the [stochastic process](https://en.wikipedia.org/wiki/Stochastic_process "Stochastic process") {*xi*, *yi*} to be [stationary](https://en.wikipedia.org/wiki/Stationary_process "Stationary process") and [ergodic](https://en.wikipedia.org/wiki/Ergodic_process "Ergodic process"); if {*xi*, *yi*} is nonstationary, OLS results are often biased unless {*xi*, *yi*} is [co-integrating](https://en.wikipedia.org/wiki/Cointegration "Cointegration").[\[33\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-33)
We still require the regressors to be *strictly exogenous*: E\[*xiĪµi*\] = 0 for all *i* = 1, ..., *n*. If they are only [predetermined](https://en.wikipedia.org/wiki/Weak_exogeneity "Weak exogeneity"), OLS is biased in finite sample;
Finally, the assumptions on the variance take the form of requiring that {*xiĪµi*} is a [martingale difference sequence](https://en.wikipedia.org/wiki/Martingale_difference_sequence "Martingale difference sequence"), with a finite matrix of second moments *Q**xxĪµ*Ā² = E\[ *Īµi*2*xi xi*T \].
#### Constrained estimation
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=22 "Edit section: Constrained estimation")\]
Suppose it is known that the coefficients in the regression satisfy a system of linear equations
where *Q* is a *p*Ć*q* matrix of full rank, and *c* is a *q*Ć1 vector of known constants, where *q \< p*. In this case least squares estimation is equivalent to minimizing the sum of squared residuals of the model subject to the constraint *A*. The **constrained least squares (CLS)** estimator can be given by an explicit formula:[\[34\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-34)
This expression for the constrained estimator is valid as long as the matrix *XTX* is invertible. It was assumed from the beginning of this article that this matrix is of full rank, and it was noted that when the rank condition fails, *Ī²* will not be identifiable. However it may happen that adding the restriction *A* makes *Ī²* identifiable, in which case one would like to find the formula for the estimator. The estimator is equal to [\[35\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Amemiya22-35)
where *R* is a *p*Ć(*p* ā *q*) matrix such that the matrix \[*Q R*\] is non-singular, and *RTQ* = 0. Such a matrix can always be found, although generally it is not unique. The second formula coincides with the first in case when *XTX* is invertible.[\[35\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-Amemiya22-35)
## Example with real data
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=23 "Edit section: Example with real data")\]
The following data set gives average heights and weights for American women aged 30ā39 (source: *The World Almanac and Book of Facts, 1975*).
| | | | | | | |
|---|---|---|---|---|---|---|
| Height (m) | 1\.47 | 1\.50 | 1\.52 | 1\.55 | 1\.57 | [](https://en.wikipedia.org/wiki/File:OLS_example_weight_vs_height_scatterplot.svg) [Scatterplot](https://en.wikipedia.org/wiki/Scatterplot "Scatterplot") of the data, the relationship is slightly curved but close to linear |
| Weight (kg) | 52\.21 | 53\.12 | 54\.48 | 55\.84 | 57\.20 | |
| Height (m) | 1\.60 | 1\.63 | 1\.65 | 1\.68 | 1\.70 | |
| Weight (kg) | 58\.57 | 59\.93 | 61\.29 | 63\.11 | 64\.47 | |
| Height (m) | 1\.73 | 1\.75 | 1\.78 | 1\.80 | 1\.83 | |
| Weight (kg) | 66\.28 | 68\.10 | 69\.92 | 72\.19 | 74\.46 | |
When only one dependent variable is being modeled, a [scatterplot](https://en.wikipedia.org/wiki/Scatterplot "Scatterplot") will suggest the form and strength of the relationship between the dependent variable and regressors. It might also reveal outliers, heteroscedasticity, and other aspects of the data that may complicate the interpretation of a fitted regression model. The scatterplot suggests that the relationship is strong and can be approximated as a quadratic function. OLS can handle non-linear relationships by introducing the regressor HEIGHT2. The regression model then becomes a multiple linear model:
[](https://en.wikipedia.org/wiki/File:OLS_example_weight_vs_height_fitted_line.svg)
Fitted regression
The output from most popular [statistical packages](https://en.wikipedia.org/wiki/List_of_statistical_packages "List of statistical packages") will look similar to this:
| | | | | |
|---|---|---|---|---|
| Method | Least squares | | | |
| Dependent variable | WEIGHT | | | |
| Observations | 15 | | | |
| Parameter | Value | [Std error](https://en.wikipedia.org/wiki/Standard_error "Standard error") | [t-statistic](https://en.wikipedia.org/wiki/T-statistic "T-statistic") | [p-value](https://en.wikipedia.org/wiki/P-value "P-value") |
|  | | | | |
In this table:
- The *Value* column gives the least squares estimates of parameters *Ī²j*
- The *Std error* column shows [standard errors](https://en.wikipedia.org/wiki/Standard_error_\(statistics\) "Standard error (statistics)") of each coefficient estimate: ![{\\displaystyle {\\hat {\\sigma }}\_{j}=\\left({\\hat {\\sigma }}^{2}\\left\[Q\_{xx}^{-1}\\right\]\_{jj}\\right)^{\\frac {1}{2}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/5087e66171bf3ef9ad3ac75decdd715274919669)
- The *[t-statistic](https://en.wikipedia.org/wiki/T-statistic "T-statistic")* and *p-value* columns are testing whether any of the coefficients might be equal to zero. The *t*\-statistic is calculated simply as . If the errors Īµ follow a normal distribution, *t* follows a Student-t distribution. Under weaker conditions, *t* is asymptotically normal. Large values of *t* indicate that the null hypothesis can be rejected and that the corresponding coefficient is not zero. The second column, [*p*\-value](https://en.wikipedia.org/wiki/P-value "P-value"), expresses the results of the hypothesis test as a [significance level](https://en.wikipedia.org/wiki/Statistical_significance "Statistical significance"). Conventionally, *p*\-values smaller than 0.05 are taken as evidence that the population coefficient is nonzero.
- *R-squared* is the [coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination "Coefficient of determination") indicating goodness-of-fit of the regression. This statistic will be equal to one if fit is perfect, and to zero when regressors *X* have no explanatory power whatsoever. This is a biased estimate of the population *R-squared*, and will never decrease if additional regressors are added, even if they are irrelevant.
- *Adjusted R-squared* is a slightly modified version of , designed to penalize for the excess number of regressors which do not add to the explanatory power of the regression. This statistic is always smaller than , can decrease as new regressors are added, and even be negative for poorly fitting models:
- *Log-likelihood* is calculated under the assumption that errors follow normal distribution. Even though the assumption is not very reasonable, this statistic may still find its use in conducting LR tests.
- *[DurbināWatson statistic](https://en.wikipedia.org/wiki/Durbin%E2%80%93Watson_statistic "DurbināWatson statistic")* tests whether there is any evidence of serial correlation between the residuals. As a rule of thumb, the value smaller than 2 will be an evidence of positive correlation.
- *[Akaike information criterion](https://en.wikipedia.org/wiki/Akaike_information_criterion "Akaike information criterion")* and *[Schwarz criterion](https://en.wikipedia.org/wiki/Schwarz_criterion "Schwarz criterion")* are both used for model selection. Generally when comparing two alternative models, smaller values of one of these criteria will indicate a better model.[\[36\]](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_note-36)
- *Standard error of regression* is an estimate of *Ļ*, standard error of the error term.
- *Total sum of squares*, *model sum of squared*, and *residual sum of squares* tell us how much of the initial variation in the sample were explained by the regression.
- *F-statistic* tries to test the hypothesis that all coefficients (except the intercept) are equal to zero. This statistic has *F*(*pā1*,*nāp*) distribution under the null hypothesis and normality assumption, and its *p-value* indicates probability that the hypothesis is indeed true. Note that when errors are not normal this statistic becomes invalid, and other tests such as [Wald test](https://en.wikipedia.org/wiki/Wald_test "Wald test") or [LR test](https://en.wikipedia.org/wiki/Likelihood_ratio_test "Likelihood ratio test") should be used.
[](https://en.wikipedia.org/wiki/File:OLS_example_weight_vs_height_residuals.svg)
Residuals plot
Ordinary least squares analysis often includes the use of diagnostic plots designed to detect departures of the data from the assumed form of the model. These are some of the common diagnostic plots:
- Residuals against the explanatory variables in the model. A non-linear relation between these variables suggests that the linearity of the conditional mean function may not hold. Different levels of variability in the residuals for different levels of the explanatory variables suggests possible heteroscedasticity.
- Residuals against explanatory variables not in the model. Any relation of the residuals to these variables would suggest considering these variables for inclusion in the model.
- Residuals against the fitted values, .
- Residuals against the preceding residual. This plot may identify serial correlations in the residuals.
An important consideration when carrying out statistical inference using regression models is how the data were sampled. In this example, the data are averages rather than measurements on individual women. The fit of the model is very good, but this does not imply that the weight of an individual woman can be predicted with high accuracy based only on her height.
### Sensitivity to rounding
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=24 "Edit section: Sensitivity to rounding")\]
This example also demonstrates that coefficients determined by these calculations are sensitive to how the data is prepared. The heights were originally given rounded to the nearest inch and have been converted and rounded to the nearest centimetre. Since the conversion factor is one inch to 2.54 cm this is *not* an exact conversion. The original inches can be recovered by Round(x/0.0254) and then re-converted to metric without rounding. If this is done the results become:
| | Const | Height | Height2 |
|---|---|---|---|
| Converted to metric with rounding. | 128\.8128 | ā143.162 | 61\.96033 |
| Converted to metric without rounding. | 119\.0205 | ā131.5076 | 58\.5046 |
[](https://en.wikipedia.org/wiki/File:HeightWeightResiduals.jpg)
Residuals to a quadratic fit for correctly and incorrectly converted data.
Using either of these equations to predict the weight of a 5' 6" (1.6764 m) woman gives similar values: 62.94 kg with rounding vs. 62.98 kg without rounding. Thus a seemingly small variation in the data has a real effect on the coefficients but a small effect on the results of the equation.
While this may look innocuous in the middle of the data range it could become significant at the extremes or in the case where the fitted model is used to project outside the data range ([extrapolation](https://en.wikipedia.org/wiki/Extrapolation "Extrapolation")).
This highlights a common error: this example is an abuse of OLS which inherently requires that the errors in the independent variable (in this case height) are zero or at least negligible. The initial rounding to nearest inch plus any actual measurement errors constitute a finite and non-negligible error. As a result, the fitted parameters are not the best estimates they are presumed to be. Though not totally spurious the error in the estimation will depend upon relative size of the *x* and *y* errors.
## Another example with less real data
\[[edit](https://en.wikipedia.org/w/index.php?title=Ordinary_least_squares&action=edit§ion=25 "Edit section: Another example with less real data")\]
We can use the least square mechanism to figure out the equation of a two body orbit in polar base co-ordinates. The equation typically used is  where  is the radius of how far the object is from one of the bodies. In the equation the parameters  and  are used to determine the path of the orbit. We have measured the following data.
|  (in degrees) |
|---|
We need to find the least-squares approximation of  and  for the given data.
First we need to represent e and p in a linear form. So we are going to rewrite the equation  as .
Furthermore, one could fit for [apsides](https://en.wikipedia.org/wiki/Apsides "Apsides") by expanding  with an extra parameter as , which is linear in both  and in the extra basis function .
We use the original two-parameter form to represent our observational data as:
where:
; ;  contains the coefficients of  in the first column, which are all 1, and the coefficients of  in the second column, given by ; and , such that:
On solving we get ,
so  and 
- [Bayesian least squares](https://en.wikipedia.org/wiki/Minimum_mean_square_error "Minimum mean square error")
- [FamaāMacBeth regression](https://en.wikipedia.org/wiki/Fama%E2%80%93MacBeth_regression "FamaāMacBeth regression")
- [Nonlinear least squares](https://en.wikipedia.org/wiki/Non-linear_least_squares "Non-linear least squares")
- [Numerical methods for linear least squares](https://en.wikipedia.org/wiki/Numerical_methods_for_linear_least_squares "Numerical methods for linear least squares")
- [Nonlinear system identification](https://en.wikipedia.org/wiki/Nonlinear_system_identification "Nonlinear system identification")
1. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-1)**
["The Origins of Ordinary Least Squares Assumptions"](https://mathvoices.ams.org/featurecolumn/2022/03/01/ordinary-least-squares/). *Feature Column*. 2022-03-01. Retrieved 2024-05-16.
2. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-2)**
["What is a complete list of the usual assumptions for linear regression?"](https://stats.stackexchange.com/q/16381). *Cross Validated*. Retrieved 2022-09-28.
3. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-3)**
[Goldberger, Arthur S.](https://en.wikipedia.org/wiki/Arthur_Goldberger "Arthur Goldberger") (1964). ["Classical Linear Regression"](https://books.google.com/books?id=KZq5AAAAIAAJ&pg=PA156). [*Econometric Theory*](https://archive.org/details/econometrictheor0000gold/page/158). New York: John Wiley & Sons. pp. [158](https://archive.org/details/econometrictheor0000gold/page/158). [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-471-31101-4](https://en.wikipedia.org/wiki/Special:BookSources/0-471-31101-4 "Special:BookSources/0-471-31101-4")
.
4. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-4)**
[Hayashi, Fumio](https://en.wikipedia.org/wiki/Fumio_Hayashi "Fumio Hayashi") (2000). *Econometrics*. Princeton University Press. p. 15. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[9780691010182](https://en.wikipedia.org/wiki/Special:BookSources/9780691010182 "Special:BookSources/9780691010182")
.
5. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-5)** [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 18).
6. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-6)**
Ghilani, Charles D.; Wolf, Paul R. (12 June 2006). [*Adjustment Computations: Spatial Data Analysis*](https://books.google.com/books?id=hZ4mAOXVowoC&pg=PA160). John Wiley & Sons. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[9780471697282](https://en.wikipedia.org/wiki/Special:BookSources/9780471697282 "Special:BookSources/9780471697282")
.
7. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-7)**
Hofmann-Wellenhof, Bernhard; Lichtenegger, Herbert; Wasle, Elmar (20 November 2007). [*GNSS ā Global Navigation Satellite Systems: GPS, GLONASS, Galileo, and more*](https://books.google.com/books?id=Np7y43HU_m8C&pg=PA263). Springer. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[9783211730171](https://en.wikipedia.org/wiki/Special:BookSources/9783211730171 "Special:BookSources/9783211730171")
.
8. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-8)**
Xu, Guochang (5 October 2007). [*GPS: Theory, Algorithms and Applications*](https://books.google.com/books?id=peYFZ69HqEsC&pg=PA134). Springer. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[9783540727156](https://en.wikipedia.org/wiki/Special:BookSources/9783540727156 "Special:BookSources/9783540727156")
.
9. ^ [***a***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hayashi_2000_loc=page_19_9-0) [***b***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hayashi_2000_loc=page_19_9-1) [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 19)
10. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-q011_10-0)**
Hoaglin, David C.; Welsch, Roy E. (1978). ["The Hat Matrix in Regression and ANOVA"](https://doi.org/10.1080%2F00031305.1978.10479237). *The American Statistician*. **32** (1): 17ā22\. [doi](https://en.wikipedia.org/wiki/Doi_\(identifier\) "Doi (identifier)"):[10\.1080/00031305.1978.10479237](https://doi.org/10.1080%2F00031305.1978.10479237). [hdl](https://en.wikipedia.org/wiki/Hdl_\(identifier\) "Hdl (identifier)"):[1721\.1/1920](https://hdl.handle.net/1721.1%2F1920). [ISSN](https://en.wikipedia.org/wiki/ISSN_\(identifier\) "ISSN (identifier)") [0003-1305](https://search.worldcat.org/issn/0003-1305).
11. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-11)** [Julian Faraway (2000), *Practical Regression and Anova using R*](https://cran.r-project.org/doc/contrib/Faraway-PRA.pdf)
12. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-12)**
Kenney, J.; Keeping, E. S. (1963). *Mathematics of Statistics*. van Nostrand. p. 187.
13. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-13)**
Zwillinger, Daniel (1995). [*Standard Mathematical Tables and Formulae*](https://en.wikipedia.org/wiki/CRC_Standard_Mathematical_Tables "CRC Standard Mathematical Tables"). Chapman\&Hall/CRC. p. 626. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-8493-2479-3](https://en.wikipedia.org/wiki/Special:BookSources/0-8493-2479-3 "Special:BookSources/0-8493-2479-3")
.
14. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-14)** [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 20)
15. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-15)**
Akbarzadeh, Vahab (7 May 2014). ["Line Estimation"](https://mlmadesimple.wordpress.com/2014/05/07/line-estimation/).
16. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-16)** [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 49)
17. ^ [***a***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hayashi_2000_loc=page_52_17-0) [***b***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hayashi_2000_loc=page_52_17-1) [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 52)
18. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hayashi_2000_loc=page_10_18-0)** [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 10)
19. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Tibshirani-1996_19-0)**
Tibshirani, Robert (1996). "Regression Shrinkage and Selection via the Lasso". *Journal of the Royal Statistical Society, Series B*. **58** (1): 267ā288\. [doi](https://en.wikipedia.org/wiki/Doi_\(identifier\) "Doi (identifier)"):[10\.1111/j.2517-6161.1996.tb02080.x](https://doi.org/10.1111%2Fj.2517-6161.1996.tb02080.x). [JSTOR](https://en.wikipedia.org/wiki/JSTOR_\(identifier\) "JSTOR (identifier)") [2346178](https://www.jstor.org/stable/2346178).
20. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Efron-2004_20-0)**
Efron, Bradley; Hastie, Trevor; Johnstone, Iain; Tibshirani, Robert (2004). "Least Angle Regression". *The Annals of Statistics*. **32** (2): 407ā451\. [arXiv](https://en.wikipedia.org/wiki/ArXiv_\(identifier\) "ArXiv (identifier)"):[math/0406456](https://arxiv.org/abs/math/0406456). [doi](https://en.wikipedia.org/wiki/Doi_\(identifier\) "Doi (identifier)"):[10\.1214/009053604000000067](https://doi.org/10.1214%2F009053604000000067). [JSTOR](https://en.wikipedia.org/wiki/JSTOR_\(identifier\) "JSTOR (identifier)") [3448465](https://www.jstor.org/stable/3448465). [S2CID](https://en.wikipedia.org/wiki/S2CID_\(identifier\) "S2CID (identifier)") [204004121](https://api.semanticscholar.org/CorpusID:204004121).
21. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Hawkins-1973_21-0)**
Hawkins, Douglas M. (1973). "On the Investigation of Alternative Regressions by Principal Component Analysis". *Journal of the Royal Statistical Society, Series C*. **22** (3): 275ā286\. [doi](https://en.wikipedia.org/wiki/Doi_\(identifier\) "Doi (identifier)"):[10\.2307/2346776](https://doi.org/10.2307%2F2346776). [JSTOR](https://en.wikipedia.org/wiki/JSTOR_\(identifier\) "JSTOR (identifier)") [2346776](https://www.jstor.org/stable/2346776).
22. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Jolliffe-1982_22-0)**
Jolliffe, Ian T. (1982). "A Note on the Use of Principal Components in Regression". *Journal of the Royal Statistical Society, Series C*. **31** (3): 300ā303\. [doi](https://en.wikipedia.org/wiki/Doi_\(identifier\) "Doi (identifier)"):[10\.2307/2348005](https://doi.org/10.2307%2F2348005). [JSTOR](https://en.wikipedia.org/wiki/JSTOR_\(identifier\) "JSTOR (identifier)") [2348005](https://www.jstor.org/stable/2348005).
23. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-23)** [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), pages 27, 30)
24. ^ [***a***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-HayashiFSP_24-0) [***b***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-HayashiFSP_24-1) [***c***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-HayashiFSP_24-2) [Hayashi (2000](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFHayashi2000), page 27)
25. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-25)**
[Amemiya, Takeshi](https://en.wikipedia.org/wiki/Takeshi_Amemiya "Takeshi Amemiya") (1985). [*Advanced Econometrics*](https://archive.org/details/advancedeconomet00amem). Harvard University Press. p. [13](https://archive.org/details/advancedeconomet00amem/page/13). [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[9780674005600](https://en.wikipedia.org/wiki/Special:BookSources/9780674005600 "Special:BookSources/9780674005600")
.
26. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-26)** [Amemiya (1985](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFAmemiya1985), page 14)
27. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-27)**
[Rao, C. R.](https://en.wikipedia.org/wiki/C._R._Rao "C. R. Rao") (1973). *Linear Statistical Inference and its Applications* (Second ed.). New York: J. Wiley & Sons. p. 319. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-471-70823-2](https://en.wikipedia.org/wiki/Special:BookSources/0-471-70823-2 "Special:BookSources/0-471-70823-2")
.
28. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-28)** [Amemiya (1985](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFAmemiya1985), page 20)
29. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-29)** [Amemiya (1985](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFAmemiya1985), page 27)
30. ^ [***a***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-DvdMck33_30-0) [***b***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-DvdMck33_30-1)
Davidson, Russell; [MacKinnon, James G.](https://en.wikipedia.org/wiki/James_G._MacKinnon "James G. MacKinnon") (1993). *Estimation and Inference in Econometrics*. New York: Oxford University Press. p. 33. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-19-506011-3](https://en.wikipedia.org/wiki/Special:BookSources/0-19-506011-3 "Special:BookSources/0-19-506011-3")
.
31. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-31)** [Davidson & MacKinnon (1993](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFDavidsonMacKinnon1993), page 36)
32. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-32)** [Davidson & MacKinnon (1993](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFDavidsonMacKinnon1993), page 20)
33. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-33)**
["Memento on EViews Output"](https://scholar.harvard.edu/files/jbenchimol/files/memento-eviews.pdf) (PDF). Retrieved 28 December 2020.
34. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-34)** [Amemiya (1985](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFAmemiya1985), page 21)
35. ^ [***a***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Amemiya22_35-0) [***b***](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-Amemiya22_35-1) [Amemiya (1985](https://en.wikipedia.org/wiki/Ordinary_least_squares#CITEREFAmemiya1985), page 22)
36. **[^](https://en.wikipedia.org/wiki/Ordinary_least_squares#cite_ref-36)**
Burnham, Kenneth P.; Anderson, David R. (2002). [*Model Selection and Multi-Model Inference*](https://archive.org/details/modelselectionmu0000burn) (2nd ed.). Springer. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-387-95364-7](https://en.wikipedia.org/wiki/Special:BookSources/0-387-95364-7 "Special:BookSources/0-387-95364-7")
.
- [Dougherty, Christopher](https://en.wikipedia.org/wiki/Christopher_Dougherty "Christopher Dougherty") (2002). *Introduction to Econometrics* (2nd ed.). New York: Oxford University Press. pp. 48ā113\. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[0-19-877643-8](https://en.wikipedia.org/wiki/Special:BookSources/0-19-877643-8 "Special:BookSources/0-19-877643-8")
.
- [Gujarati, Damodar N.](https://en.wikipedia.org/wiki/Damodar_N._Gujarati "Damodar N. Gujarati"); [Porter, Dawn C.](https://en.wikipedia.org/wiki/Dawn_C._Porter "Dawn C. Porter") (2009). *Basic Econometics* (Fifth ed.). Boston: McGraw-Hill Irwin. pp. 55ā96\. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[978-0-07-337577-9](https://en.wikipedia.org/wiki/Special:BookSources/978-0-07-337577-9 "Special:BookSources/978-0-07-337577-9")
.
- [Heij, Christiaan](https://en.wikipedia.org/wiki/Christiaan_Heij "Christiaan Heij"); Boer, Paul; [Franses, Philip H.](https://en.wikipedia.org/wiki/Philip_Hans_Franses "Philip Hans Franses"); [Kloek, Teun](https://en.wikipedia.org/wiki/Teun_Kloek "Teun Kloek"); [van Dijk, Herman K.](https://en.wikipedia.org/wiki/Herman_K._van_Dijk "Herman K. van Dijk") (2004). *Econometric Methods with Applications in Business and Economics* (1st ed.). Oxford: Oxford University Press. pp. 76ā115\. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[978-0-19-926801-6](https://en.wikipedia.org/wiki/Special:BookSources/978-0-19-926801-6 "Special:BookSources/978-0-19-926801-6")
.
- Hill, R. Carter; Griffiths, William E.; Lim, Guay C. (2008). *Principles of Econometrics* (3rd ed.). Hoboken, NJ: John Wiley & Sons. pp. 8ā47\. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[978-0-471-72360-8](https://en.wikipedia.org/wiki/Special:BookSources/978-0-471-72360-8 "Special:BookSources/978-0-471-72360-8")
.
- [Wooldridge, Jeffrey](https://en.wikipedia.org/wiki/Jeffrey_Wooldridge "Jeffrey Wooldridge") (2008). ["The Simple Regression Model"](https://books.google.com/books?id=64vt5TDBNLwC&pg=PA22). *Introductory Econometrics: A Modern Approach* (4th ed.). Mason, OH: Cengage Learning. pp. 22ā67\. [ISBN](https://en.wikipedia.org/wiki/ISBN_\(identifier\) "ISBN (identifier)")
[978-0-324-58162-1](https://en.wikipedia.org/wiki/Special:BookSources/978-0-324-58162-1 "Special:BookSources/978-0-324-58162-1")
. | |||||||||
| ML Classification | ||||||||||
| ML Categories |
Raw JSON{
"/Science": 890,
"/Science/Mathematics": 865,
"/Science/Mathematics/Statistics": 841
} | |||||||||
| ML Page Types |
Raw JSON{
"/Article": 997,
"/Article/Wiki": 606
} | |||||||||
| ML Intent Types |
Raw JSON{
"Informational": 999
} | |||||||||
| Content Metadata | ||||||||||
| Language | en | |||||||||
| Author | null | |||||||||
| Publish Time | not set | |||||||||
| Original Publish Time | 2013-08-24 22:28:28 (12 years ago) | |||||||||
| Republished | No | |||||||||
| Word Count (Total) | 11,876 | |||||||||
| Word Count (Content) | 8,168 | |||||||||
| Links | ||||||||||
| External Links | 58 | |||||||||
| Internal Links | 430 | |||||||||
| Technical SEO | ||||||||||
| Meta Nofollow | No | |||||||||
| Meta Noarchive | No | |||||||||
| JS Rendered | No | |||||||||
| Redirect Target | null | |||||||||
| Performance | ||||||||||
| Download Time (ms) | 57 | |||||||||
| TTFB (ms) | 33 | |||||||||
| Download Size (bytes) | 74,827 | |||||||||
| Shard | 152 (laksa) | |||||||||
| Root Hash | 17790707453426894952 | |||||||||
| Unparsed URL | org,wikipedia!en,/wiki/Ordinary_least_squares s443 | |||||||||