ℹ️ Skipped - page is already crawled
| Filter | Status | Condition | Details |
|---|---|---|---|
| HTTP status | PASS | download_http_code = 200 | HTTP 200 |
| Age cutoff | PASS | download_stamp > now() - 6 MONTH | 0.1 months ago |
| History drop | PASS | isNull(history_drop_reason) | No drop reason |
| Spam/ban | PASS | fh_dont_index != 1 AND ml_spam_score = 0 | ml_spam_score=0 |
| Canonical | PASS | meta_canonical IS NULL OR = '' OR = src_unparsed | Not set |
| Property | Value |
|---|---|
| URL | https://shainarace.github.io/LinearAlgebra/eigen.html |
| Last Crawled | 2026-04-08 12:29:03 (4 days ago) |
| First Indexed | 2021-08-27 22:31:15 (4 years ago) |
| HTTP Status Code | 200 |
| Meta Title | Chapter 12 Eigenvalues and Eigenvectors | Linear Algebra for Data Science |
| Meta Description | A traditional textbook fused with a collection of data science case studies that was engineered to weave practicality and applied problem solving into a linear algebra curriculum |
| Meta Canonical | null |
| Boilerpipe Text | Eigenvalues and eigenvectors are (scalar, vector)-pairs that form the “essence” of a matrix. The prefix eigen- is adopted from the German word
eigen
which means “characteristic, inherent, own” and was introduced by David Hilbert in 1904, but the study of these characteristic directions and magnitudes dates back to Euler’s study of the rotational motion of rigid bodies in the
18
t
h
century.
Definition 12.1 (Eigenvalues and Eigenvectors)
For a square matrix
A
n
×
n
, a scalar
λ
is called an
eigenvalue
of
A
if there is a nonzero vector
x
such that
A
x
=
λ
x
.
Such a vector,
x
is called an
eigenvector
of
A
corresponding to the
eigenvalue
λ
. We sometimes refer to the pair
(
λ
,
x
)
as an
eigenpair.
Eigenvalues and eigenvectors have numerous applications from graphic design to quantum mechanics to geology to epidemiology. The main application of note for data scientists is Principal Component Analysis, but we will also see eigenvalue equations used in social network analysis to determine important players in a network and to detect communities in the network. Before we dive into those applications, let’s first get a handle on the definition by exploring some examples.
Example 12.1 (Eigenvalues and Eigenvectors)
Determine whether
x
=
(
1
1
)
is an eigenvector of
A
=
(
3
1
1
3
)
and if so, find the corresponding eigenvalue.\
To determine whether
x
is an eigenvector, we want to compute
A
x
and observe whether the result is a multiple of
x
. If this is the case, then the multiplication factor is the corresponding eigenvalue:
A
x
=
(
3
1
1
3
)
(
1
1
)
=
(
4
4
)
=
4
(
1
1
)
From this it follows that
x
is
an eigenvector of
A
and the corresponding eigenvalue is
λ
=
4
.\
Is the vector
y
=
(
2
2
)
an eigenvector?
A
y
=
(
3
1
1
3
)
(
2
2
)
=
(
8
8
)
=
4
(
2
2
)
=
4
y
Yes, it is and it corresponds to the
same
eigenvalue,
λ
=
4
Example
12.1
shows a very important property of eigenvalue-eigenvector pairs. If
(
λ
,
x
)
is an eigenpair then any scalar multiple of
x
is also an eigenvector corresponding to
λ
. To see this, let
(
λ
,
x
)
be an eigenpair for a matrix
A
(which means that
A
x
=
λ
x
) and let
y
=
α
x
be any scalar multiple of
x
. Then we have,
A
y
=
A
(
α
x
)
=
α
(
A
x
)
=
α
(
λ
x
)
=
λ
(
α
x
)
=
λ
y
which shows that
y
(or any scalar multiple of
x
) is also an eigenvector associated with the eigenvalue
λ
.
Thus, for each eigenvalue we have infinitely many eigenvectors. In the preceding example, the eigenvectors associated with
λ
=
4
will be scalar multiples of
x
=
(
1
1
)
. You may recall from Chapter
7
that the set of all scalar multiples of
x
is denoted
s
p
a
n
(
x
)
. The
s
p
a
n
(
x
)
in this example represents the
eigenspace
of
λ
.
Note: when using software to compute eigenvectors, it is standard practice for the software to provide the normalized/unit eigenvector.
In some situations, an eigenvalue can have multiple eigenvectors which are linearly independent. The number of linearly independent eigenvectors associated with an eigenvalue is called the
geometric multiplicity
of the eigenvalue. Example
12.2
clarifies this concept.
Example 12.2 (Geometric Multiplicity)
Consider the matrix
A
=
(
3
0
0
3
)
. It should be straightforward to see that
x
1
=
(
1
0
)
and
x
2
=
(
0
1
)
are both eigenvectors corresponding to the eigenvalue
λ
=
3
.
x
1
and
x
2
are linearly independent, therefore the geometric multiplicity of
λ
=
3
is 2.
What happens if we take a linear combination of
x
1
and
x
2
? Is that also an eigenvector?
Consider
y
=
(
2
3
)
=
2
x
1
+
3
x
2
. Then
A
y
=
(
3
0
0
3
)
(
2
3
)
=
(
6
9
)
=
3
(
2
3
)
=
3
y
shows that
y
is also an eigenvector associated with
λ
=
3
.
The
eigenspace
corresponding to
λ
=
3
is the set of all linear combinations of
x
1
and
x
2
, i.e. the
s
p
a
n
(
x
1
,
x
2
)
.
We can generalize the result that we saw in Example
12.2
for any square matrix and any geometric multiplicity. Let
A
n
×
n
have an eigenvalue
λ
with geometric multiplicity
k
. This means there are
k
linearly independent eigenvectors,
x
1
,
x
2
,
…
,
x
k
such that
A
x
i
=
λ
x
i
for each eigenvector
x
i
. Now if we let
y
be a vector in the
s
p
a
n
(
x
1
,
x
2
,
…
,
x
k
)
then
y
is some linear combination of the
x
i
’s:
y
=
α
1
x
2
+
α
2
x
2
+
⋯
+
α
k
x
k
Observe what happens when we multiply
y
by
A
:
A
y
=
A
(
α
1
x
2
+
α
2
x
2
+
⋯
+
α
k
x
k
)
=
α
1
(
A
x
1
)
+
α
2
(
A
x
2
)
+
⋯
+
α
k
(
A
x
k
)
=
α
1
(
λ
x
1
)
+
α
2
(
λ
x
2
)
+
⋯
+
α
k
(
λ
x
k
)
=
λ
(
α
1
x
2
+
α
2
x
2
+
⋯
+
α
k
x
k
)
=
λ
y
which shows that
y
(or any vector in the
s
p
a
n
(
x
1
,
x
2
,
…
,
x
k
)
) is an eigenvector of
A
corresponding to
λ
.
This proof allows us to formally define the concept of an eigenspace.
Definition 12.2 (Eigenspace)
Let
A
be a square matrix and let
λ
be an eigenvalue of
A
. The set of all eigenvectors corresponding to
λ
, together with the zero vector, is called the
eigenspace
of
λ
. The number of basis vectors required to form the eigenspace is called the
geometric multiplicity
of
λ
.
Now, let’s attempt the eigenvalue problem from the other side. Given an eigenvalue, we will find the corresponding eigenspace in Example
12.3
.
Example 12.3 (Eigenvalues and Eigenvectors)
Show that
λ
=
5
is an eigenvalue of
A
=
(
1
2
4
3
)
and determine the eigenspace of
λ
=
5
.
Attempting the problem from this angle requires slightly more work. We want to find a vector
x
such that
A
x
=
5
x
. Setting this up, we have:
A
x
=
5
x
.
What we want to do is move both terms to one side and factor out the vector
x
. In order to do this, we must use an identity matrix, otherwise the equation wouldn’t make sense (we’d be subtracting a constant from a matrix).
A
x
−
5
x
=
0
(
A
−
5
I
)
x
=
0
(
(
1
2
4
3
)
−
(
5
0
0
5
)
)
(
x
1
x
2
)
=
(
0
0
)
(
−
4
2
4
−
2
)
(
x
1
x
2
)
=
(
0
0
)
Clearly, the matrix
A
−
λ
I
is singular (i.e. does not have linearly independent rows/columns). This will always be the case by the definition
A
x
=
λ
x
, and is often used as an alternative definition.\
In order to solve this homogeneous system of equations, we use Gaussian elimination:
(
−
4
2
0
4
−
2
0
)
⟶
(
1
−
1
2
0
0
0
0
)
This implies that any vector
x
for which
x
1
−
1
2
x
2
=
0
satisfies the eigenvector equation. We can pick any such vector, for example
x
=
(
1
2
)
, and say that the eigenspace of
λ
=
5
is
s
p
a
n
{
(
1
2
)
}
If we didn’t know either an eigenvalue or eigenvector of
A
and instead wanted to find both, we would first find eigenvalues by determining all possible
λ
such that
A
−
λ
I
is singular and then find the associated eigenvectors. There are some tricks which allow us to do this by hand for
2
×
2
and
3
×
3
matrices, but after that the computation time is unworthy of the effort. Now that we have a good understanding of how to interpret eigenvalues and eigenvectors algebraically, let’s take a look at some of the things that they can do, starting with one important fact.
Definition 12.3 (Eigenvalues and the Trace of a Matrix)
Let
A
be an
n
×
n
matrix with eigenvalues
λ
1
,
λ
2
,
…
,
λ
n
. Then the sum of the eigenvalues is equal to the trace of the matrix (recall that the trace of a matrix is the sum of its diagonal elements).
T
r
a
c
e
(
A
)
=
∑
i
=
1
n
λ
i
.
Example 12.4 (Trace of Covariance Matrix)
Suppose that we had a collection of
n
observations on
p
variables,
x
1
,
x
2
,
…
,
x
p
. After centering the data to have zero mean, we can compute the sample variances as:
v
a
r
(
x
i
)
=
1
n
−
1
x
i
T
x
i
=
‖
x
i
‖
2
These variances form the diagonal elements of the sample covariance matrix,
Σ
=
1
n
−
1
X
T
X
Thus, the total variance of this data is
1
n
−
1
∑
i
=
1
n
‖
x
i
‖
2
=
T
r
a
c
e
(
Σ
)
=
∑
i
=
1
n
λ
i
.
In other words, the sum of the eigenvalues of a covariance matrix provides the total variance in the variables
x
1
,
…
,
x
p
.
Diagonalization
Let’s take another look at Example
12.3
. We already showed that
λ
1
=
5
and
v
1
=
(
1
2
)
is an eigenpair for the matrix
A
=
(
1
2
4
3
)
. You may verify that
λ
2
=
−
1
and
v
2
=
(
1
−
1
)
is another eigenpair. Suppose we create a matrix of eigenvectors:
V
=
(
v
1
,
v
2
)
=
(
1
1
2
−
1
)
and a diagonal matrix containing the corresponding eigenvalues:
D
=
(
5
0
0
−
1
)
Then it is easy to verify that
A
V
=
V
D
:
A
V
=
(
1
2
4
3
)
(
1
1
2
−
1
)
=
(
5
−
1
10
1
)
=
(
1
1
2
−
1
)
(
5
0
0
−
1
)
=
V
D
If the columns of
V
are linearly independent, which they are in this case, we can write:
V
−
1
A
V
=
D
What we have just done is develop a way to transform a matrix
A
into a diagonal matrix
D
. This is known as
diagonalization.
Definition 12.4 (Diagonalizable)
An
n
×
n
matrix
A
is said to be
diagonalizable
if there exists an invertible matrix
P
and a diagonal matrix
D
such that
P
−
1
A
P
=
D
This is possible if and only if the matrix
A
has
n
linearly independent eigenvectors (known as a
complete set of eigenvectors
). The matrix
P
is then the matrix of eigenvectors and the matrix
D
contains the corresponding eigenvalues on the diagonal.
Determining whether or not a matrix
A
n
×
n
is diagonalizable is a little tricky. Having
r
a
n
k
(
A
)
=
n
is
not
a sufficient condition for having
n
linearly independent eigenvectors. The following matrix stands as a counter example:
A
=
(
−
3
1
−
3
20
3
10
2
−
2
4
)
This matrix has full rank but only two linearly independent eigenvectors. Fortunately, for our primary application of diagonalization, we will be dealing with a symmetric matrix, which can always be diagonalized. In fact, symmetric matrices have an additional property which makes this diagonalization particularly nice, as we will see in Chapter
13
.
Geometric Interpretation of Eigenvalues and Eigenvectors
Since any scalar multiple of an eigenvector is still an eigenvector, let’s consider for the present discussion unit eigenvectors
x
of a square matrix
A
- those with length
‖
x
‖
=
1
. By the definition, we know that
A
x
=
λ
x
We know that geometrically, if we multiply
x
by
A
, the resulting vector points in the same direction as
x
. Geometrically, it turns out that multiplying the unit circle or unit sphere by a matrix
A
carves out an ellipse, or an ellipsoid. We can see eigenvectors visually by watching how multiplication by a matrix
A
changes the unit vectors. Figure
12.1
illustrates this. The blue arrows represent (a sampling of) the unit circle, all vectors
x
for which
‖
x
‖
=
1
. The red arrows represent the image of the blue arrows after multiplication by
A
, or
A
x
for each vector
x
. We can see how almost every vector changes direction when multiplied by
A
, except the eigenvector directions which are marked in black. Such a picture provides a nice geometrical interpretation of eigenvectors for a general matrix, but we will see in Chapter
13
just how powerful these eigenvector directions are when we look at symmetric matrix.
Figure 12.1: Visualizing eigenvectors (in black) using the image (in red) of the unit sphere (in blue) after multiplication by
A
.
Exercises
Show that
v
is an eigenvector of
A
and find the corresponding eigenvalue:
$= & 2 \2 & 1 \-3 $
$= & 1 \6 & 0 \-2 $
$= & -2 \5 & -7 \2 $
Show that
λ
is an eigenvalue of
A
and list two eigenvectors corresponding to this eigenvalue:
A
=
(
0
4
−
1
5
)
λ
=
4
A
=
(
0
4
−
1
5
)
λ
=
1
Based on the eigenvectors you found in exercises 2, can the matrix
A
be diagonalized? Why or why not? If diagonalization is possible, explain how it would be done.
Can a rectangular matrix have eigenvalues/eigenvectors? |
| Markdown | \\usepackage
a
m
s
f
o
n
t
s
\\usepackage
c
a
n
c
e
l
\\usepackage
a
m
s
m
a
t
h
\\usepackage
s
y
s
t
e
m
e
\\usepackage
a
m
s
t
h
m
\\usepackage
x
c
o
l
o
r
\\usepackage
b
o
l
d
s
y
m
b
o
l
\\newtheorem
d
e
f
i
n
i
t
i
o
n
D
e
f
i
n
i
t
i
o
n
\\newtheorem
l
e
m
m
a
L
e
m
m
a
\\newtheorem
f
a
c
t
F
a
c
t
\\newtheorem
t
h
m
T
h
e
o
r
e
m
\\newtheorem
e
x
a
m
p
l
e
E
x
a
m
p
l
e
\[
s
e
c
t
i
o
n
\]
\\usepackage
p
d
f
p
a
g
e
s
,
c
a
n
c
e
l
Type to search
- 
- **Linear Algebra for Data Science**
- **with examples in R**
- - [Preface](https://shainarace.github.io/LinearAlgebra/index.html)
- [Structure of the book](https://shainarace.github.io/LinearAlgebra/index.html#structure-of-the-book)
- [About the author](https://shainarace.github.io/LinearAlgebra/index.html#about-the-author)
- [Acknowledgements](https://shainarace.github.io/LinearAlgebra/index.html#acknowledgements)
- [**1** Introduction](https://shainarace.github.io/LinearAlgebra/intro.html)
- [**1\.1** What is Linear Algebra?](https://shainarace.github.io/LinearAlgebra/intro.html#what-is-linear-algebra)
- [**1\.2** Why Linear Algebra](https://shainarace.github.io/LinearAlgebra/intro.html#why-linear-algebra)
- [**1\.3** Describing Matrices and Vectors](https://shainarace.github.io/LinearAlgebra/intro.html#describing-matrices-and-vectors)
- [**1\.3.1** Dimension/Size of a Matrix](https://shainarace.github.io/LinearAlgebra/intro.html#dimensionsize-of-a-matrix)
- [**1\.3.2** ( i , j ) Notation](https://shainarace.github.io/LinearAlgebra/intro.html#ij-notation)
- [Example: Defining social networks](https://shainarace.github.io/LinearAlgebra/intro.html#example-defining-social-networks)
- [**1\.3.3** Example: Correlation matrices](https://shainarace.github.io/LinearAlgebra/intro.html#introcorr)
- [**1\.4** Vectors](https://shainarace.github.io/LinearAlgebra/intro.html#vectors)
- [**1\.4.1** Vector Geometry: n-space](https://shainarace.github.io/LinearAlgebra/intro.html#vector-geometry-n-space)
- [**1\.5** Matrix Operations](https://shainarace.github.io/LinearAlgebra/intro.html#matrix-operations)
- [**1\.5.1** Transpose](https://shainarace.github.io/LinearAlgebra/intro.html#transpose)
- [**1\.5.2** Trace of a Matrix](https://shainarace.github.io/LinearAlgebra/intro.html#trace-of-a-matrix)
- [**1\.6** Special Matrices and Vectors](https://shainarace.github.io/LinearAlgebra/intro.html#special)
- [**1\.7** Summary of Conventional Notation](https://shainarace.github.io/LinearAlgebra/intro.html#summary-of-conventional-notation)
- [**1\.8** Exercises](https://shainarace.github.io/LinearAlgebra/intro.html#exercises)
- [**1\.9** List of Key Terms](https://shainarace.github.io/LinearAlgebra/intro.html#list-of-key-terms)
- [**2** Matrix Arithmetic](https://shainarace.github.io/LinearAlgebra/mult.html)
- [**2\.1** Matrix Addition, Subtraction, and Scalar Multiplication](https://shainarace.github.io/LinearAlgebra/mult.html#matrix-addition-subtraction-and-scalar-multiplication)
- [**2\.2** Geometry of Vector Addition and Scalar Multiplication](https://shainarace.github.io/LinearAlgebra/mult.html#sec:vectoradd)
- [**2\.3** Linear Combinations](https://shainarace.github.io/LinearAlgebra/mult.html#linear-combinations)
- [**2\.4** Matrix Multiplication](https://shainarace.github.io/LinearAlgebra/mult.html#matrix-multiplication)
- [**2\.4.1** The Inner Product](https://shainarace.github.io/LinearAlgebra/mult.html#the-inner-product)
- [**2\.4.2** Matrix Product](https://shainarace.github.io/LinearAlgebra/mult.html#matrix-product)
- [**2\.4.3** Matrix-Vector Product](https://shainarace.github.io/LinearAlgebra/mult.html#matrix-vector-product)
- [Linear Combination view of Matrix Products](https://shainarace.github.io/LinearAlgebra/mult.html#linear-combination-view-of-matrix-products)
- [**2\.5** Vector Outer Products](https://shainarace.github.io/LinearAlgebra/mult.html#vector-outer-products)
- [**2\.6** The Identity and the Matrix Inverse](https://shainarace.github.io/LinearAlgebra/mult.html#the-identity-and-the-matrix-inverse)
- [**2\.7** Exercises](https://shainarace.github.io/LinearAlgebra/mult.html#exercises-1)
- [List of Key Terms](https://shainarace.github.io/LinearAlgebra/mult.html#list-of-key-terms-1)
- [**3** Applications of Matrix Multiplication](https://shainarace.github.io/LinearAlgebra/multapp.html)
- [**3\.1** Systems of Equations](https://shainarace.github.io/LinearAlgebra/multapp.html#systems-of-equations)
- [**3\.1.1** *Big* Systems of Equations](https://shainarace.github.io/LinearAlgebra/multapp.html#big-systems-of-equations)
- [**3\.2** Regression Analysis](https://shainarace.github.io/LinearAlgebra/multapp.html#regression-analysis)
- [**3\.3** Linear Combinations](https://shainarace.github.io/LinearAlgebra/multapp.html#linear-combinations-1)
- [**3\.4** Exercises](https://shainarace.github.io/LinearAlgebra/multapp.html#multapp-ex)
- [**4** R Programming Basics](https://shainarace.github.io/LinearAlgebra/r-programming-basics.html)
- [**5** Solving Systems of Equations](https://shainarace.github.io/LinearAlgebra/solvesys.html)
- [**5\.1** Gaussian Elimination](https://shainarace.github.io/LinearAlgebra/solvesys.html#gaussian-elimination)
- [**5\.1.1** Row Operations](https://shainarace.github.io/LinearAlgebra/solvesys.html#row-operations)
- [**5\.1.2** The Augmented Matrix](https://shainarace.github.io/LinearAlgebra/solvesys.html#the-augmented-matrix)
- [**5\.1.3** Gaussian Elimination Summary](https://shainarace.github.io/LinearAlgebra/solvesys.html#gaussian-elimination-summary)
- [**5\.2** Gauss-Jordan Elimination](https://shainarace.github.io/LinearAlgebra/solvesys.html#gauss-jordan-elimination)
- [**5\.2.1** Gauss-Jordan Elimination Summary](https://shainarace.github.io/LinearAlgebra/solvesys.html#gauss-jordan-elimination-summary)
- [**5\.3** Three Types of Systems](https://shainarace.github.io/LinearAlgebra/solvesys.html#three-types-of-systems)
- [**5\.3.1** The Unique Solution Case](https://shainarace.github.io/LinearAlgebra/solvesys.html#uniquesol)
- [**5\.3.2** The Inconsistent Case](https://shainarace.github.io/LinearAlgebra/solvesys.html#inconsistent)
- [**5\.3.3** The Infinite Solutions Case](https://shainarace.github.io/LinearAlgebra/solvesys.html#infinitesol)
- [**5\.3.4** Matrix Rank](https://shainarace.github.io/LinearAlgebra/solvesys.html#matrix-rank)
- [**5\.4** Solving Matrix Equations](https://shainarace.github.io/LinearAlgebra/solvesys.html#solving-matrix-equations)
- [**5\.4.1** Solving for the Inverse of a Matrix](https://shainarace.github.io/LinearAlgebra/solvesys.html#solving-for-the-inverse-of-a-matrix)
- [**5\.5** Gauss-Jordan Elimination in R](https://shainarace.github.io/LinearAlgebra/solvesys.html#gauss-jordan-elimination-in-r)
- [**5\.6** Exercises](https://shainarace.github.io/LinearAlgebra/solvesys.html#exercises-2)
- [**5\.7** List of Key Terms](https://shainarace.github.io/LinearAlgebra/solvesys.html#list-of-key-terms-2)
- [**6** Norms, Similarity, and Distance](https://shainarace.github.io/LinearAlgebra/norms.html)
- [**6\.1** Norms and Distances](https://shainarace.github.io/LinearAlgebra/norms.html#sec-norms)
- [**6\.2** Other useful norms and distances](https://shainarace.github.io/LinearAlgebra/norms.html#other-useful-norms-and-distances)
- [**6\.2.1** 1-norm, ‖ ⋆ ‖ 1.](https://shainarace.github.io/LinearAlgebra/norms.html#norm-star_1.)
- [**6\.2.2** ∞-norm, ‖ ⋆ ‖ ∞.](https://shainarace.github.io/LinearAlgebra/norms.html#infty-norm-star_infty.)
- [**6\.3** Inner Products](https://shainarace.github.io/LinearAlgebra/norms.html#inner-products)
- [**6\.3.1** Covariance](https://shainarace.github.io/LinearAlgebra/norms.html#covariance)
- [**6\.3.2** Mahalanobis Distance](https://shainarace.github.io/LinearAlgebra/norms.html#mahalanobis-distance)
- [**6\.3.3** Angular Distance](https://shainarace.github.io/LinearAlgebra/norms.html#angular-distance)
- [**6\.3.4** Correlation](https://shainarace.github.io/LinearAlgebra/norms.html#correlation)
- [**6\.4** Outer Products](https://shainarace.github.io/LinearAlgebra/norms.html#outer-products)
- [**6\.5** Exercises](https://shainarace.github.io/LinearAlgebra/norms.html#exercises-3)
- [**7** Linear Independence](https://shainarace.github.io/LinearAlgebra/linind.html)
- [**7\.1** Linear Independence](https://shainarace.github.io/LinearAlgebra/linind.html#linear-independence)
- [**7\.1.1** Determining Linear Independence](https://shainarace.github.io/LinearAlgebra/linind.html#determining-linear-independence)
- [**7\.2** Span of Vectors](https://shainarace.github.io/LinearAlgebra/linind.html#span)
- [**7\.3** Exercises](https://shainarace.github.io/LinearAlgebra/linind.html#exercises-4)
- [List of Key Terms](https://shainarace.github.io/LinearAlgebra/linind.html#list-of-key-terms-3)
- [**8** Basis and Change of Basis](https://shainarace.github.io/LinearAlgebra/basis.html)
- [**8\.1** Vector Space Models](https://shainarace.github.io/LinearAlgebra/basis.html#vector-space-models)
- [**8\.2** Exercises](https://shainarace.github.io/LinearAlgebra/basis.html#exercises-5)
- [List of Key Terms](https://shainarace.github.io/LinearAlgebra/basis.html#list-of-key-terms-4)
- [**9** Orthogonality](https://shainarace.github.io/LinearAlgebra/orthog.html)
- [**9\.1** Orthonormal Basis](https://shainarace.github.io/LinearAlgebra/orthog.html#orthonormal-basis)
- [**9\.2** Orthogonal Projection](https://shainarace.github.io/LinearAlgebra/orthog.html#orthogonal-projection)
- [**9\.3** Why??](https://shainarace.github.io/LinearAlgebra/orthog.html#why)
- [**9\.4** Exercises](https://shainarace.github.io/LinearAlgebra/orthog.html#exercises-6)
- [**10** Least Squares](https://shainarace.github.io/LinearAlgebra/leastsquares.html)
- [**10\.1** Introducing Error](https://shainarace.github.io/LinearAlgebra/leastsquares.html#introducing-error)
- [**10\.2** Why the normal equations?](https://shainarace.github.io/LinearAlgebra/leastsquares.html#why-the-normal-equations)
- [**10\.2.1** Geometrical Interpretation](https://shainarace.github.io/LinearAlgebra/leastsquares.html#geometrical-interpretation)
- [**10\.2.2** Calculus Derivation](https://shainarace.github.io/LinearAlgebra/leastsquares.html#calculus-derivation)
- [**11** Applications of Least Squares](https://shainarace.github.io/LinearAlgebra/lsapp.html)
- [**11\.1** Simple Linear Regression](https://shainarace.github.io/LinearAlgebra/lsapp.html#simple-linear-regression)
- [**11\.1.1** Cars Data](https://shainarace.github.io/LinearAlgebra/lsapp.html#cars-data)
- [**11\.1.2** Setting up the Normal Equations](https://shainarace.github.io/LinearAlgebra/lsapp.html#setting-up-the-normal-equations)
- [**11\.1.3** Solving for Parameter Estimates and Statistics](https://shainarace.github.io/LinearAlgebra/lsapp.html#solving-for-parameter-estimates-and-statistics)
- [**11\.1.4** OLS in R via `lm()`](https://shainarace.github.io/LinearAlgebra/lsapp.html#ols-in-r-via-lm)
- [**11\.2** Multiple Linear Regression](https://shainarace.github.io/LinearAlgebra/lsapp.html#multiple-linear-regression)
- [**11\.2.1** Bike Sharing Dataset](https://shainarace.github.io/LinearAlgebra/lsapp.html#bike-sharing-dataset)
- [**12** Eigenvalues and Eigenvectors](https://shainarace.github.io/LinearAlgebra/eigen.html)
- [**12\.1** Diagonalization](https://shainarace.github.io/LinearAlgebra/eigen.html#diagonalization)
- [**12\.2** Geometric Interpretation of Eigenvalues and Eigenvectors](https://shainarace.github.io/LinearAlgebra/eigen.html#geometric-interpretation-of-eigenvalues-and-eigenvectors "12.2 Geometric Interpretation of Eigenvalues and Eigenvectors")
- [**12\.3** Exercises](https://shainarace.github.io/LinearAlgebra/eigen.html#exercises-7)
- [**13** Principal Components Analysis](https://shainarace.github.io/LinearAlgebra/pca.html)
- [**13\.1** God’s Flashlight](https://shainarace.github.io/LinearAlgebra/pca.html#gods-flashlight)
- [**13\.2** PCA Details](https://shainarace.github.io/LinearAlgebra/pca.html#pca-details)
- [**13\.3** Geometrical comparison with Least Squares](https://shainarace.github.io/LinearAlgebra/pca.html#geometrical-comparison-with-least-squares)
- [**13\.4** Covariance or Correlation Matrix?](https://shainarace.github.io/LinearAlgebra/pca.html#covariance-or-correlation-matrix)
- [**13\.5** PCA in R](https://shainarace.github.io/LinearAlgebra/pca.html#pca-in-r)
- [**13\.5.1** Covariance PCA](https://shainarace.github.io/LinearAlgebra/pca.html#covariance-pca)
- [**13\.5.2** Principal Components, Loadings, and Variance Explained](https://shainarace.github.io/LinearAlgebra/pca.html#principal-components-loadings-and-variance-explained)
- [**13\.5.3** Scores and PCA Projection](https://shainarace.github.io/LinearAlgebra/pca.html#scores-and-pca-projection)
- [**13\.5.4** PCA functions in R](https://shainarace.github.io/LinearAlgebra/pca.html#pca-functions-in-r)
- [**13\.5.5** The Biplot](https://shainarace.github.io/LinearAlgebra/pca.html#the-biplot)
- [**13\.6** Variable Clustering with PCA](https://shainarace.github.io/LinearAlgebra/pca.html#variable-clustering-with-pca)
- [**13\.6.1** Correlation PCA](https://shainarace.github.io/LinearAlgebra/pca.html#correlation-pca)
- [**13\.6.2** Which Projection is Better?](https://shainarace.github.io/LinearAlgebra/pca.html#which-projection-is-better)
- [**13\.6.3** Beware of biplots](https://shainarace.github.io/LinearAlgebra/pca.html#beware-of-biplots)
- [**14** Applications of Principal Components](https://shainarace.github.io/LinearAlgebra/pcaapp.html "14 Applications of Principal Components")
- [**14\.1** Dimension reduction](https://shainarace.github.io/LinearAlgebra/pcaapp.html#dimension-reduction)
- [**14\.1.1** Feature Selection](https://shainarace.github.io/LinearAlgebra/pcaapp.html#feature-selection)
- [**14\.1.2** Feature Extraction](https://shainarace.github.io/LinearAlgebra/pcaapp.html#feature-extraction)
- [**14\.2** Exploratory Analysis](https://shainarace.github.io/LinearAlgebra/pcaapp.html#exploratory-analysis)
- [**14\.2.1** UK Food Consumption](https://shainarace.github.io/LinearAlgebra/pcaapp.html#uk-food-consumption)
- [**14\.3** FIFA Soccer Players](https://shainarace.github.io/LinearAlgebra/pcaapp.html#fifa-soccer-players)
- [**14\.4** Cancer Genetics](https://shainarace.github.io/LinearAlgebra/pcaapp.html#cancer-genetics)
- [**14\.4.1** Computing the PCA](https://shainarace.github.io/LinearAlgebra/pcaapp.html#computing-the-pca)
- [**14\.4.2** 3D plot with package](https://shainarace.github.io/LinearAlgebra/pcaapp.html#d-plot-with-package)
- [**14\.4.3** 3D plot with package](https://shainarace.github.io/LinearAlgebra/pcaapp.html#d-plot-with-package-1)
- [**14\.4.4** Variance explained](https://shainarace.github.io/LinearAlgebra/pcaapp.html#variance-explained)
- [**14\.4.5** Using Correlation PCA](https://shainarace.github.io/LinearAlgebra/pcaapp.html#using-correlation-pca)
- [**14\.4.6** Range standardization as an alternative to covariance PCA](https://shainarace.github.io/LinearAlgebra/pcaapp.html#range-standardization-as-an-alternative-to-covariance-pca)
- [**15** The Singular Value Decomposition (SVD)](https://shainarace.github.io/LinearAlgebra/svd.html "15 The Singular Value Decomposition (SVD)")
- [**15\.1** Resolving a Matrix into Components](https://shainarace.github.io/LinearAlgebra/svd.html#resolving-a-matrix-into-components)
- [**15\.2** Data Compression](https://shainarace.github.io/LinearAlgebra/svd.html#data-compression)
- [**15\.3** Noise Reduction](https://shainarace.github.io/LinearAlgebra/svd.html#noise-reduction)
- [**16** Applications of SVD](https://shainarace.github.io/LinearAlgebra/svdapp.html)
- [**16\.1** Text Mining](https://shainarace.github.io/LinearAlgebra/svdapp.html#tm)
- [**16\.1.1** Note About Rows vs. Columns](https://shainarace.github.io/LinearAlgebra/svdapp.html#note-about-rows-vs.-columns)
- [**16\.1.2** Term Weighting](https://shainarace.github.io/LinearAlgebra/svdapp.html#term-weighting)
- [**16\.1.3** Other Considerations](https://shainarace.github.io/LinearAlgebra/svdapp.html#other-considerations)
- [**16\.1.4** Latent Semantic Indexing](https://shainarace.github.io/LinearAlgebra/svdapp.html#latent-semantic-indexing)
- [**16\.1.5** Example](https://shainarace.github.io/LinearAlgebra/svdapp.html#example)
- [**16\.2** Image Compression](https://shainarace.github.io/LinearAlgebra/svdapp.html#rappasvd)
- [**16\.2.1** Image data in R](https://shainarace.github.io/LinearAlgebra/svdapp.html#image-data-in-r)
- [**16\.2.2** Computing the SVD of Dr. Rappa](https://shainarace.github.io/LinearAlgebra/svdapp.html#computing-the-svd-of-dr.-rappa)
- [**16\.2.3** The Noise](https://shainarace.github.io/LinearAlgebra/svdapp.html#the-noise)
- [**17** Factor Analysis](https://shainarace.github.io/LinearAlgebra/fa.html)
- [**17\.1** Assumptions of Factor Analysis](https://shainarace.github.io/LinearAlgebra/fa.html#assumptions-of-factor-analysis)
- [**17\.2** Determining Factorability](https://shainarace.github.io/LinearAlgebra/fa.html#determining-factorability)
- [**17\.2.1** Visual Examination of Correlation Matrix](https://shainarace.github.io/LinearAlgebra/fa.html#visual-examination-of-correlation-matrix)
- [**17\.2.2** Barlett’s Sphericity Test](https://shainarace.github.io/LinearAlgebra/fa.html#barletts-sphericity-test)
- [**17\.2.3** Kaiser-Meyer-Olkin (KMO) Measure of Sampling Adequacy](https://shainarace.github.io/LinearAlgebra/fa.html#kaiser-meyer-olkin-kmo-measure-of-sampling-adequacy)
- [**17\.2.4** Significant factor loadings](https://shainarace.github.io/LinearAlgebra/fa.html#significant-factor-loadings)
- [**17\.3** Communalities](https://shainarace.github.io/LinearAlgebra/fa.html#communalities)
- [**17\.4** Number of Factors](https://shainarace.github.io/LinearAlgebra/fa.html#number-of-factors)
- [**17\.5** Rotation of Factors](https://shainarace.github.io/LinearAlgebra/fa.html#rotation-of-factors)
- [**17\.6** Methods of Factor Analysis](https://shainarace.github.io/LinearAlgebra/fa.html#fa-apps)
- [**17\.6.1** PCA Rotations](https://shainarace.github.io/LinearAlgebra/fa.html#pca-rotations)
- [**17\.7** Case Study: Personality Tests](https://shainarace.github.io/LinearAlgebra/fa.html#case-study-personality-tests)
- [**17\.7.1** Raw PCA Factors](https://shainarace.github.io/LinearAlgebra/fa.html#raw-pca-factors)
- [**17\.7.2** Rotated Principal Components](https://shainarace.github.io/LinearAlgebra/fa.html#rotated-principal-components)
- [**17\.7.3** Visualizing Rotation via BiPlots](https://shainarace.github.io/LinearAlgebra/fa.html#visualizing-rotation-via-biplots)
- [**18** Dimension Reduction for Visualization](https://shainarace.github.io/LinearAlgebra/otherdimred.html "18 Dimension Reduction for Visualization")
- [**18\.1** Multidimensional Scaling](https://shainarace.github.io/LinearAlgebra/otherdimred.html#multidimensional-scaling)
- [**18\.1.1** MDS of Iris Data](https://shainarace.github.io/LinearAlgebra/otherdimred.html#mds-of-iris-data)
- [**18\.1.2** MDS of Leukemia dataset](https://shainarace.github.io/LinearAlgebra/otherdimred.html#mds-of-leukemia-dataset)
- [A note on standardization](https://shainarace.github.io/LinearAlgebra/otherdimred.html#a-note-on-standardization)
- [**19** Social Network Analysis](https://shainarace.github.io/LinearAlgebra/sna.html)
- [**19\.1** Working with Network Data](https://shainarace.github.io/LinearAlgebra/sna.html#working-with-network-data)
- [**19\.2** Network Visualization - `igraph` package](https://shainarace.github.io/LinearAlgebra/sna.html#network-visualization---igraph-package)
- [**19\.2.1** Layout algorithms for `igraph` package](https://shainarace.github.io/LinearAlgebra/sna.html#layout-algorithms-for-igraph-package)
- [**19\.2.2** Adding attribute information to your visualization](https://shainarace.github.io/LinearAlgebra/sna.html#adding-attribute-information-to-your-visualization)
- [**19\.3** Package `networkD3`](https://shainarace.github.io/LinearAlgebra/sna.html#package-networkd3)
- [**19\.3.1** Preparing the data for `networkD3`](https://shainarace.github.io/LinearAlgebra/sna.html#preparing-the-data-for-networkd3)
- [**19\.3.2** Creating an Interactive Visualization with `networkD3`](https://shainarace.github.io/LinearAlgebra/sna.html#creating-an-interactive-visualization-with-networkd3)
- [**19\.3.3** Saving your Interactive Visualization to .html](https://shainarace.github.io/LinearAlgebra/sna.html#saving-your-interactive-visualization-to-.html)
- **Clustering**
- [**20** Introduction](https://shainarace.github.io/LinearAlgebra/clusintro.html)
- [**20\.1** Mathematical Setup](https://shainarace.github.io/LinearAlgebra/clusintro.html#mathematical-setup)
- [**20\.1.1** Data](https://shainarace.github.io/LinearAlgebra/clusintro.html#data)
- [**20\.2** The Number of Clusters, k](https://shainarace.github.io/LinearAlgebra/clusintro.html#the-number-of-clusters-k)
- [**20\.3** Partitioning of Graphs and Networks](https://shainarace.github.io/LinearAlgebra/clusintro.html#partitioning-of-graphs-and-networks)
- [**20\.4** History of Data Clustering](https://shainarace.github.io/LinearAlgebra/clusintro.html#history-of-data-clustering)
- [**21** Algorithms for Data Clustering](https://shainarace.github.io/LinearAlgebra/clusteralgos.html)
- [**21\.1** Hierarchical Algorithms](https://shainarace.github.io/LinearAlgebra/clusteralgos.html#hc)
- [**21\.1.1** Agglomerative Hierarchical Clustering](https://shainarace.github.io/LinearAlgebra/clusteralgos.html#agglomerative-hierarchical-clustering)
- [**21\.1.2** Principal Direction Divisive Partitioning (PDDP)](https://shainarace.github.io/LinearAlgebra/clusteralgos.html#principal-direction-divisive-partitioning-pddp)
- [**21\.2** Iterative Partitional Algorithms](https://shainarace.github.io/LinearAlgebra/clusteralgos.html#kmeanshistory)
- [**21\.2.1** Early Partitional Algorithms](https://shainarace.github.io/LinearAlgebra/clusteralgos.html#early-partitional-algorithms)
- [**21\.2.2** k-means](https://shainarace.github.io/LinearAlgebra/clusteralgos.html#kmeans)
- [**21\.2.3** The Expectation-Maximization (EM) Clustering Algorithm](https://shainarace.github.io/LinearAlgebra/clusteralgos.html#the-expectation-maximization-em-clustering-algorithm)
- [**21\.3** Density Search Algorithms](https://shainarace.github.io/LinearAlgebra/clusteralgos.html#density-search-algorithms)
- [**21\.3.1** Density Based Spacial Clustering of Applications with Noise (DBSCAN)](https://shainarace.github.io/LinearAlgebra/clusteralgos.html#density-based-spacial-clustering-of-applications-with-noise-dbscan)
- [**21\.4** Conclusion](https://shainarace.github.io/LinearAlgebra/clusteralgos.html#conclusion)
- [**22** Algorithms for Graph Partitioning](https://shainarace.github.io/LinearAlgebra/chap1-5.html)
- [**22\.1** Spectral Clustering](https://shainarace.github.io/LinearAlgebra/chap1-5.html#spectral)
- [**22\.2** Fiedler Partitioning](https://shainarace.github.io/LinearAlgebra/chap1-5.html#fiedler-partitioning)
- [**22\.2.1** Linear Algebraic Motivation for the Fiedler vector](https://shainarace.github.io/LinearAlgebra/chap1-5.html#linear-algebraic-motivation-for-the-fiedler-vector)
- [**22\.2.2** Graph Cuts](https://shainarace.github.io/LinearAlgebra/chap1-5.html#graph-cuts)
- [**22\.2.3** Power Iteration Clustering](https://shainarace.github.io/LinearAlgebra/chap1-5.html#pic)
- [**22\.2.4** Clustering via Modularity Maximization](https://shainarace.github.io/LinearAlgebra/chap1-5.html#modularity)
- [**22\.3** Stochastic Clustering](https://shainarace.github.io/LinearAlgebra/chap1-5.html#stochastic-clustering)
- [**22\.3.1** Stochastic Clustering Algorithm (SCA)](https://shainarace.github.io/LinearAlgebra/chap1-5.html#stochastic-clustering-algorithm-sca)
- [**23** Cluster Validation](https://shainarace.github.io/LinearAlgebra/validation.html)
- [**23\.1** Internal Validity Metrics](https://shainarace.github.io/LinearAlgebra/validation.html#internal-validity-metrics)
- [**23\.1.1** Common Measures of Cohesion and Separation](https://shainarace.github.io/LinearAlgebra/validation.html#common-measures-of-cohesion-and-separation)
- [**23\.2** External Validity Metrics](https://shainarace.github.io/LinearAlgebra/validation.html#external)
- [**23\.2.1** Accuracy](https://shainarace.github.io/LinearAlgebra/validation.html#accuracy)
- [**23\.2.2** Entropy](https://shainarace.github.io/LinearAlgebra/validation.html#entropy)
- [**23\.2.3** Purity](https://shainarace.github.io/LinearAlgebra/validation.html#purity)
- [**23\.2.4** Mutual Information (MI) and Normalized Mutual Information (NMI)](https://shainarace.github.io/LinearAlgebra/validation.html#mutual-information-mi-and-normalized-mutual-information-nmi)
- [**23\.2.5** Other External Measures of Validity](https://shainarace.github.io/LinearAlgebra/validation.html#other-external-measures-of-validity)
- [**23\.2.6** Summary Table](https://shainarace.github.io/LinearAlgebra/validation.html#summary-table)
- [**24** Determining the Number of Clusters k](https://shainarace.github.io/LinearAlgebra/findk.html "24 Determining the Number of Clusters \(k\)")
- [**24\.1** Methods based on Cluster Validity (Stopping Rules)](https://shainarace.github.io/LinearAlgebra/findk.html#methods-based-on-cluster-validity-stopping-rules)
- [**24\.2** Sum Squared Error (SSE) Cohesion Plots](https://shainarace.github.io/LinearAlgebra/findk.html#sum-squared-error-sse-cohesion-plots)
- [**24\.2.1** Cosine-Cohesion Plots for Text Data](https://shainarace.github.io/LinearAlgebra/findk.html#cosine-cohesion-plots-for-text-data)
- [**24\.2.2** Ray and Turi’s Method](https://shainarace.github.io/LinearAlgebra/findk.html#ray-and-turis-method)
- [**24\.2.3** The Gap Statistic](https://shainarace.github.io/LinearAlgebra/findk.html#the-gap-statistic)
- [**24\.3** Graph Methods Based on Eigenvalues (Perron Cluster Analysis)](https://shainarace.github.io/LinearAlgebra/findk.html#perroncluster)
- [**25** References](https://shainarace.github.io/LinearAlgebra/references.html)
- [Published with bookdown](https://github.com/rstudio/bookdown)
Facebook
Twitter
LinkedIn
Weibo
Instapaper
A
A
Serif
Sans
White
Sepia
Night
# [Linear Algebra for Data Science](https://shainarace.github.io/LinearAlgebra/)
# Chapter 12 Eigenvalues and Eigenvectors
Eigenvalues and eigenvectors are (scalar, vector)-pairs that form the “essence” of a matrix. The prefix eigen- is adopted from the German word *eigen* which means “characteristic, inherent, own” and was introduced by David Hilbert in 1904, but the study of these characteristic directions and magnitudes dates back to Euler’s study of the rotational motion of rigid bodies in the 18 t h century.
**Definition 12.1 (Eigenvalues and Eigenvectors)** For a square matrix A n × n, a scalar λ is called an **eigenvalue** of A if there is a nonzero vector x such that A x \= λ x . Such a vector, x is called an **eigenvector** of A corresponding to the **eigenvalue** λ. We sometimes refer to the pair ( λ , x ) as an **eigenpair.**
Eigenvalues and eigenvectors have numerous applications from graphic design to quantum mechanics to geology to epidemiology. The main application of note for data scientists is Principal Component Analysis, but we will also see eigenvalue equations used in social network analysis to determine important players in a network and to detect communities in the network. Before we dive into those applications, let’s first get a handle on the definition by exploring some examples.
**Example 12.1 (Eigenvalues and Eigenvectors)** Determine whether x \= ( 1 1 ) is an eigenvector of A \= ( 3 1 1 3 ) and if so, find the corresponding eigenvalue.\\ To determine whether x is an eigenvector, we want to compute A x and observe whether the result is a multiple of x. If this is the case, then the multiplication factor is the corresponding eigenvalue: A x \= ( 3 1 1 3 ) ( 1 1 ) \= ( 4 4 ) \= 4 ( 1 1 ) From this it follows that x *is* an eigenvector of A and the corresponding eigenvalue is λ \= 4.\\
Is the vector y \= ( 2 2 ) an eigenvector? A y \= ( 3 1 1 3 ) ( 2 2 ) \= ( 8 8 ) \= 4 ( 2 2 ) \= 4 y Yes, it is and it corresponds to the *same* eigenvalue, λ \= 4
Example [12\.1](https://shainarace.github.io/LinearAlgebra/eigen.html#exm:eig1) shows a very important property of eigenvalue-eigenvector pairs. If ( λ , x ) is an eigenpair then any scalar multiple of x is also an eigenvector corresponding to λ. To see this, let ( λ , x ) be an eigenpair for a matrix A (which means that A x \= λ x) and let y \= α x be any scalar multiple of x. Then we have, A y \= A ( α x ) \= α ( A x ) \= α ( λ x ) \= λ ( α x ) \= λ y which shows that y (or any scalar multiple of x) is also an eigenvector associated with the eigenvalue λ.
Thus, for each eigenvalue we have infinitely many eigenvectors. In the preceding example, the eigenvectors associated with λ \= 4 will be scalar multiples of x \= ( 1 1 ). You may recall from Chapter [7](https://shainarace.github.io/LinearAlgebra/linind.html#linind) that the set of all scalar multiples of x is denoted s p a n ( x ). The s p a n ( x ) in this example represents the **eigenspace** of λ. *Note: when using software to compute eigenvectors, it is standard practice for the software to provide the normalized/unit eigenvector.*
In some situations, an eigenvalue can have multiple eigenvectors which are linearly independent. The number of linearly independent eigenvectors associated with an eigenvalue is called the **geometric multiplicity** of the eigenvalue. Example [12\.2](https://shainarace.github.io/LinearAlgebra/eigen.html#exm:eig2) clarifies this concept.
**Example 12.2 (Geometric Multiplicity)** Consider the matrix A \= ( 3 0 0 3 ). It should be straightforward to see that x 1 \= ( 1 0 ) and x 2 \= ( 0 1 ) are both eigenvectors corresponding to the eigenvalue λ \= 3. x 1 and x 2 are linearly independent, therefore the geometric multiplicity of λ \= 3 is 2.
What happens if we take a linear combination of x 1 and x 2? Is that also an eigenvector? Consider y \= ( 2 3 ) \= 2 x 1 \+ 3 x 2. Then A y \= ( 3 0 0 3 ) ( 2 3 ) \= ( 6 9 ) \= 3 ( 2 3 ) \= 3 y shows that y is also an eigenvector associated with λ \= 3.
The **eigenspace** corresponding to λ \= 3 is the set of all linear combinations of x 1 and x 2, i.e. the s p a n ( x 1 , x 2 ).
We can generalize the result that we saw in Example [12\.2](https://shainarace.github.io/LinearAlgebra/eigen.html#exm:eig2) for any square matrix and any geometric multiplicity. Let A n × n have an eigenvalue λ with geometric multiplicity k. This means there are k linearly independent eigenvectors, x 1 , x 2 , … , x k such that A x i \= λ x i for each eigenvector x i. Now if we let y be a vector in the s p a n ( x 1 , x 2 , … , x k ) then y is some linear combination of the x i’s: y \= α 1 x 2 \+ α 2 x 2 \+ ⋯ \+ α k x k Observe what happens when we multiply y by A: A y \= A ( α 1 x 2 \+ α 2 x 2 \+ ⋯ \+ α k x k ) \= α 1 ( A x 1 ) \+ α 2 ( A x 2 ) \+ ⋯ \+ α k ( A x k ) \= α 1 ( λ x 1 ) \+ α 2 ( λ x 2 ) \+ ⋯ \+ α k ( λ x k ) \= λ ( α 1 x 2 \+ α 2 x 2 \+ ⋯ \+ α k x k ) \= λ y which shows that y (or any vector in the s p a n ( x 1 , x 2 , … , x k )) is an eigenvector of A corresponding to λ.
This proof allows us to formally define the concept of an eigenspace.
**Definition 12.2 (Eigenspace)** Let A be a square matrix and let λ be an eigenvalue of A. The set of all eigenvectors corresponding to λ, together with the zero vector, is called the **eigenspace** of λ. The number of basis vectors required to form the eigenspace is called the **geometric multiplicity** of λ.
Now, let’s attempt the eigenvalue problem from the other side. Given an eigenvalue, we will find the corresponding eigenspace in Example [12\.3](https://shainarace.github.io/LinearAlgebra/eigen.html#exm:eig3).
**Example 12.3 (Eigenvalues and Eigenvectors)** Show that λ \= 5 is an eigenvalue of A \= ( 1 2 4 3 ) and determine the eigenspace of λ \= 5.
Attempting the problem from this angle requires slightly more work. We want to find a vector x such that A x \= 5 x. Setting this up, we have: A x \= 5 x . What we want to do is move both terms to one side and factor out the vector x. In order to do this, we must use an identity matrix, otherwise the equation wouldn’t make sense (we’d be subtracting a constant from a matrix). A x − 5 x \= 0 ( A − 5 I ) x \= 0 ( ( 1 2 4 3 ) − ( 5 0 0 5 ) ) ( x 1 x 2 ) \= ( 0 0 ) ( − 4 2 4 − 2 ) ( x 1 x 2 ) \= ( 0 0 ) Clearly, the matrix A − λ I is singular (i.e. does not have linearly independent rows/columns). This will always be the case by the definition A x \= λ x, and is often used as an alternative definition.\\ In order to solve this homogeneous system of equations, we use Gaussian elimination: ( − 4 2 0 4 − 2 0 ) ⟶ ( 1 − 1 2 0 0 0 0 ) This implies that any vector x for which x 1 − 1 2 x 2 \= 0 satisfies the eigenvector equation. We can pick any such vector, for example x \= ( 1 2 ), and say that the eigenspace of λ \= 5 is s p a n { ( 1 2 ) }
If we didn’t know either an eigenvalue or eigenvector of A and instead wanted to find both, we would first find eigenvalues by determining all possible λ such that A − λ I is singular and then find the associated eigenvectors. There are some tricks which allow us to do this by hand for 2 × 2 and 3 × 3 matrices, but after that the computation time is unworthy of the effort. Now that we have a good understanding of how to interpret eigenvalues and eigenvectors algebraically, let’s take a look at some of the things that they can do, starting with one important fact.
**Definition 12.3 (Eigenvalues and the Trace of a Matrix)** Let A be an n × n matrix with eigenvalues λ 1 , λ 2 , … , λ n. Then the sum of the eigenvalues is equal to the trace of the matrix (recall that the trace of a matrix is the sum of its diagonal elements). T r a c e ( A ) \= ∑ i \= 1 n λ i .
**Example 12.4 (Trace of Covariance Matrix)** Suppose that we had a collection of n observations on p variables, x 1 , x 2 , … , x p. After centering the data to have zero mean, we can compute the sample variances as: v a r ( x i ) \= 1 n − 1 x i T x i \= ‖ x i ‖ 2 These variances form the diagonal elements of the sample covariance matrix, Σ \= 1 n − 1 X T X Thus, the total variance of this data is 1 n − 1 ∑ i \= 1 n ‖ x i ‖ 2 \= T r a c e ( Σ ) \= ∑ i \= 1 n λ i .
In other words, the sum of the eigenvalues of a covariance matrix provides the total variance in the variables x 1 , … , x p.
## 12\.1 Diagonalization
Let’s take another look at Example [12\.3](https://shainarace.github.io/LinearAlgebra/eigen.html#exm:eig3). We already showed that λ 1 \= 5 and v 1 \= ( 1 2 ) is an eigenpair for the matrix A \= ( 1 2 4 3 ). You may verify that λ 2 \= − 1 and v 2 \= ( 1 − 1 ) is another eigenpair. Suppose we create a matrix of eigenvectors: V \= ( v 1 , v 2 ) \= ( 1 1 2 − 1 ) and a diagonal matrix containing the corresponding eigenvalues: D \= ( 5 0 0 − 1 ) Then it is easy to verify that A V \= V D: A V \= ( 1 2 4 3 ) ( 1 1 2 − 1 ) \= ( 5 − 1 10 1 ) \= ( 1 1 2 − 1 ) ( 5 0 0 − 1 ) \= V D If the columns of V are linearly independent, which they are in this case, we can write: V − 1 A V \= D
What we have just done is develop a way to transform a matrix A into a diagonal matrix D. This is known as **diagonalization.**
**Definition 12.4 (Diagonalizable)** An n × n matrix A is said to be **diagonalizable** if there exists an invertible matrix P and a diagonal matrix D such that P − 1 A P \= D This is possible if and only if the matrix A has n linearly independent eigenvectors (known as a *complete set of eigenvectors*). The matrix P is then the matrix of eigenvectors and the matrix D contains the corresponding eigenvalues on the diagonal.
Determining whether or not a matrix A n × n is diagonalizable is a little tricky. Having r a n k ( A ) \= n is *not* a sufficient condition for having n linearly independent eigenvectors. The following matrix stands as a counter example: A \= ( − 3 1 − 3 20 3 10 2 − 2 4 ) This matrix has full rank but only two linearly independent eigenvectors. Fortunately, for our primary application of diagonalization, we will be dealing with a symmetric matrix, which can always be diagonalized. In fact, symmetric matrices have an additional property which makes this diagonalization particularly nice, as we will see in Chapter [13](https://shainarace.github.io/LinearAlgebra/pca.html#pca).
## 12\.2 Geometric Interpretation of Eigenvalues and Eigenvectors
Since any scalar multiple of an eigenvector is still an eigenvector, let’s consider for the present discussion unit eigenvectors x of a square matrix A - those with length ‖ x ‖ \= 1. By the definition, we know that A x \= λ x We know that geometrically, if we multiply x by A, the resulting vector points in the same direction as x. Geometrically, it turns out that multiplying the unit circle or unit sphere by a matrix A carves out an ellipse, or an ellipsoid. We can see eigenvectors visually by watching how multiplication by a matrix A changes the unit vectors. Figure [12\.1](https://shainarace.github.io/LinearAlgebra/eigen.html#fig:eigenarrows) illustrates this. The blue arrows represent (a sampling of) the unit circle, all vectors x for which ‖ x ‖ \= 1. The red arrows represent the image of the blue arrows after multiplication by A, or A x for each vector x. We can see how almost every vector changes direction when multiplied by A, except the eigenvector directions which are marked in black. Such a picture provides a nice geometrical interpretation of eigenvectors for a general matrix, but we will see in Chapter [13](https://shainarace.github.io/LinearAlgebra/pca.html#pca) just how powerful these eigenvector directions are when we look at symmetric matrix.
Figure 12.1: Visualizing eigenvectors (in black) using the image (in red) of the unit sphere (in blue) after multiplication by A.
## 12\.3 Exercises
1. Show that
v
is an eigenvector of
A
and find the corresponding eigenvalue:
1. \$= & 2 \\2 & 1 \\-3 \$
2. \$= & 1 \\6 & 0 \\-2 \$
3. \$= & -2 \\5 & -7 \\2 \$
2. Show that
λ
is an eigenvalue of
A
and list two eigenvectors corresponding to this eigenvalue:
1. A
\=
(
0
4
−
1
5
)
λ
\=
4
2. A
\=
(
0
4
−
1
5
)
λ
\=
1
3. Based on the eigenvectors you found in exercises 2, can the matrix
A
be diagonalized? Why or why not? If diagonalization is possible, explain how it would be done.
4. Can a rectangular matrix have eigenvalues/eigenvectors? |
| Readable Markdown | Eigenvalues and eigenvectors are (scalar, vector)-pairs that form the “essence” of a matrix. The prefix eigen- is adopted from the German word *eigen* which means “characteristic, inherent, own” and was introduced by David Hilbert in 1904, but the study of these characteristic directions and magnitudes dates back to Euler’s study of the rotational motion of rigid bodies in the 18 t h century.
**Definition 12.1 (Eigenvalues and Eigenvectors)** For a square matrix A n × n, a scalar λ is called an **eigenvalue** of A if there is a nonzero vector x such that A x \= λ x . Such a vector, x is called an **eigenvector** of A corresponding to the **eigenvalue** λ. We sometimes refer to the pair ( λ , x ) as an **eigenpair.**
Eigenvalues and eigenvectors have numerous applications from graphic design to quantum mechanics to geology to epidemiology. The main application of note for data scientists is Principal Component Analysis, but we will also see eigenvalue equations used in social network analysis to determine important players in a network and to detect communities in the network. Before we dive into those applications, let’s first get a handle on the definition by exploring some examples.
**Example 12.1 (Eigenvalues and Eigenvectors)** Determine whether x \= ( 1 1 ) is an eigenvector of A \= ( 3 1 1 3 ) and if so, find the corresponding eigenvalue.\\ To determine whether x is an eigenvector, we want to compute A x and observe whether the result is a multiple of x. If this is the case, then the multiplication factor is the corresponding eigenvalue: A x \= ( 3 1 1 3 ) ( 1 1 ) \= ( 4 4 ) \= 4 ( 1 1 ) From this it follows that x *is* an eigenvector of A and the corresponding eigenvalue is λ \= 4.\\
Is the vector y \= ( 2 2 ) an eigenvector? A y \= ( 3 1 1 3 ) ( 2 2 ) \= ( 8 8 ) \= 4 ( 2 2 ) \= 4 y Yes, it is and it corresponds to the *same* eigenvalue, λ \= 4
Example [12\.1](https://shainarace.github.io/LinearAlgebra/eigen.html#exm:eig1) shows a very important property of eigenvalue-eigenvector pairs. If ( λ , x ) is an eigenpair then any scalar multiple of x is also an eigenvector corresponding to λ. To see this, let ( λ , x ) be an eigenpair for a matrix A (which means that A x \= λ x) and let y \= α x be any scalar multiple of x. Then we have, A y \= A ( α x ) \= α ( A x ) \= α ( λ x ) \= λ ( α x ) \= λ y which shows that y (or any scalar multiple of x) is also an eigenvector associated with the eigenvalue λ.
Thus, for each eigenvalue we have infinitely many eigenvectors. In the preceding example, the eigenvectors associated with λ \= 4 will be scalar multiples of x \= ( 1 1 ). You may recall from Chapter [7](https://shainarace.github.io/LinearAlgebra/linind.html#linind) that the set of all scalar multiples of x is denoted s p a n ( x ). The s p a n ( x ) in this example represents the **eigenspace** of λ. *Note: when using software to compute eigenvectors, it is standard practice for the software to provide the normalized/unit eigenvector.*
In some situations, an eigenvalue can have multiple eigenvectors which are linearly independent. The number of linearly independent eigenvectors associated with an eigenvalue is called the **geometric multiplicity** of the eigenvalue. Example [12\.2](https://shainarace.github.io/LinearAlgebra/eigen.html#exm:eig2) clarifies this concept.
**Example 12.2 (Geometric Multiplicity)** Consider the matrix A \= ( 3 0 0 3 ). It should be straightforward to see that x 1 \= ( 1 0 ) and x 2 \= ( 0 1 ) are both eigenvectors corresponding to the eigenvalue λ \= 3. x 1 and x 2 are linearly independent, therefore the geometric multiplicity of λ \= 3 is 2.
What happens if we take a linear combination of x 1 and x 2? Is that also an eigenvector? Consider y \= ( 2 3 ) \= 2 x 1 \+ 3 x 2. Then A y \= ( 3 0 0 3 ) ( 2 3 ) \= ( 6 9 ) \= 3 ( 2 3 ) \= 3 y shows that y is also an eigenvector associated with λ \= 3.
The **eigenspace** corresponding to λ \= 3 is the set of all linear combinations of x 1 and x 2, i.e. the s p a n ( x 1 , x 2 ).
We can generalize the result that we saw in Example [12\.2](https://shainarace.github.io/LinearAlgebra/eigen.html#exm:eig2) for any square matrix and any geometric multiplicity. Let A n × n have an eigenvalue λ with geometric multiplicity k. This means there are k linearly independent eigenvectors, x 1 , x 2 , … , x k such that A x i \= λ x i for each eigenvector x i. Now if we let y be a vector in the s p a n ( x 1 , x 2 , … , x k ) then y is some linear combination of the x i’s: y \= α 1 x 2 \+ α 2 x 2 \+ ⋯ \+ α k x k Observe what happens when we multiply y by A: A y \= A ( α 1 x 2 \+ α 2 x 2 \+ ⋯ \+ α k x k ) \= α 1 ( A x 1 ) \+ α 2 ( A x 2 ) \+ ⋯ \+ α k ( A x k ) \= α 1 ( λ x 1 ) \+ α 2 ( λ x 2 ) \+ ⋯ \+ α k ( λ x k ) \= λ ( α 1 x 2 \+ α 2 x 2 \+ ⋯ \+ α k x k ) \= λ y which shows that y (or any vector in the s p a n ( x 1 , x 2 , … , x k )) is an eigenvector of A corresponding to λ.
This proof allows us to formally define the concept of an eigenspace.
**Definition 12.2 (Eigenspace)** Let A be a square matrix and let λ be an eigenvalue of A. The set of all eigenvectors corresponding to λ, together with the zero vector, is called the **eigenspace** of λ. The number of basis vectors required to form the eigenspace is called the **geometric multiplicity** of λ.
Now, let’s attempt the eigenvalue problem from the other side. Given an eigenvalue, we will find the corresponding eigenspace in Example [12\.3](https://shainarace.github.io/LinearAlgebra/eigen.html#exm:eig3).
**Example 12.3 (Eigenvalues and Eigenvectors)** Show that λ \= 5 is an eigenvalue of A \= ( 1 2 4 3 ) and determine the eigenspace of λ \= 5.
Attempting the problem from this angle requires slightly more work. We want to find a vector x such that A x \= 5 x. Setting this up, we have: A x \= 5 x . What we want to do is move both terms to one side and factor out the vector x. In order to do this, we must use an identity matrix, otherwise the equation wouldn’t make sense (we’d be subtracting a constant from a matrix). A x − 5 x \= 0 ( A − 5 I ) x \= 0 ( ( 1 2 4 3 ) − ( 5 0 0 5 ) ) ( x 1 x 2 ) \= ( 0 0 ) ( − 4 2 4 − 2 ) ( x 1 x 2 ) \= ( 0 0 ) Clearly, the matrix A − λ I is singular (i.e. does not have linearly independent rows/columns). This will always be the case by the definition A x \= λ x, and is often used as an alternative definition.\\ In order to solve this homogeneous system of equations, we use Gaussian elimination: ( − 4 2 0 4 − 2 0 ) ⟶ ( 1 − 1 2 0 0 0 0 ) This implies that any vector x for which x 1 − 1 2 x 2 \= 0 satisfies the eigenvector equation. We can pick any such vector, for example x \= ( 1 2 ), and say that the eigenspace of λ \= 5 is s p a n { ( 1 2 ) }
If we didn’t know either an eigenvalue or eigenvector of A and instead wanted to find both, we would first find eigenvalues by determining all possible λ such that A − λ I is singular and then find the associated eigenvectors. There are some tricks which allow us to do this by hand for 2 × 2 and 3 × 3 matrices, but after that the computation time is unworthy of the effort. Now that we have a good understanding of how to interpret eigenvalues and eigenvectors algebraically, let’s take a look at some of the things that they can do, starting with one important fact.
**Definition 12.3 (Eigenvalues and the Trace of a Matrix)** Let A be an n × n matrix with eigenvalues λ 1 , λ 2 , … , λ n. Then the sum of the eigenvalues is equal to the trace of the matrix (recall that the trace of a matrix is the sum of its diagonal elements). T r a c e ( A ) \= ∑ i \= 1 n λ i .
**Example 12.4 (Trace of Covariance Matrix)** Suppose that we had a collection of n observations on p variables, x 1 , x 2 , … , x p. After centering the data to have zero mean, we can compute the sample variances as: v a r ( x i ) \= 1 n − 1 x i T x i \= ‖ x i ‖ 2 These variances form the diagonal elements of the sample covariance matrix, Σ \= 1 n − 1 X T X Thus, the total variance of this data is 1 n − 1 ∑ i \= 1 n ‖ x i ‖ 2 \= T r a c e ( Σ ) \= ∑ i \= 1 n λ i .
In other words, the sum of the eigenvalues of a covariance matrix provides the total variance in the variables x 1 , … , x p.
## Diagonalization
Let’s take another look at Example [12\.3](https://shainarace.github.io/LinearAlgebra/eigen.html#exm:eig3). We already showed that λ 1 \= 5 and v 1 \= ( 1 2 ) is an eigenpair for the matrix A \= ( 1 2 4 3 ). You may verify that λ 2 \= − 1 and v 2 \= ( 1 − 1 ) is another eigenpair. Suppose we create a matrix of eigenvectors: V \= ( v 1 , v 2 ) \= ( 1 1 2 − 1 ) and a diagonal matrix containing the corresponding eigenvalues: D \= ( 5 0 0 − 1 ) Then it is easy to verify that A V \= V D: A V \= ( 1 2 4 3 ) ( 1 1 2 − 1 ) \= ( 5 − 1 10 1 ) \= ( 1 1 2 − 1 ) ( 5 0 0 − 1 ) \= V D If the columns of V are linearly independent, which they are in this case, we can write: V − 1 A V \= D
What we have just done is develop a way to transform a matrix A into a diagonal matrix D. This is known as **diagonalization.**
**Definition 12.4 (Diagonalizable)** An n × n matrix A is said to be **diagonalizable** if there exists an invertible matrix P and a diagonal matrix D such that P − 1 A P \= D This is possible if and only if the matrix A has n linearly independent eigenvectors (known as a *complete set of eigenvectors*). The matrix P is then the matrix of eigenvectors and the matrix D contains the corresponding eigenvalues on the diagonal.
Determining whether or not a matrix A n × n is diagonalizable is a little tricky. Having r a n k ( A ) \= n is *not* a sufficient condition for having n linearly independent eigenvectors. The following matrix stands as a counter example: A \= ( − 3 1 − 3 20 3 10 2 − 2 4 ) This matrix has full rank but only two linearly independent eigenvectors. Fortunately, for our primary application of diagonalization, we will be dealing with a symmetric matrix, which can always be diagonalized. In fact, symmetric matrices have an additional property which makes this diagonalization particularly nice, as we will see in Chapter [13](https://shainarace.github.io/LinearAlgebra/pca.html#pca).
## Geometric Interpretation of Eigenvalues and Eigenvectors
Since any scalar multiple of an eigenvector is still an eigenvector, let’s consider for the present discussion unit eigenvectors x of a square matrix A - those with length ‖ x ‖ \= 1. By the definition, we know that A x \= λ x We know that geometrically, if we multiply x by A, the resulting vector points in the same direction as x. Geometrically, it turns out that multiplying the unit circle or unit sphere by a matrix A carves out an ellipse, or an ellipsoid. We can see eigenvectors visually by watching how multiplication by a matrix A changes the unit vectors. Figure [12\.1](https://shainarace.github.io/LinearAlgebra/eigen.html#fig:eigenarrows) illustrates this. The blue arrows represent (a sampling of) the unit circle, all vectors x for which ‖ x ‖ \= 1. The red arrows represent the image of the blue arrows after multiplication by A, or A x for each vector x. We can see how almost every vector changes direction when multiplied by A, except the eigenvector directions which are marked in black. Such a picture provides a nice geometrical interpretation of eigenvectors for a general matrix, but we will see in Chapter [13](https://shainarace.github.io/LinearAlgebra/pca.html#pca) just how powerful these eigenvector directions are when we look at symmetric matrix.
Figure 12.1: Visualizing eigenvectors (in black) using the image (in red) of the unit sphere (in blue) after multiplication by A.
## Exercises
1. Show that
v
is an eigenvector of
A
and find the corresponding eigenvalue:
1. \$= & 2 \\2 & 1 \\-3 \$
2. \$= & 1 \\6 & 0 \\-2 \$
3. \$= & -2 \\5 & -7 \\2 \$
2. Show that
λ
is an eigenvalue of
A
and list two eigenvectors corresponding to this eigenvalue:
1. A
\=
(
0
4
−
1
5
)
λ
\=
4
2. A
\=
(
0
4
−
1
5
)
λ
\=
1
3. Based on the eigenvectors you found in exercises 2, can the matrix
A
be diagonalized? Why or why not? If diagonalization is possible, explain how it would be done.
4. Can a rectangular matrix have eigenvalues/eigenvectors? |
| Shard | 143 (laksa) |
| Root Hash | 2566890010099092343 |
| Unparsed URL | io,github!shainarace,/LinearAlgebra/eigen.html s443 |