🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 45 (from laksa171)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa045.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/\')), getAhrefsUnparsedNoserviceFromURL(\'https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/","crawl_time":1776172595,"first_indexed_time":1704144788,"http_code":200,"src_unparsed":"pub,pressbooks!\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/ s443","src_root_hash":"2282000291418685645","history_drop_reason":null,"meta_title":"ORDINARY LEAST-SQUARES – Linear Algebra and Applications","meta_descriptions":[],"attrs_boilerpipe_text":"The Ordinary Least-Squares (OLS, or LS) problem is defined as\n \n \nwhere\nare given. Together, the pair\nis referred to as the\nproblem data\n. The vector\nis often referred to as the ‘‘measurement” or “output” vector, and the data matrix\nas the ‘‘design‘‘ or ‘‘input‘‘ matrix. The vector\nis referred to as the\nresidual error\n vector.\nNote that the problem is equivalent to one where the norm is not squared. Taking the squares is done for the convenience of the solution.\nInterpretation as projection on the range\nSee also\n:\nImage compression via least-squares.\nInterpretation as minimum distance to feasibility\nThe OLS problem is usually applied to problems where the linear\n is not \nfeasible\n, that is, there is no solution to\n.\nThe OLS can be interpreted as finding the smallest (in Euclidean norm sense) perturbation of the right-hand side,\n, such that the linear equation\n \n \nbecomes feasible. In this sense, the OLS formulation implicitly assumes that the data matrix\n of the problem is known exactly, while only the right-hand side is subject to perturbation, or measurement errors. A more elaborate model, \ntotal least-squares\n, takes into account errors in both\nand\n.\nInterpretation as regression\nWe can also interpret the problem in terms of the rows of\n, as follows. Assume that\n, where\nis the\n-th row of\n. The problem reads\n \n \nIn this sense, we are trying to fit each component of\nas a linear combination of the corresponding input\n, with\nas the coefficients of this linear combination.\nSee also:\nLinear regression.\nAuto-regressive models for time series prediction\n.\nPower law model fitting.\nAssume that the matrix\nis tall (\n) and full column rank. Then the solution to the problem is unique and given by\n \n \nThis can be seen by simply taking the \ngradient\n(vector of derivatives) of the objective function, which leads to the optimality condition\n. Geometrically, the residual vector\nis orthogonal to the span of the columns of\n, as seen in the picture above.\nWe can also prove this via the \nQR decomposition\nof the matrix\nwith\na\nmatrix with orthonormal columns (\n) and\na\nupper-triangular, invertible matrix. Noting that\n \n \nand exploiting the fact that\nis invertible, we obtain the optimal solution\n. This is the same as the formula above, since\n \n \nThus, to find the solution based on the QR decomposition, we just need to implement two steps:\nRotate the output vector: set\n.\nSolve the triangular system\n by \nbackward substitution\n.\nRecall that the \noptimal set\nof a minimization problem is its set of minimizers. For least-squares problems, the optimal set is an affine set, which reduces to a singleton when\n is full column rank.\nIn the general case (\nis not necessarily tall, and \/or not full rank) then the solution may not be unique. If\nis a particular solution, then\nis also a solution, if\nis such that\n, that is,\n. That is, the nullspace of\ndescribes the \nambiguity\n of solutions. In mathematical terms:\n \n \nThe formal expression for the set of minimizers to the least-squares problem can be found again via the QR decomposition. This is shown \nhere\n.","attrs_markdown":"\"\n\n[Skip to content](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/#content)\n\nMenu\n\nPrimary Navigation\n\n- [Home](https:\/\/pressbooks.pub\/linearalgebraandapplications)\n- [Read](https:\/\/pressbooks.pub\/linearalgebraandapplications\/front-matter\/introduction\/)\n- [Buy](https:\/\/pressbooks.pub\/linearalgebraandapplications\/buy\/)\n- [Sign in](https:\/\/pressbooks.pub\/linearalgebraandapplications\/wp-login.php?redirect_to=https%3A%2F%2Fpressbooks.pub%2Flinearalgebraandapplications%2Fchapter%2Fordinary-least-squares%2F)\n\nWant to create or adapt books like this? [Learn more](https:\/\/pressbooks.com\/adapt-open-textbooks?utm_source=book&utm_medium=banner&utm_campaign=bbc) about how Pressbooks supports open publishing practices.\n\nBook Contents Navigation\n\nContents\n\n1. [INTRODUCTION](https:\/\/pressbooks.pub\/linearalgebraandapplications\/front-matter\/introduction\/)\n2. [I. VECTORS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/part\/main-body\/)\n   1. [1\\. BASICS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/basics\/)\n      1. [1\\.1. Definitions](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/basics\/#chapter-26-section-1)\n      2. [1\\.2. Independence](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/basics\/#chapter-26-section-2)\n      3. [1\\.3. Subspace, span, affine sets](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/basics\/#chapter-26-section-3)\n      4. [1\\.4. Basis, dimension](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/basics\/#chapter-26-section-4)\n   2. [2\\. SCALAR PRODUCT, NORMS AND ANGLES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/scalar-product-norms-and-angles\/)\n      1. [2\\.1. Scalar product](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/scalar-product-norms-and-angles\/#chapter-36-section-1)\n      2. [2\\.2. Norms](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/scalar-product-norms-and-angles\/#chapter-36-section-2)\n      3. [2\\.3. Three popular norms](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/scalar-product-norms-and-angles\/#chapter-36-section-3)\n      4. [2\\.4. Cauchy-Schwarz inequality](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/scalar-product-norms-and-angles\/#chapter-36-section-4)\n      5. [2\\.5. Angles between vectors](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/scalar-product-norms-and-angles\/#chapter-36-section-5)\n   3. [3\\. PROJECTION ON A LINE](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/projection-on-a-line\/)\n      1. [3\\.1. Definition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/projection-on-a-line\/#chapter-37-section-1)\n      2. [3\\.2. Closed-form expression](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/projection-on-a-line\/#chapter-37-section-2)\n      3. [3\\.3. Interpreting the scalar product](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/projection-on-a-line\/#chapter-37-section-3)\n   4. [4\\. ORTHOGONALIZATION: THE GRAM-SCHMIDT PROCEDURE](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/orthogonalization-the-gram-schmidt-procedure\/)\n      1. [4\\.1. What is orthogonalization?](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/orthogonalization-the-gram-schmidt-procedure\/#chapter-39-section-1)\n      2. [4\\.2. Basic step: projection on a line](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/orthogonalization-the-gram-schmidt-procedure\/#chapter-39-section-2)\n      3. [4\\.3. Gram-Schmidt procedure](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/orthogonalization-the-gram-schmidt-procedure\/#chapter-39-section-3)\n   5. [5\\. HYPERPLANES AND HALF-SPACES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/hyperplanes-and-half-spaces\/)\n      1. [5\\.1. Hyperplanes](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/hyperplanes-and-half-spaces\/#chapter-41-section-1)\n      2. [5\\.2. Projection on a hyperplane](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/hyperplanes-and-half-spaces\/#chapter-41-section-2)\n      3. [5\\.3. Geometry of hyperplanes](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/hyperplanes-and-half-spaces\/#chapter-41-section-3)\n      4. [5\\.4. Half-spaces](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/hyperplanes-and-half-spaces\/#chapter-41-section-4)\n   6. [6\\. LINEAR FUNCTIONS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/linear-functions\/)\n      1. [6\\.1. Linear and affine functions](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/linear-functions\/#chapter-42-section-1)\n      2. [6\\.2. First-order approximation of non-linear functions](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/linear-functions\/#chapter-42-section-2)\n      3. [6\\.3. Other sources of linear models](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/linear-functions\/#chapter-42-section-3)\n   7. [7\\. APPLICATION: DATA VISUALIZATION BY PROJECTION ON A LINE](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/application-data-visualization-by-projection-on-a-line\/)\n      1. [7\\.1. Senate voting data](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/application-data-visualization-by-projection-on-a-line\/#chapter-43-section-1)\n      2. [7\\.2. Visualization of high-dimensional data via projection](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/application-data-visualization-by-projection-on-a-line\/#chapter-43-section-2)\n      3. [7\\.3. Examples](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/application-data-visualization-by-projection-on-a-line\/#chapter-43-section-3)\n   8. [8\\. EXERCISES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises\/)\n      1. [8\\.1. Subspaces](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises\/#chapter-44-section-1)\n      2. [8\\.2. Projections, scalar product, angles](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises\/#chapter-44-section-2)\n      3. [8\\.3. Orthogonalization](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises\/#chapter-44-section-3)\n      4. [8\\.4. Generalized Cauchy-Schwarz inequalities](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises\/#chapter-44-section-4)\n      5. [8\\.5. Linear functions](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises\/#chapter-44-section-5)\n3. [II. MATRICES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/part\/chapter-ii-matrices\/)\n   1. [9\\. BASICS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/basics-2\/)\n      1. [9\\.1. Matrices as collections of column vectors](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/basics-2\/#chapter-47-section-1)\n      2. [9\\.2. Transpose](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/basics-2\/#chapter-47-section-2)\n      3. [9\\.3. Matrices as collections of rows](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/basics-2\/#chapter-47-section-3)\n      4. [9\\.4. Sparse Matrices](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/basics-2\/#chapter-47-section-4)\n   2. [10\\. MATRIX-VECTOR AND MATRIX-MATRIX MULTIPLICATION, SCALAR PRODUCT](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-vector-and-matrix-matrix-multiplication-scalar-product\/)\n      1. [10\\.1. Matrix-vector product](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-vector-and-matrix-matrix-multiplication-scalar-product\/#chapter-48-section-1)\n      2. [10\\.2. Matrix-matrix product](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-vector-and-matrix-matrix-multiplication-scalar-product\/#chapter-48-section-2)\n      3. [10\\.3. Block Matrix Products](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-vector-and-matrix-matrix-multiplication-scalar-product\/#chapter-48-section-3)\n      4. [10\\.4. Trace, scalar product](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-vector-and-matrix-matrix-multiplication-scalar-product\/#chapter-48-section-4)\n   3. [11\\. SPECIAL CLASSES OF MATRICES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/special-classes-of-matrices\/)\n      1. [11\\.1 Some special square matrices](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/special-classes-of-matrices\/#chapter-49-section-1)\n      2. [11\\.2 Dyads](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/special-classes-of-matrices\/#chapter-49-section-2)\n   4. [12\\. QR DECOMPOSITION OF A MATRIX](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/qr-decomposition-of-a-matrix\/)\n      1. [12\\.1 Basic idea](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/qr-decomposition-of-a-matrix\/#chapter-50-section-1)\n      2. [12\\.2 Case being full column rank](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/qr-decomposition-of-a-matrix\/#chapter-50-section-2)\n      3. [12\\.3 Case when the columns are not independent](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/qr-decomposition-of-a-matrix\/#chapter-50-section-3)\n      4. [12\\.4 Full QR decomposition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/qr-decomposition-of-a-matrix\/#chapter-50-section-4)\n   5. [13\\. MATRIX INVERSES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-inverses\/)\n      1. [13\\.1 Square full-rank matrices and their inverse](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-inverses\/#chapter-51-section-1)\n      2. [13\\.2 Full column rank matrices and left inverses](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-inverses\/#chapter-51-section-2)\n      3. [13\\.3 Full-row rank matrices and right inverses](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-inverses\/#chapter-51-section-3)\n   6. [14\\. LINEAR MAPS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/linear-maps\/)\n      1. [14\\.1 Definition and Interpretation](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/linear-maps\/#chapter-52-section-1)\n      2. [14\\.2 First-order approximation of non-linear maps](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/linear-maps\/#chapter-52-section-2)\n   7. [15\\. MATRIX NORMS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-norms\/)\n      1. [15\\.1 Motivating example: effect of noise in a linear system](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-norms\/#chapter-53-section-1)\n      2. [15\\.2 RMS gain: the Frobenius norm](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-norms\/#chapter-53-section-2)\n      3. [15\\.3 Peak gain: the largest singular value norm](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-norms\/#chapter-53-section-3)\n      4. [15\\.4 Applications](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-norms\/#chapter-53-section-4)\n   8. [16\\. APPLICATIONS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications\/)\n      1. [16\\.1 State-space models of linear dynamical systems.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications\/#chapter-54-section-1)\n   9. [17\\. EXERCISES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-2\/)\n      1. [17\\.1. Matrix products](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-2\/#chapter-55-section-1)\n      2. [17\\.2 Special matrices](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-2\/#chapter-55-section-2)\n      3. [17\\.3. Linear maps, dynamical systems](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-2\/#chapter-55-section-3)\n      4. [17\\.4 Matrix inverses, norms](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-2\/#chapter-55-section-4)\n4. [III. LINEAR EQUATIONS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/part\/chapter-iii-linear-equations\/)\n   1. [18\\. MOTIVATING EXAMPLE](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/motivating-example\/)\n      1. [18\\.1. Overview](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/motivating-example\/#chapter-59-section-1)\n      2. [18\\.2 From 1D to 2D: axial tomography](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/motivating-example\/#chapter-59-section-2)\n      3. [18\\.3. Linear equations for a single slice](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/motivating-example\/#chapter-59-section-3)\n      4. [18\\.4. Issues](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/motivating-example\/#chapter-59-section-4)\n   2. [19\\. EXISTENCE AND UNICITY OF SOLUTIONS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/existence-and-unicity-of-solutions\/)\n      1. [19\\.1. Set of solutions](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/existence-and-unicity-of-solutions\/#chapter-61-section-1)\n      2. [19\\.2. Existence: range and rank of a matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/existence-and-unicity-of-solutions\/#chapter-61-section-2)\n      3. [19\\.3. Unicity: nullspace of a matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/existence-and-unicity-of-solutions\/#chapter-61-section-3)\n      4. [19\\.4. Fundamental facts](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/existence-and-unicity-of-solutions\/#chapter-61-section-4)\n   3. [20\\. SOLVING LINEAR EQUATIONS VIA QR DECOMPOSITION](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-linear-equations-via-qr-decomposition\/)\n      1. [20\\.1. Basic idea: reduction to triangular systems of equations](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-linear-equations-via-qr-decomposition\/#chapter-62-section-1)\n      2. [20\\.2. The QR decomposition of a matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-linear-equations-via-qr-decomposition\/#chapter-62-section-2)\n      3. [20\\.3. Using the full QR decomposition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-linear-equations-via-qr-decomposition\/#chapter-62-section-3)\n      4. [20\\.4. Set of solutions](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-linear-equations-via-qr-decomposition\/#chapter-62-section-4)\n   4. [21\\. APPLICATIONS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-2\/)\n      1. [21\\.1. Trilateration by distance measurements](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-2\/#chapter-63-section-1)\n      2. [21\\.2. Estimation of traffic flow](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-2\/#chapter-63-section-2)\n   5. [22\\. EXERCISES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-3\/)\n      1. [22\\.1 Nullspace, rank and range](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-3\/#chapter-64-section-1)\n5. [IV. LEAST-SQUARES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/part\/chapter-iv-least-squares\/)\n   1. [23\\. ORDINARY LEAST-SQUARES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/)\n      1. [23\\.1. Definition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/#chapter-66-section-1)\n      2. [23\\.2. Interpretations](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/#chapter-66-section-2)\n      3. [23\\.3. Solution via QR decomposition (full rank case)](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/#chapter-66-section-3)\n      4. [23\\.4. Optimal solution and optimal set](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/#chapter-66-section-4)\n   2. [24\\. VARIANTS OF THE LEAST-SQUARES PROBLEM](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/variants-of-the-least-squares-problem\/)\n      1. [24\\.1. Linearly constrained least-squares](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/variants-of-the-least-squares-problem\/#chapter-67-section-1)\n      2. [24\\.2. Minimum-norm solution to linear equations](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/variants-of-the-least-squares-problem\/#chapter-67-section-2)\n      3. [24\\.3. Regularized least-squares](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/variants-of-the-least-squares-problem\/#chapter-67-section-3)\n   3. [25\\. KERNELS FOR LEAST-SQUARES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/kernels-for-least-squares\/)\n      1. [25\\.1. Motivations](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/kernels-for-least-squares\/#chapter-68-section-1)\n      2. [25\\.2. The kernel trick](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/kernels-for-least-squares\/#chapter-68-section-2)\n      3. [25\\.3. Nonlinear case](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/kernels-for-least-squares\/#chapter-68-section-3)\n      4. [25\\.4. Examples of kernels](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/kernels-for-least-squares\/#chapter-68-section-4)\n      5. [25\\.5. Kernels in practice](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/kernels-for-least-squares\/#chapter-68-section-5)\n   4. [26\\. APPLICATIONS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-3\/)\n      1. [26\\.1. Linear regression via least-squares](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-3\/#chapter-69-section-1)\n      2. [26\\.2. Auto-regressive models for time-series prediction.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-3\/#chapter-69-section-2)\n   5. [27\\. EXERCISES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-4\/)\n      1. [27\\.1. Standard forms](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-4\/#chapter-70-section-1)\n      2. [27\\.2. Applications](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-4\/#chapter-70-section-2)\n6. [V. EIGENVALUES FOR SYMMETRIC MATRICES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/part\/chapter-v-eigenvalues-for-symmetric-matrices\/)\n   1. [28\\. QUADRATIC FUNCTIONS AND SYMMETRIC MATRICES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/quadratic-functions-and-symmetric-matrices\/)\n      1. [28\\.1. Symmetric matrices and quadratic functions](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/quadratic-functions-and-symmetric-matrices\/#chapter-72-section-1)\n      2. [28\\.2. Second-order approximations of non-quadratic functions](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/quadratic-functions-and-symmetric-matrices\/#chapter-72-section-2)\n      3. [28\\.3. Special symmetric matrices](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/quadratic-functions-and-symmetric-matrices\/#chapter-72-section-3)\n   2. [29\\. SPECTRAL THEOREM](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/spectral-theorem\/)\n      1. [29\\.1. Eigenvalues and eigenvectors of symmetric matrices](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/spectral-theorem\/#chapter-73-section-1)\n      2. [29\\.2. Spectral theorem](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/spectral-theorem\/#chapter-73-section-2)\n      3. [29\\.3. Rayleigh quotients](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/spectral-theorem\/#chapter-73-section-3)\n   3. [30\\. POSITIVE SEMI-DEFINITE MATRICES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/positive-semi-definite-matrices\/)\n      1. [30\\.1. Definitions](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/positive-semi-definite-matrices\/#chapter-74-section-1)\n      2. [30\\.2. Special cases and examples](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/positive-semi-definite-matrices\/#chapter-74-section-2)\n      3. [30\\.3. Square root and Cholesky decomposition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/positive-semi-definite-matrices\/#chapter-74-section-3)\n      4. [30\\.4. Ellipsoids](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/positive-semi-definite-matrices\/#chapter-74-section-4)\n   4. [31\\. PRINCIPAL COMPONENT ANALYSIS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/principal-component-analysis\/)\n      1. [31\\.1. Projection on a line via variance maximization](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/principal-component-analysis\/#chapter-75-section-1)\n      2. [31\\.2. Principal component analysis](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/principal-component-analysis\/#chapter-75-section-2)\n      3. [31\\.3. Explained variance](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/principal-component-analysis\/#chapter-75-section-3)\n   5. [32\\. APPLICATIONS: PCA OF SENATE VOTING DATA](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-pca-of-senate-voting-data\/)\n      1. [32\\.1 Introduction](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-pca-of-senate-voting-data\/#chapter-76-section-1)\n      2. [32\\.2. Senate voting data and the visualization problem](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-pca-of-senate-voting-data\/#chapter-76-section-2)\n      3. [32\\.3. Projecting on a line](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-pca-of-senate-voting-data\/#chapter-76-section-3)\n      4. [32\\.4. Projecting on a plane](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-pca-of-senate-voting-data\/#chapter-76-section-4)\n      5. [32\\.5. Direction of maximal variance](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-pca-of-senate-voting-data\/#chapter-76-section-5)\n      6. [32\\.6. Principal component analysis](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-pca-of-senate-voting-data\/#chapter-76-section-6)\n      7. [32\\.7. Sparse PCA](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-pca-of-senate-voting-data\/#chapter-76-section-7)\n      8. [32\\.8 Sparse maximal variance problem](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-pca-of-senate-voting-data\/#chapter-76-section-8)\n   6. [33\\. EXERCISES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-5\/)\n      1. [33\\.1. Interpretation of covariance matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-5\/#chapter-77-section-1)\n      2. [33\\.2. Eigenvalue decomposition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-5\/#chapter-77-section-2)\n      3. [33\\.3. Positive-definite matrices, ellipsoids](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-5\/#chapter-77-section-3)\n      4. [33\\.4. Least-squares estimation](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-5\/#chapter-77-section-4)\n7. [VI. SINGULAR VALUES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/part\/chapter-vi-singular-values\/)\n   1. [34\\. THE SVD THEOREM](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/the-svd-theorem\/)\n      1. [34\\.1. The SVD theorem](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/the-svd-theorem\/#chapter-82-section-1)\n      2. [34\\.2. Geometry](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/the-svd-theorem\/#chapter-82-section-2)\n      3. [34\\.3. Link with the SED (Spectral Theorem)](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/the-svd-theorem\/#chapter-82-section-3)\n   2. [35\\. MATRIX PROPERTIES VIA SVD](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-properties-via-svd\/)\n      1. [35\\.1. Nullspace](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-properties-via-svd\/#chapter-83-section-1)\n      2. [35\\.2. Range, rank via the SVD](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-properties-via-svd\/#chapter-83-section-2)\n      3. [35\\.3. Fundamental theorem of linear algebra](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-properties-via-svd\/#chapter-83-section-3)\n      4. [35\\.4. Matrix norms, condition number](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/matrix-properties-via-svd\/#chapter-83-section-4)\n   3. [36\\. SOLVING LINEAR SYSTEMS VIA SVD](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-linear-systems-via-svd\/)\n      1. [36\\.1. Solution set](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-linear-systems-via-svd\/#chapter-84-section-1)\n      2. [36\\.2. Pseudo-inverse](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-linear-systems-via-svd\/#chapter-84-section-2)\n      3. [36\\.3. Sensitivity analysis and condition number](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-linear-systems-via-svd\/#chapter-84-section-3)\n   4. [37\\. LEAST-SQUARES AND SVD](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/least-squares-and-svd\/)\n      1. [37\\.1. Set of solutions](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/least-squares-and-svd\/#chapter-85-section-1)\n      2. [37\\.2. Sensitivity analysis](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/least-squares-and-svd\/#chapter-85-section-2)\n      3. [37\\.3. BLUE property](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/least-squares-and-svd\/#chapter-85-section-3)\n   5. [38\\. LOW-RANK APPROXIMATIONS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/low-rank-approximations\/)\n      1. [38\\.1. Low-rank approximations](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/low-rank-approximations\/#chapter-86-section-1)\n   6. [39\\. APPLICATIONS](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-4\/)\n      1. [39\\.1 Image compression](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-4\/#chapter-87-section-1)\n      2. [39\\.2 Market data analysis](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-4\/#chapter-87-section-2)\n   7. [40\\. EXERCISES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-6\/)\n      1. [40\\.1. SVD of simple matrices](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-6\/#chapter-88-section-1)\n      2. [40\\.2. Rank and SVD](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-6\/#chapter-88-section-2)\n      3. [40\\.3. Procrustes problem](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-6\/#chapter-88-section-3)\n      4. [40\\.4. SVD and projections](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-6\/#chapter-88-section-4)\n      5. [40\\.5. SVD and least-squares](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/exercises-6\/#chapter-88-section-5)\n8. VII. EXAMPLES\n   1. [Dimension of an affine subspace](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/examples\/)\n   2. [Sample and weighted average](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/sample-and-weighted-average\/)\n   3. [Sample average of vectors](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/sample-average-of-vectors\/)\n   4. [Euclidean projection on a set](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/euclidean-project-on-a-set\/)\n   5. [Orthogonal complement of a subspace](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/orthogonal-complement-of-a-subspace\/)\n   6. [Power laws](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/power-laws\/)\n   7. [Power law model fitting](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/power-law-model-fitting\/)\n   8. [Definition: Vector norm](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/definition-vector-norm-2\/)\n   9. [An infeasible linear system](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/an-infeasible-linear-system\/)\n   10. [Sample variance and standard deviation](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/sample-variance-and-standard-deviation\/)\n   11. [Functions and maps](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/functions-and-maps-2\/)\n       1. [Functions](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/functions-and-maps-2\/#chapter-727-section-1)\n       2. [Maps](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/functions-and-maps-2\/#chapter-727-section-2)\n   12. [Dual norm](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/dual-norm\/)\n   13. [Incidence matrix of a network](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/incidence-matrix-of-a-network-2\/)\n   14. [Nullspace of a transpose incidence matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/nullspace-of-a-transpose-incidence-matrix\/)\n   15. [Rank properties of the arc-node incidence matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/rank-properties-of-the-arc-node-incidence-matrix\/)\n   16. [Permutation matrices](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/permutation-matrices\/)\n   17. [QR decomposition: Examples](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/qr-decomposition-examples-2\/)\n   18. [Backwards substitution for solving triangular linear systems.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/backwards-substitution-for-solving-triangular-linear-systems\/)\n   19. [Solving triangular systems of equations: Backwards substitution example](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-triangular-systems-of-equations-backwards-substitution-example\/)\n   20. [Linear regression via least squares](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/linear-regression-via-least-squares-2\/)\n   21. [Nomenclature](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/nomenclature-2\/)\n       1. [Feasible set](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/nomenclature-2\/#chapter-747-section-1)\n       2. [What is a solution?](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/nomenclature-2\/#chapter-747-section-2)\n       3. [Local vs. global optimal points](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/nomenclature-2\/#chapter-747-section-3)\n   22. [Standard forms](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/standard-forms-2\/)\n       1. [Functional form](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/standard-forms-2\/#chapter-749-section-1)\n       2. [Epigraph form](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/standard-forms-2\/#chapter-749-section-2)\n       3. [Other standard forms](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/standard-forms-2\/#chapter-749-section-3)\n   23. [A two-dimensional toy optimization problem](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/a-two-dimensional-toy-optimization-problem-2\/)\n   24. [Global vs. local minima](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/global-vs-local-minima-2\/)\n   25. [Gradient of a function](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/gradient-of-function\/)\n       1. [Definition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/gradient-of-function\/#chapter-755-section-1)\n       2. [Composition rule with an affine function](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/gradient-of-function\/#chapter-755-section-2)\n       3. [Geometric interpretation](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/gradient-of-function\/#chapter-755-section-3)\n   26. [Set of solutions to the least-squares problem via QR decomposition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/set-of-solutions-to-the-least-squares-problem-via-qr-decomposition-2\/)\n   27. [Sample covariance matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/sample-covariance-matrix-2\/)\n       1. [Definition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/sample-covariance-matrix-2\/#chapter-760-section-1)\n       2. [Properties](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/sample-covariance-matrix-2\/#chapter-760-section-2)\n   28. [Optimal set of least-squares via SVD](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/optimal-set-of-least-squares-via-svd\/)\n   29. [Pseudo-inverse of a matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/pseudo-inverse-of-a-matrix-2\/)\n   30. [SVD: A 4x4 example](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/svd-a-4x4-example\/)\n   31. [Singular value decomposition of a 4 x 5 matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/singular-value-decomposition-of-a-4-x-5-matrix\/)\n   32. [Representation of a two-variable quadratic function](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/representation-of-a-two-variable-quadratic-function\/)\n   33. [Edge weight matrix of a graph](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/edge-weight-matrix-of-a-graph-2\/)\n   34. [Network flow](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/network-flow-2\/)\n   35. [Laplacian matrix of a graph](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/laplacian-matrix-of-a-graph\/)\n   36. [Hessian of a function](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/hessian-of-a-function-2\/)\n       1. [Definition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/hessian-of-a-function-2\/#chapter-780-section-1)\n       2. [Examples](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/hessian-of-a-function-2\/#chapter-780-section-2)\n   37. [Hessian of a quadratic function](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/hessian-of-a-quadratic-function-2\/)\n   38. [Gram matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/gram-matrix\/)\n   39. [Quadratic functions in two variables](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/quadratic-functions-in-two-variables-2\/)\n   40. [Quadratic approximation of the log-sum-exp function](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/quadratic-approximation-of-the-log-sum-exp-function-2\/)\n   41. [Determinant of a square matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/determinant-of-a-square-matrix-2\/)\n       1. [Definition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/determinant-of-a-square-matrix-2\/#chapter-791-section-1)\n       2. [Important result](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/determinant-of-a-square-matrix-2\/#chapter-791-section-2)\n       3. [Some properties](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/determinant-of-a-square-matrix-2\/#chapter-791-section-3)\n   42. [A squared linear function](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/a-squared-linear-function-2\/)\n   43. [Eigenvalue decomposition of a symmetric matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/eigenvalue-decomposition-of-a-symmetric-matrix-2\/)\n   44. [Rayleigh quotients](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/rayleigh-quotients\/)\n   45. [Largest singular value norm of a matrix](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/largest-singular-value-norm-of-a-matrix\/)\n   46. [Nullspace of a 4x5 matrix via its SVD](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/nullspace-of-a-4x5-matrix-via-its-svd\/)\n   47. [Range of a 4x5 matrix via its SVD](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/range-of-a-4x5-matrix-via-its-svd\/)\n   48. [Low-rank approximation of a 4x5 matrix via its SVD](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/low-rank-approximation-of-a-4x5-matrix-via-its-svd\/)\n   49. [Pseudo-inverse of a 4x5 matrix via its SVD](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/pseudo-inverse-of-a-4x5-matrix-via-its-svd\/)\n9. VIII. APPLICATIONS\n   1. [Image compression via least-squares](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-5\/)\n   2. [Senate voting data matrix.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/senate-voting-data-matrix-2\/)\n   3. [Senate voting analysis and visualization.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/senate-voting-analysis-and-visualization-2\/)\n   4. [Beer-Lambert law in absorption spectrometry](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/beer-lambert-law-in-absorption-spectrometry-2\/)\n   5. [Absorption spectrometry: Using measurements at different light frequencies.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/absorption-spectrometry-using-measurements-at-different-light-frequencies\/)\n   6. [Similarity of two documents](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/similarity-of-two-documents\/)\n   7. [Image compression](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/image-compression-2\/)\n   8. [Temperatures at different airports](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/temperatures-at-different-airports-2\/)\n   9. [Navigation by range measurement](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/navigation-by-range-measurement-2\/)\n   10. [Bag-of-words representation of text](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/bag-of-words-representation-of-text-2\/)\n   11. [Bag-of-words representation of text: Measure of document similarity](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/bag-of-words-representation-of-text-measure-of-document-similarity\/)\n   12. [Rate of return of a financial portfolio](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/rate-of-return-of-financial-portfolio\/)\n       1. [Rate of return of a single asset](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/rate-of-return-of-financial-portfolio\/#chapter-834-section-1)\n       2. [Log-returns](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/rate-of-return-of-financial-portfolio\/#chapter-834-section-2)\n       3. [Rate of return of a portfolio](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/rate-of-return-of-financial-portfolio\/#chapter-834-section-3)\n   13. [Single factor model of financial price data](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/single-factor-model-of-financial-price-data\/)\n   14. [The problem of Gauss](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/the-problem-of-gauss\/)\n   15. [Control of a unit mass](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/control-of-unit-mass\/)\n   16. [Portfolio optimization via linearly constrained least-squares.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/portfolio-optimization-via-linearly-constrained-least-squares-2\/)\n10. IX. THEOREMS\n    1. [Cauchy-Schwarz inequality proof](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/cauchy-schwarz-inequality-proof\/)\n    2. [Dimension of hyperplanes](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/dimension-of-hyperplanes\/)\n    3. [Spectral theorem: Eigenvalue decomposition for symmetric matrices](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/spectral-theorem-eigenvalue-decomposition-for-symmetric-matrices-2\/)\n    4. [Singular value decomposition (SVD) theorem](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/singular-value-decomposition-svd-theorem-2\/)\n    5. [Rank-one matrices](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/rank-one-matrices-2\/)\n    6. [Rank-one matrices: A representation theorem](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/rank-one-matrices-a-representation-theorem-2\/)\n    7. [Full rank matrices](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/full-rank-matrices\/)\n    8. [Rank-nullity theorem](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/rank-nullity-theorem-2\/)\n    9. [A theorem on positive semidefinite forms and eigenvalues](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/a-theorem-on-positive-semidefinitive-forms-and-eigenvalues\/)\n    10. [Fundamental theorem of linear algebra](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/fundamental-theorem-of-linear-algebra-2\/)\n11. [X. AI Learn](https:\/\/pressbooks.pub\/linearalgebraandapplications\/part\/ai-learn\/)\n# [Linear Algebra and Applications](https:\/\/pressbooks.pub\/linearalgebraandapplications\/)\n[Buy](https:\/\/pressbooks.pub\/linearalgebraandapplications\/buy\/)\n\n# 23 ORDINARY LEAST-SQUARES\n- [Definition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/#chapter-66-section-1)\n- [Interpretations](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/#chapter-66-section-2)\n- [Solution via QR decomposition (full rank case)](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/#chapter-66-section-3)\n- [Optimal solution (general case)](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/ordinary-least-squares\/#chapter-66-section-4)\n##\n# 23\\.1. Definition\nThe Ordinary Least-Squares (OLS, or LS) problem is defined as\n\n![\\\\\\[ \\\\min\\_x \\\\\\|A x - y\\\\\\|\\_2^2 \\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-5bd0471818070b19f7583c98d8b55ec8_l3.png)\n\nwhere ![A \\\\in \\\\mathbb{R}^{m \\\\times n}, \\\\; y \\\\in \\\\mathbb{R}^m](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-cf13aa7cfaa4d59b455f8b49c9f800c1_l3.png) are given. Together, the pair ![(A, y)](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-58f05f4a7a2602759ede7a7aa013defb_l3.png) is referred to as the *problem data*. The vector ![y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) is often referred to as the ‘‘measurement” or “output” vector, and the data matrix ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) as the ‘‘design‘‘ or ‘‘input‘‘ matrix. The vector ![r:=y-A x](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-f095a0905d773a419f8e00325d188c66_l3.png) is referred to as the *residual error* vector.\n\nNote that the problem is equivalent to one where the norm is not squared. Taking the squares is done for the convenience of the solution.\n\n# 23\\.2. Interpretations\n## Interpretation as projection on the range\n|   |   |\n|---|---|\n| [![](https:\/\/pressbooks.pub\/app\/uploads\/sites\/12458\/2023\/09\/Image14b.svg)](https:\/\/pressbooks.pub\/app\/uploads\/sites\/12458\/2023\/09\/Image14b.svg) | We can interpret the problem in terms of the columns of ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png), as follows. Assume that ![A=\\\\left\\[a\\_1, \\\\ldots, a\\_n\\\\right\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-85063cf6b2a138d9eac87d4d665d3190_l3.png), where ![a\\_j \\\\in \\\\mathbb{R}^m](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-bf42b2f5b7d6bb663c613821a8326585_l3.png) is the ![j](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-43c82d5bb00a7568d935a12e3bd969dd_l3.png)\\-th column of ![A \\\\; (j=1, \\\\ldots, n)](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-d8445476ac0186f1504b5590f6640870_l3.png). The problem reads |\n| ![\\\\\\[\\\\min \\_x\\\\left\\\\\\|\\\\sum\\_{j=1}^n x\\_j a\\_j-y\\\\right\\\\\\|\\_2 .\\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-9991a188d7e7374b521755f7b245a599_l3.png) |   |\n| In this sense, we are trying to find the best approximation of ![y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) in terms of a linear combination of the columns of ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png). Thus, the OLS problem amounts to *project* (find the minimum Euclidean distance) the vector ![y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) on the span of the vectors ![a\\_j](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-e205fc6b45b906f467e64c8105604f2e_l3.png) ‘s (that is to say: the range of ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png)). |   |\n| As seen in the picture, at optimum the *residual vector* ![A x-y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-759e1f254cc24999796dca49b37126d8_l3.png) is orthogonal to the range of ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png). |   |\n\n**See also**: [Image compression via least-squares.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-5\/)\n\n## Interpretation as minimum distance to feasibility\nThe OLS problem is usually applied to problems where the linear ![Ax=y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-ef9b2318eeb07429fb9de17d2fb480d2_l3.png) is not *feasible*, that is, there is no solution to ![Ax=y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-ef9b2318eeb07429fb9de17d2fb480d2_l3.png).\n\nThe OLS can be interpreted as finding the smallest (in Euclidean norm sense) perturbation of the right-hand side, ![\\\\delta y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-d03ae29673df3c029780366f946c6365_l3.png), such that the linear equation\n\n![\\\\\\[A x=y+\\\\delta y\\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-d3a4458cd9e22689e0e143d24a6438cc_l3.png)\n\nbecomes feasible. In this sense, the OLS formulation implicitly assumes that the data matrix ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) of the problem is known exactly, while only the right-hand side is subject to perturbation, or measurement errors. A more elaborate model, *total least-squares*, takes into account errors in both ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) and ![y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png).\n\n## Interpretation as regression\nWe can also interpret the problem in terms of the rows of ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png), as follows. Assume that ![A^T=\\\\left\\[a\\_1, \\\\ldots, a\\_m\\\\right\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-12b4c958054b51d74064f4eece5d232a_l3.png), where ![a\\_i \\\\in \\\\mathbb{R}^{n}](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-719241859d7e38a4b8d47c5c3a6b89b9_l3.png) is the ![i](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-695d9d59bd04859c6c99e7feb11daab6_l3.png)\\-th row of ![A \\\\;\\\\; (i=1, \\\\ldots, m)](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-c93ea69bac711703213ed1a70b7cbd95_l3.png). The problem reads\n\n![\\\\\\[\\\\min \\_x \\\\sum\\_{i=1}^m\\\\left(y\\_i-a\\_i^T x\\\\right)^2 .\\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-4a5056353ea968efd379e142105e803a_l3.png)\n\nIn this sense, we are trying to fit each component of ![y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) as a linear combination of the corresponding input ![a\\_i](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-f91083f3035e5168a6f0b3e6335d6858_l3.png), with ![x](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-ede05c264bba0eda080918aaa09c4658_l3.png) as the coefficients of this linear combination.\n\n**See also:**\n\n- [Linear regression.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/linear-regression-via-least-squares\/)\n- [Auto-regressive models for time series prediction](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-3\/#chapter-69-section-2).\n- [Power law model fitting.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/power-law-model-fitting\/)\n# 23\\.3. Solution via QR decomposition (full rank case)\nAssume that the matrix ![A \\\\in \\\\mathbb{R}^{m \\\\times n}](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-ab44a6e443678ee25caa73021f1128e5_l3.png) is tall (![m \\\\geq n](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-0f33e5f65664c8347d9ff5217b0362ba_l3.png)) and full column rank. Then the solution to the problem is unique and given by\n\n![\\\\\\[x^\\*=\\\\left(A^T A\\\\right)^{-1} A^T y .\\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-e9b9d24d2820056a4a50245137bc6906_l3.png)\n\nThis can be seen by simply taking the [gradient](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/gradient-of-function\/) (vector of derivatives) of the objective function, which leads to the optimality condition ![A^T(A x-y)=0](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-62b1fdaa42792ca372332edfbb350b93_l3.png). Geometrically, the residual vector ![A x-y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-759e1f254cc24999796dca49b37126d8_l3.png) is orthogonal to the span of the columns of ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png), as seen in the picture above.\n\nWe can also prove this via the [QR decomposition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/qr-decomposition-of-a-matrix\/) of the matrix ![A: A=Q R](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-ed471941fd66aa841ad8410d218cd07e_l3.png) with ![Q](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-2c758bec4c272382411b95fc0e7ee250_l3.png) a ![m \\\\times n](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-7d0c5e4f172b3def521d0e4c97406eed_l3.png) matrix with orthonormal columns ( ![Q^T Q=I\\_n](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-171ecf61e543ebd2ace369733aa2f26a_l3.png) ) and ![R](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-dae6bae3dcdac4629730754352c5e329_l3.png) a ![n \\\\times n](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-d5c3031c3a485131fabc8d0972e90f76_l3.png) upper-triangular, invertible matrix. Noting that\n\n![\\\\begin{align\\*} \\\\\\|A x-y\\\\\\|\\_2^2 & = x^T A^T A x - 2 x^T A^T y + y^T y \\\\\\\\ & = x^T R^T R x - 2 x^T R^T Q^T y + y^T y \\\\\\\\ & = \\\\left\\\\\\|R x-Q^T y\\\\right\\\\\\|\\_2^2 + y^T\\\\left(I-Q Q^T\\\\right) y \\\\end{align\\*}](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-983b7a914be8e937ef2d04c0fa2b9de0_l3.png)\n\nand exploiting the fact that ![R](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-dae6bae3dcdac4629730754352c5e329_l3.png) is invertible, we obtain the optimal solution ![x^\\*=R^{-1} Q^T y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-d22d352a05a3a1b01b456d7d27a53c4e_l3.png). This is the same as the formula above, since\n\n![\\\\\\[\\\\left(A^T A\\\\right)^{-1} A^T y=\\\\left(R^T Q^T Q R\\\\right)^{-1} R^T Q^T y=\\\\left(R^T R\\\\right)^{-1} R^T Q^T y=R^{-1} Q^T y .\\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-026a92d4e54e678b6058e0bf35b41dd5_l3.png)\n\nThus, to find the solution based on the QR decomposition, we just need to implement two steps:\n\n1. Rotate the output vector: set ![\\\\bar{y}=Q^T y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-d3952cd3ed4956c497c4b6b23e38bdbd_l3.png).\n2. Solve the triangular system ![R x=\\\\bar{y}](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-aa9c97f23d5499bb1ccde4164324347b_l3.png) by [backward substitution](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-triangular-systems-of-equations-backwards-substitution-example\/).\n# 23\\.4. Optimal solution and optimal set\nRecall that the [optimal set](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/global-vs-local-minima\/) of a minimization problem is its set of minimizers. For least-squares problems, the optimal set is an affine set, which reduces to a singleton when ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) is full column rank.\n\nIn the general case (![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) is not necessarily tall, and \/or not full rank) then the solution may not be unique. If ![x^0](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-7c37f91603121b8b9fd93982223c4445_l3.png) is a particular solution, then ![x=x^0+z](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-9aae2736ef0c40a87dd2feb50cfb15a9_l3.png) is also a solution, if ![z](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-4586e340cb83d5b642972e97a288fec2_l3.png) is such that ![A z=0](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-295470b91e02b1abda500af0ecb4e2ec_l3.png), that is, ![z \\\\in \\\\mathbf{N}(A)](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-19a4654afae0e46f3dd5c8d6c72027ae_l3.png). That is, the nullspace of ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) describes the *ambiguity* of solutions. In mathematical terms:\n\n![\\\\\\[ \\\\underset{x}{\\\\text{argmin}} \\\\\\|A x-y\\\\\\|\\_2 = x^0 + \\\\mathbf{N}(A) . \\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-9c1906c394f150dd46f4a4e5f402f9eb_l3.png)\n\nThe formal expression for the set of minimizers to the least-squares problem can be found again via the QR decomposition. This is shown [here](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/set-of-solutions-to-the-least-squares-problem-via-qr-decomposition\/).\n\nPrevious\/next navigation\n\n[Previous: LEAST-SQUARES](https:\/\/pressbooks.pub\/linearalgebraandapplications\/part\/chapter-iv-least-squares\/)\n\n[Next: VARIANTS OF THE LEAST-SQUARES PROBLEM](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/variants-of-the-least-squares-problem\/)\n\nBack to top\n\n## License\n![Icon for the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License](https:\/\/pressbooks.pub\/app\/themes\/pressbooks-book\/packages\/buckram\/assets\/images\/cc-by-nc-nd.svg)\n\n[Linear Algebra and Applications](https:\/\/pressbooks.pub\/linearalgebraandapplications) Copyright © 2023 by VinUiversity is licensed under a [Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License](https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/), except where otherwise noted.\n## Share This Book\n\n[Pressbooks](https:\/\/pressbooks.com\/)\n\n[Powered by Pressbooks](https:\/\/pressbooks.com\/)\n\n- [Pressbooks User Guide](https:\/\/guide.pressbooks.com\/)\n- \\|[Pressbooks Directory](https:\/\/pressbooks.directory\/)\n- \\|[Contact](https:\/\/pressbooks.com\/support\/)\n\n[Pressbooks on YouTube](https:\/\/www.youtube.com\/user\/pressbooks)\n\n[Pressbooks on LinkedIn](https:\/\/www.linkedin.com\/company\/pressbooks\/?originalSubdomain=ca)","attrs_readable_markdown":"The Ordinary Least-Squares (OLS, or LS) problem is defined as\n\n![\\\\\\[ \\\\min\\_x \\\\\\|A x - y\\\\\\|\\_2^2 \\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-5bd0471818070b19f7583c98d8b55ec8_l3.png)\n\nwhere ![A \\\\in \\\\mathbb{R}^{m \\\\times n}, \\\\; y \\\\in \\\\mathbb{R}^m](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-cf13aa7cfaa4d59b455f8b49c9f800c1_l3.png) are given. Together, the pair ![(A, y)](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-58f05f4a7a2602759ede7a7aa013defb_l3.png) is referred to as the *problem data*. The vector ![y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) is often referred to as the ‘‘measurement” or “output” vector, and the data matrix ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) as the ‘‘design‘‘ or ‘‘input‘‘ matrix. The vector ![r:=y-A x](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-f095a0905d773a419f8e00325d188c66_l3.png) is referred to as the *residual error* vector.\n\nNote that the problem is equivalent to one where the norm is not squared. Taking the squares is done for the convenience of the solution.\n\n## Interpretation as projection on the range\n**See also**: [Image compression via least-squares.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-5\/)\n\n## Interpretation as minimum distance to feasibility\nThe OLS problem is usually applied to problems where the linear ![Ax=y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-ef9b2318eeb07429fb9de17d2fb480d2_l3.png) is not *feasible*, that is, there is no solution to ![Ax=y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-ef9b2318eeb07429fb9de17d2fb480d2_l3.png).\n\nThe OLS can be interpreted as finding the smallest (in Euclidean norm sense) perturbation of the right-hand side, ![\\\\delta y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-d03ae29673df3c029780366f946c6365_l3.png), such that the linear equation\n\n![\\\\\\[A x=y+\\\\delta y\\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-d3a4458cd9e22689e0e143d24a6438cc_l3.png)\n\nbecomes feasible. In this sense, the OLS formulation implicitly assumes that the data matrix ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) of the problem is known exactly, while only the right-hand side is subject to perturbation, or measurement errors. A more elaborate model, *total least-squares*, takes into account errors in both ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) and ![y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png).\n\n## Interpretation as regression\nWe can also interpret the problem in terms of the rows of ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png), as follows. Assume that ![A^T=\\\\left\\[a\\_1, \\\\ldots, a\\_m\\\\right\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-12b4c958054b51d74064f4eece5d232a_l3.png), where ![a\\_i \\\\in \\\\mathbb{R}^{n}](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-719241859d7e38a4b8d47c5c3a6b89b9_l3.png) is the ![i](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-695d9d59bd04859c6c99e7feb11daab6_l3.png)\\-th row of ![A \\\\;\\\\; (i=1, \\\\ldots, m)](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-c93ea69bac711703213ed1a70b7cbd95_l3.png). The problem reads\n\n![\\\\\\[\\\\min \\_x \\\\sum\\_{i=1}^m\\\\left(y\\_i-a\\_i^T x\\\\right)^2 .\\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-4a5056353ea968efd379e142105e803a_l3.png)\n\nIn this sense, we are trying to fit each component of ![y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) as a linear combination of the corresponding input ![a\\_i](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-f91083f3035e5168a6f0b3e6335d6858_l3.png), with ![x](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-ede05c264bba0eda080918aaa09c4658_l3.png) as the coefficients of this linear combination.\n\n**See also:**\n\n- [Linear regression.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/linear-regression-via-least-squares\/)\n- [Auto-regressive models for time series prediction](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/applications-3\/#chapter-69-section-2).\n- [Power law model fitting.](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/power-law-model-fitting\/)\n\nAssume that the matrix ![A \\\\in \\\\mathbb{R}^{m \\\\times n}](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-ab44a6e443678ee25caa73021f1128e5_l3.png) is tall (![m \\\\geq n](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-0f33e5f65664c8347d9ff5217b0362ba_l3.png)) and full column rank. Then the solution to the problem is unique and given by\n\n![\\\\\\[x^\\*=\\\\left(A^T A\\\\right)^{-1} A^T y .\\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-e9b9d24d2820056a4a50245137bc6906_l3.png)\n\nThis can be seen by simply taking the [gradient](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/gradient-of-function\/) (vector of derivatives) of the objective function, which leads to the optimality condition ![A^T(A x-y)=0](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-62b1fdaa42792ca372332edfbb350b93_l3.png). Geometrically, the residual vector ![A x-y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-759e1f254cc24999796dca49b37126d8_l3.png) is orthogonal to the span of the columns of ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png), as seen in the picture above.\n\nWe can also prove this via the [QR decomposition](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/qr-decomposition-of-a-matrix\/) of the matrix ![A: A=Q R](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-ed471941fd66aa841ad8410d218cd07e_l3.png) with ![Q](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-2c758bec4c272382411b95fc0e7ee250_l3.png) a ![m \\\\times n](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-7d0c5e4f172b3def521d0e4c97406eed_l3.png) matrix with orthonormal columns ( ![Q^T Q=I\\_n](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-171ecf61e543ebd2ace369733aa2f26a_l3.png) ) and ![R](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-dae6bae3dcdac4629730754352c5e329_l3.png) a ![n \\\\times n](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-d5c3031c3a485131fabc8d0972e90f76_l3.png) upper-triangular, invertible matrix. Noting that\n\n![\\\\begin{align\\*} \\\\\\|A x-y\\\\\\|\\_2^2 & = x^T A^T A x - 2 x^T A^T y + y^T y \\\\\\\\ & = x^T R^T R x - 2 x^T R^T Q^T y + y^T y \\\\\\\\ & = \\\\left\\\\\\|R x-Q^T y\\\\right\\\\\\|\\_2^2 + y^T\\\\left(I-Q Q^T\\\\right) y \\\\end{align\\*}](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-983b7a914be8e937ef2d04c0fa2b9de0_l3.png)\n\nand exploiting the fact that ![R](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-dae6bae3dcdac4629730754352c5e329_l3.png) is invertible, we obtain the optimal solution ![x^\\*=R^{-1} Q^T y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-d22d352a05a3a1b01b456d7d27a53c4e_l3.png). This is the same as the formula above, since\n\n![\\\\\\[\\\\left(A^T A\\\\right)^{-1} A^T y=\\\\left(R^T Q^T Q R\\\\right)^{-1} R^T Q^T y=\\\\left(R^T R\\\\right)^{-1} R^T Q^T y=R^{-1} Q^T y .\\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-026a92d4e54e678b6058e0bf35b41dd5_l3.png)\n\nThus, to find the solution based on the QR decomposition, we just need to implement two steps:\n\n1. Rotate the output vector: set ![\\\\bar{y}=Q^T y](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-d3952cd3ed4956c497c4b6b23e38bdbd_l3.png).\n2. Solve the triangular system ![R x=\\\\bar{y}](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-aa9c97f23d5499bb1ccde4164324347b_l3.png) by [backward substitution](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/solving-triangular-systems-of-equations-backwards-substitution-example\/).\n\nRecall that the [optimal set](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/global-vs-local-minima\/) of a minimization problem is its set of minimizers. For least-squares problems, the optimal set is an affine set, which reduces to a singleton when ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) is full column rank.\n\nIn the general case (![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) is not necessarily tall, and \/or not full rank) then the solution may not be unique. If ![x^0](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-7c37f91603121b8b9fd93982223c4445_l3.png) is a particular solution, then ![x=x^0+z](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-9aae2736ef0c40a87dd2feb50cfb15a9_l3.png) is also a solution, if ![z](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-4586e340cb83d5b642972e97a288fec2_l3.png) is such that ![A z=0](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-295470b91e02b1abda500af0ecb4e2ec_l3.png), that is, ![z \\\\in \\\\mathbf{N}(A)](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-19a4654afae0e46f3dd5c8d6c72027ae_l3.png). That is, the nullspace of ![A](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) describes the *ambiguity* of solutions. In mathematical terms:\n\n![\\\\\\[ \\\\underset{x}{\\\\text{argmin}} \\\\\\|A x-y\\\\\\|\\_2 = x^0 + \\\\mathbf{N}(A) . \\\\\\]](https:\/\/pressbooks.pub\/app\/uploads\/quicklatex\/quicklatex.com-9c1906c394f150dd46f4a4e5f402f9eb_l3.png)\n\nThe formal expression for the set of minimizers to the least-squares problem can be found again via the QR decomposition. This is shown [here](https:\/\/pressbooks.pub\/linearalgebraandapplications\/chapter\/set-of-solutions-to-the-least-squares-problem-via-qr-decomposition\/).","meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

3 days ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0.1 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/
Last Crawled	2026-04-14 13:16:35 (3 days ago)
First Indexed	2024-01-01 21:33:08 (2 years ago)
HTTP Status Code	200
Meta Title	ORDINARY LEAST-SQUARES – Linear Algebra and Applications
Meta Description	null
Meta Canonical	null
Boilerpipe Text	The Ordinary Least-Squares (OLS, or LS) problem is defined as where are given. Together, the pair is referred to as the problem data . The vector is often referred to as the ‘‘measurement” or “output” vector, and the data matrix as the ‘‘design‘‘ or ‘‘input‘‘ matrix. The vector is referred to as the residual error vector. Note that the problem is equivalent to one where the norm is not squared. Taking the squares is done for the convenience of the solution. Interpretation as projection on the range See also : Image compression via least-squares. Interpretation as minimum distance to feasibility The OLS problem is usually applied to problems where the linear is not feasible , that is, there is no solution to . The OLS can be interpreted as finding the smallest (in Euclidean norm sense) perturbation of the right-hand side, , such that the linear equation becomes feasible. In this sense, the OLS formulation implicitly assumes that the data matrix of the problem is known exactly, while only the right-hand side is subject to perturbation, or measurement errors. A more elaborate model, total least-squares , takes into account errors in both and . Interpretation as regression We can also interpret the problem in terms of the rows of , as follows. Assume that , where is the -th row of . The problem reads In this sense, we are trying to fit each component of as a linear combination of the corresponding input , with as the coefficients of this linear combination. See also: Linear regression. Auto-regressive models for time series prediction . Power law model fitting. Assume that the matrix is tall ( ) and full column rank. Then the solution to the problem is unique and given by This can be seen by simply taking the gradient (vector of derivatives) of the objective function, which leads to the optimality condition . Geometrically, the residual vector is orthogonal to the span of the columns of , as seen in the picture above. We can also prove this via the QR decomposition of the matrix with a matrix with orthonormal columns ( ) and a upper-triangular, invertible matrix. Noting that and exploiting the fact that is invertible, we obtain the optimal solution . This is the same as the formula above, since Thus, to find the solution based on the QR decomposition, we just need to implement two steps: Rotate the output vector: set . Solve the triangular system by backward substitution . Recall that the optimal set of a minimization problem is its set of minimizers. For least-squares problems, the optimal set is an affine set, which reduces to a singleton when is full column rank. In the general case ( is not necessarily tall, and /or not full rank) then the solution may not be unique. If is a particular solution, then is also a solution, if is such that , that is, . That is, the nullspace of describes the ambiguity of solutions. In mathematical terms: The formal expression for the set of minimizers to the least-squares problem can be found again via the QR decomposition. This is shown here .
Markdown	" [Skip to content](https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/#content) Menu Primary Navigation - [Home](https://pressbooks.pub/linearalgebraandapplications) - [Read](https://pressbooks.pub/linearalgebraandapplications/front-matter/introduction/) - [Buy](https://pressbooks.pub/linearalgebraandapplications/buy/) - [Sign in](https://pressbooks.pub/linearalgebraandapplications/wp-login.php?redirect_to=https%3A%2F%2Fpressbooks.pub%2Flinearalgebraandapplications%2Fchapter%2Fordinary-least-squares%2F) Want to create or adapt books like this? [Learn more](https://pressbooks.com/adapt-open-textbooks?utm_source=book&utm_medium=banner&utm_campaign=bbc) about how Pressbooks supports open publishing practices. Book Contents Navigation Contents 1. [INTRODUCTION](https://pressbooks.pub/linearalgebraandapplications/front-matter/introduction/) 2. [I. VECTORS](https://pressbooks.pub/linearalgebraandapplications/part/main-body/) 1. [1\. BASICS](https://pressbooks.pub/linearalgebraandapplications/chapter/basics/) 1. [1\.1. Definitions](https://pressbooks.pub/linearalgebraandapplications/chapter/basics/#chapter-26-section-1) 2. [1\.2. Independence](https://pressbooks.pub/linearalgebraandapplications/chapter/basics/#chapter-26-section-2) 3. [1\.3. Subspace, span, affine sets](https://pressbooks.pub/linearalgebraandapplications/chapter/basics/#chapter-26-section-3) 4. [1\.4. Basis, dimension](https://pressbooks.pub/linearalgebraandapplications/chapter/basics/#chapter-26-section-4) 2. [2\. SCALAR PRODUCT, NORMS AND ANGLES](https://pressbooks.pub/linearalgebraandapplications/chapter/scalar-product-norms-and-angles/) 1. [2\.1. Scalar product](https://pressbooks.pub/linearalgebraandapplications/chapter/scalar-product-norms-and-angles/#chapter-36-section-1) 2. [2\.2. Norms](https://pressbooks.pub/linearalgebraandapplications/chapter/scalar-product-norms-and-angles/#chapter-36-section-2) 3. [2\.3. Three popular norms](https://pressbooks.pub/linearalgebraandapplications/chapter/scalar-product-norms-and-angles/#chapter-36-section-3) 4. [2\.4. Cauchy-Schwarz inequality](https://pressbooks.pub/linearalgebraandapplications/chapter/scalar-product-norms-and-angles/#chapter-36-section-4) 5. [2\.5. Angles between vectors](https://pressbooks.pub/linearalgebraandapplications/chapter/scalar-product-norms-and-angles/#chapter-36-section-5) 3. [3\. PROJECTION ON A LINE](https://pressbooks.pub/linearalgebraandapplications/chapter/projection-on-a-line/) 1. [3\.1. Definition](https://pressbooks.pub/linearalgebraandapplications/chapter/projection-on-a-line/#chapter-37-section-1) 2. [3\.2. Closed-form expression](https://pressbooks.pub/linearalgebraandapplications/chapter/projection-on-a-line/#chapter-37-section-2) 3. [3\.3. Interpreting the scalar product](https://pressbooks.pub/linearalgebraandapplications/chapter/projection-on-a-line/#chapter-37-section-3) 4. [4\. ORTHOGONALIZATION: THE GRAM-SCHMIDT PROCEDURE](https://pressbooks.pub/linearalgebraandapplications/chapter/orthogonalization-the-gram-schmidt-procedure/) 1. [4\.1. What is orthogonalization?](https://pressbooks.pub/linearalgebraandapplications/chapter/orthogonalization-the-gram-schmidt-procedure/#chapter-39-section-1) 2. [4\.2. Basic step: projection on a line](https://pressbooks.pub/linearalgebraandapplications/chapter/orthogonalization-the-gram-schmidt-procedure/#chapter-39-section-2) 3. [4\.3. Gram-Schmidt procedure](https://pressbooks.pub/linearalgebraandapplications/chapter/orthogonalization-the-gram-schmidt-procedure/#chapter-39-section-3) 5. [5\. HYPERPLANES AND HALF-SPACES](https://pressbooks.pub/linearalgebraandapplications/chapter/hyperplanes-and-half-spaces/) 1. [5\.1. Hyperplanes](https://pressbooks.pub/linearalgebraandapplications/chapter/hyperplanes-and-half-spaces/#chapter-41-section-1) 2. [5\.2. Projection on a hyperplane](https://pressbooks.pub/linearalgebraandapplications/chapter/hyperplanes-and-half-spaces/#chapter-41-section-2) 3. [5\.3. Geometry of hyperplanes](https://pressbooks.pub/linearalgebraandapplications/chapter/hyperplanes-and-half-spaces/#chapter-41-section-3) 4. [5\.4. Half-spaces](https://pressbooks.pub/linearalgebraandapplications/chapter/hyperplanes-and-half-spaces/#chapter-41-section-4) 6. [6\. LINEAR FUNCTIONS](https://pressbooks.pub/linearalgebraandapplications/chapter/linear-functions/) 1. [6\.1. Linear and affine functions](https://pressbooks.pub/linearalgebraandapplications/chapter/linear-functions/#chapter-42-section-1) 2. [6\.2. First-order approximation of non-linear functions](https://pressbooks.pub/linearalgebraandapplications/chapter/linear-functions/#chapter-42-section-2) 3. [6\.3. Other sources of linear models](https://pressbooks.pub/linearalgebraandapplications/chapter/linear-functions/#chapter-42-section-3) 7. [7\. APPLICATION: DATA VISUALIZATION BY PROJECTION ON A LINE](https://pressbooks.pub/linearalgebraandapplications/chapter/application-data-visualization-by-projection-on-a-line/) 1. [7\.1. Senate voting data](https://pressbooks.pub/linearalgebraandapplications/chapter/application-data-visualization-by-projection-on-a-line/#chapter-43-section-1) 2. [7\.2. Visualization of high-dimensional data via projection](https://pressbooks.pub/linearalgebraandapplications/chapter/application-data-visualization-by-projection-on-a-line/#chapter-43-section-2) 3. [7\.3. Examples](https://pressbooks.pub/linearalgebraandapplications/chapter/application-data-visualization-by-projection-on-a-line/#chapter-43-section-3) 8. [8\. EXERCISES](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises/) 1. [8\.1. Subspaces](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises/#chapter-44-section-1) 2. [8\.2. Projections, scalar product, angles](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises/#chapter-44-section-2) 3. [8\.3. Orthogonalization](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises/#chapter-44-section-3) 4. [8\.4. Generalized Cauchy-Schwarz inequalities](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises/#chapter-44-section-4) 5. [8\.5. Linear functions](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises/#chapter-44-section-5) 3. [II. MATRICES](https://pressbooks.pub/linearalgebraandapplications/part/chapter-ii-matrices/) 1. [9\. BASICS](https://pressbooks.pub/linearalgebraandapplications/chapter/basics-2/) 1. [9\.1. Matrices as collections of column vectors](https://pressbooks.pub/linearalgebraandapplications/chapter/basics-2/#chapter-47-section-1) 2. [9\.2. Transpose](https://pressbooks.pub/linearalgebraandapplications/chapter/basics-2/#chapter-47-section-2) 3. [9\.3. Matrices as collections of rows](https://pressbooks.pub/linearalgebraandapplications/chapter/basics-2/#chapter-47-section-3) 4. [9\.4. Sparse Matrices](https://pressbooks.pub/linearalgebraandapplications/chapter/basics-2/#chapter-47-section-4) 2. [10\. MATRIX-VECTOR AND MATRIX-MATRIX MULTIPLICATION, SCALAR PRODUCT](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-vector-and-matrix-matrix-multiplication-scalar-product/) 1. [10\.1. Matrix-vector product](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-vector-and-matrix-matrix-multiplication-scalar-product/#chapter-48-section-1) 2. [10\.2. Matrix-matrix product](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-vector-and-matrix-matrix-multiplication-scalar-product/#chapter-48-section-2) 3. [10\.3. Block Matrix Products](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-vector-and-matrix-matrix-multiplication-scalar-product/#chapter-48-section-3) 4. [10\.4. Trace, scalar product](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-vector-and-matrix-matrix-multiplication-scalar-product/#chapter-48-section-4) 3. [11\. SPECIAL CLASSES OF MATRICES](https://pressbooks.pub/linearalgebraandapplications/chapter/special-classes-of-matrices/) 1. [11\.1 Some special square matrices](https://pressbooks.pub/linearalgebraandapplications/chapter/special-classes-of-matrices/#chapter-49-section-1) 2. [11\.2 Dyads](https://pressbooks.pub/linearalgebraandapplications/chapter/special-classes-of-matrices/#chapter-49-section-2) 4. [12\. QR DECOMPOSITION OF A MATRIX](https://pressbooks.pub/linearalgebraandapplications/chapter/qr-decomposition-of-a-matrix/) 1. [12\.1 Basic idea](https://pressbooks.pub/linearalgebraandapplications/chapter/qr-decomposition-of-a-matrix/#chapter-50-section-1) 2. [12\.2 Case being full column rank](https://pressbooks.pub/linearalgebraandapplications/chapter/qr-decomposition-of-a-matrix/#chapter-50-section-2) 3. [12\.3 Case when the columns are not independent](https://pressbooks.pub/linearalgebraandapplications/chapter/qr-decomposition-of-a-matrix/#chapter-50-section-3) 4. [12\.4 Full QR decomposition](https://pressbooks.pub/linearalgebraandapplications/chapter/qr-decomposition-of-a-matrix/#chapter-50-section-4) 5. [13\. MATRIX INVERSES](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-inverses/) 1. [13\.1 Square full-rank matrices and their inverse](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-inverses/#chapter-51-section-1) 2. [13\.2 Full column rank matrices and left inverses](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-inverses/#chapter-51-section-2) 3. [13\.3 Full-row rank matrices and right inverses](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-inverses/#chapter-51-section-3) 6. [14\. LINEAR MAPS](https://pressbooks.pub/linearalgebraandapplications/chapter/linear-maps/) 1. [14\.1 Definition and Interpretation](https://pressbooks.pub/linearalgebraandapplications/chapter/linear-maps/#chapter-52-section-1) 2. [14\.2 First-order approximation of non-linear maps](https://pressbooks.pub/linearalgebraandapplications/chapter/linear-maps/#chapter-52-section-2) 7. [15\. MATRIX NORMS](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-norms/) 1. [15\.1 Motivating example: effect of noise in a linear system](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-norms/#chapter-53-section-1) 2. [15\.2 RMS gain: the Frobenius norm](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-norms/#chapter-53-section-2) 3. [15\.3 Peak gain: the largest singular value norm](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-norms/#chapter-53-section-3) 4. [15\.4 Applications](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-norms/#chapter-53-section-4) 8. [16\. APPLICATIONS](https://pressbooks.pub/linearalgebraandapplications/chapter/applications/) 1. [16\.1 State-space models of linear dynamical systems.](https://pressbooks.pub/linearalgebraandapplications/chapter/applications/#chapter-54-section-1) 9. [17\. EXERCISES](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-2/) 1. [17\.1. Matrix products](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-2/#chapter-55-section-1) 2. [17\.2 Special matrices](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-2/#chapter-55-section-2) 3. [17\.3. Linear maps, dynamical systems](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-2/#chapter-55-section-3) 4. [17\.4 Matrix inverses, norms](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-2/#chapter-55-section-4) 4. [III. LINEAR EQUATIONS](https://pressbooks.pub/linearalgebraandapplications/part/chapter-iii-linear-equations/) 1. [18\. MOTIVATING EXAMPLE](https://pressbooks.pub/linearalgebraandapplications/chapter/motivating-example/) 1. [18\.1. Overview](https://pressbooks.pub/linearalgebraandapplications/chapter/motivating-example/#chapter-59-section-1) 2. [18\.2 From 1D to 2D: axial tomography](https://pressbooks.pub/linearalgebraandapplications/chapter/motivating-example/#chapter-59-section-2) 3. [18\.3. Linear equations for a single slice](https://pressbooks.pub/linearalgebraandapplications/chapter/motivating-example/#chapter-59-section-3) 4. [18\.4. Issues](https://pressbooks.pub/linearalgebraandapplications/chapter/motivating-example/#chapter-59-section-4) 2. [19\. EXISTENCE AND UNICITY OF SOLUTIONS](https://pressbooks.pub/linearalgebraandapplications/chapter/existence-and-unicity-of-solutions/) 1. [19\.1. Set of solutions](https://pressbooks.pub/linearalgebraandapplications/chapter/existence-and-unicity-of-solutions/#chapter-61-section-1) 2. [19\.2. Existence: range and rank of a matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/existence-and-unicity-of-solutions/#chapter-61-section-2) 3. [19\.3. Unicity: nullspace of a matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/existence-and-unicity-of-solutions/#chapter-61-section-3) 4. [19\.4. Fundamental facts](https://pressbooks.pub/linearalgebraandapplications/chapter/existence-and-unicity-of-solutions/#chapter-61-section-4) 3. [20\. SOLVING LINEAR EQUATIONS VIA QR DECOMPOSITION](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-linear-equations-via-qr-decomposition/) 1. [20\.1. Basic idea: reduction to triangular systems of equations](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-linear-equations-via-qr-decomposition/#chapter-62-section-1) 2. [20\.2. The QR decomposition of a matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-linear-equations-via-qr-decomposition/#chapter-62-section-2) 3. [20\.3. Using the full QR decomposition](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-linear-equations-via-qr-decomposition/#chapter-62-section-3) 4. [20\.4. Set of solutions](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-linear-equations-via-qr-decomposition/#chapter-62-section-4) 4. [21\. APPLICATIONS](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-2/) 1. [21\.1. Trilateration by distance measurements](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-2/#chapter-63-section-1) 2. [21\.2. Estimation of traffic flow](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-2/#chapter-63-section-2) 5. [22\. EXERCISES](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-3/) 1. [22\.1 Nullspace, rank and range](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-3/#chapter-64-section-1) 5. [IV. LEAST-SQUARES](https://pressbooks.pub/linearalgebraandapplications/part/chapter-iv-least-squares/) 1. [23\. ORDINARY LEAST-SQUARES](https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/) 1. [23\.1. Definition](https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/#chapter-66-section-1) 2. [23\.2. Interpretations](https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/#chapter-66-section-2) 3. [23\.3. Solution via QR decomposition (full rank case)](https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/#chapter-66-section-3) 4. [23\.4. Optimal solution and optimal set](https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/#chapter-66-section-4) 2. [24\. VARIANTS OF THE LEAST-SQUARES PROBLEM](https://pressbooks.pub/linearalgebraandapplications/chapter/variants-of-the-least-squares-problem/) 1. [24\.1. Linearly constrained least-squares](https://pressbooks.pub/linearalgebraandapplications/chapter/variants-of-the-least-squares-problem/#chapter-67-section-1) 2. [24\.2. Minimum-norm solution to linear equations](https://pressbooks.pub/linearalgebraandapplications/chapter/variants-of-the-least-squares-problem/#chapter-67-section-2) 3. [24\.3. Regularized least-squares](https://pressbooks.pub/linearalgebraandapplications/chapter/variants-of-the-least-squares-problem/#chapter-67-section-3) 3. [25\. KERNELS FOR LEAST-SQUARES](https://pressbooks.pub/linearalgebraandapplications/chapter/kernels-for-least-squares/) 1. [25\.1. Motivations](https://pressbooks.pub/linearalgebraandapplications/chapter/kernels-for-least-squares/#chapter-68-section-1) 2. [25\.2. The kernel trick](https://pressbooks.pub/linearalgebraandapplications/chapter/kernels-for-least-squares/#chapter-68-section-2) 3. [25\.3. Nonlinear case](https://pressbooks.pub/linearalgebraandapplications/chapter/kernels-for-least-squares/#chapter-68-section-3) 4. [25\.4. Examples of kernels](https://pressbooks.pub/linearalgebraandapplications/chapter/kernels-for-least-squares/#chapter-68-section-4) 5. [25\.5. Kernels in practice](https://pressbooks.pub/linearalgebraandapplications/chapter/kernels-for-least-squares/#chapter-68-section-5) 4. [26\. APPLICATIONS](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-3/) 1. [26\.1. Linear regression via least-squares](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-3/#chapter-69-section-1) 2. [26\.2. Auto-regressive models for time-series prediction.](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-3/#chapter-69-section-2) 5. [27\. EXERCISES](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-4/) 1. [27\.1. Standard forms](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-4/#chapter-70-section-1) 2. [27\.2. Applications](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-4/#chapter-70-section-2) 6. [V. EIGENVALUES FOR SYMMETRIC MATRICES](https://pressbooks.pub/linearalgebraandapplications/part/chapter-v-eigenvalues-for-symmetric-matrices/) 1. [28\. QUADRATIC FUNCTIONS AND SYMMETRIC MATRICES](https://pressbooks.pub/linearalgebraandapplications/chapter/quadratic-functions-and-symmetric-matrices/) 1. [28\.1. Symmetric matrices and quadratic functions](https://pressbooks.pub/linearalgebraandapplications/chapter/quadratic-functions-and-symmetric-matrices/#chapter-72-section-1) 2. [28\.2. Second-order approximations of non-quadratic functions](https://pressbooks.pub/linearalgebraandapplications/chapter/quadratic-functions-and-symmetric-matrices/#chapter-72-section-2) 3. [28\.3. Special symmetric matrices](https://pressbooks.pub/linearalgebraandapplications/chapter/quadratic-functions-and-symmetric-matrices/#chapter-72-section-3) 2. [29\. SPECTRAL THEOREM](https://pressbooks.pub/linearalgebraandapplications/chapter/spectral-theorem/) 1. [29\.1. Eigenvalues and eigenvectors of symmetric matrices](https://pressbooks.pub/linearalgebraandapplications/chapter/spectral-theorem/#chapter-73-section-1) 2. [29\.2. Spectral theorem](https://pressbooks.pub/linearalgebraandapplications/chapter/spectral-theorem/#chapter-73-section-2) 3. [29\.3. Rayleigh quotients](https://pressbooks.pub/linearalgebraandapplications/chapter/spectral-theorem/#chapter-73-section-3) 3. [30\. POSITIVE SEMI-DEFINITE MATRICES](https://pressbooks.pub/linearalgebraandapplications/chapter/positive-semi-definite-matrices/) 1. [30\.1. Definitions](https://pressbooks.pub/linearalgebraandapplications/chapter/positive-semi-definite-matrices/#chapter-74-section-1) 2. [30\.2. Special cases and examples](https://pressbooks.pub/linearalgebraandapplications/chapter/positive-semi-definite-matrices/#chapter-74-section-2) 3. [30\.3. Square root and Cholesky decomposition](https://pressbooks.pub/linearalgebraandapplications/chapter/positive-semi-definite-matrices/#chapter-74-section-3) 4. [30\.4. Ellipsoids](https://pressbooks.pub/linearalgebraandapplications/chapter/positive-semi-definite-matrices/#chapter-74-section-4) 4. [31\. PRINCIPAL COMPONENT ANALYSIS](https://pressbooks.pub/linearalgebraandapplications/chapter/principal-component-analysis/) 1. [31\.1. Projection on a line via variance maximization](https://pressbooks.pub/linearalgebraandapplications/chapter/principal-component-analysis/#chapter-75-section-1) 2. [31\.2. Principal component analysis](https://pressbooks.pub/linearalgebraandapplications/chapter/principal-component-analysis/#chapter-75-section-2) 3. [31\.3. Explained variance](https://pressbooks.pub/linearalgebraandapplications/chapter/principal-component-analysis/#chapter-75-section-3) 5. [32\. APPLICATIONS: PCA OF SENATE VOTING DATA](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-pca-of-senate-voting-data/) 1. [32\.1 Introduction](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-pca-of-senate-voting-data/#chapter-76-section-1) 2. [32\.2. Senate voting data and the visualization problem](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-pca-of-senate-voting-data/#chapter-76-section-2) 3. [32\.3. Projecting on a line](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-pca-of-senate-voting-data/#chapter-76-section-3) 4. [32\.4. Projecting on a plane](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-pca-of-senate-voting-data/#chapter-76-section-4) 5. [32\.5. Direction of maximal variance](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-pca-of-senate-voting-data/#chapter-76-section-5) 6. [32\.6. Principal component analysis](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-pca-of-senate-voting-data/#chapter-76-section-6) 7. [32\.7. Sparse PCA](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-pca-of-senate-voting-data/#chapter-76-section-7) 8. [32\.8 Sparse maximal variance problem](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-pca-of-senate-voting-data/#chapter-76-section-8) 6. [33\. EXERCISES](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-5/) 1. [33\.1. Interpretation of covariance matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-5/#chapter-77-section-1) 2. [33\.2. Eigenvalue decomposition](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-5/#chapter-77-section-2) 3. [33\.3. Positive-definite matrices, ellipsoids](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-5/#chapter-77-section-3) 4. [33\.4. Least-squares estimation](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-5/#chapter-77-section-4) 7. [VI. SINGULAR VALUES](https://pressbooks.pub/linearalgebraandapplications/part/chapter-vi-singular-values/) 1. [34\. THE SVD THEOREM](https://pressbooks.pub/linearalgebraandapplications/chapter/the-svd-theorem/) 1. [34\.1. The SVD theorem](https://pressbooks.pub/linearalgebraandapplications/chapter/the-svd-theorem/#chapter-82-section-1) 2. [34\.2. Geometry](https://pressbooks.pub/linearalgebraandapplications/chapter/the-svd-theorem/#chapter-82-section-2) 3. [34\.3. Link with the SED (Spectral Theorem)](https://pressbooks.pub/linearalgebraandapplications/chapter/the-svd-theorem/#chapter-82-section-3) 2. [35\. MATRIX PROPERTIES VIA SVD](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-properties-via-svd/) 1. [35\.1. Nullspace](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-properties-via-svd/#chapter-83-section-1) 2. [35\.2. Range, rank via the SVD](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-properties-via-svd/#chapter-83-section-2) 3. [35\.3. Fundamental theorem of linear algebra](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-properties-via-svd/#chapter-83-section-3) 4. [35\.4. Matrix norms, condition number](https://pressbooks.pub/linearalgebraandapplications/chapter/matrix-properties-via-svd/#chapter-83-section-4) 3. [36\. SOLVING LINEAR SYSTEMS VIA SVD](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-linear-systems-via-svd/) 1. [36\.1. Solution set](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-linear-systems-via-svd/#chapter-84-section-1) 2. [36\.2. Pseudo-inverse](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-linear-systems-via-svd/#chapter-84-section-2) 3. [36\.3. Sensitivity analysis and condition number](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-linear-systems-via-svd/#chapter-84-section-3) 4. [37\. LEAST-SQUARES AND SVD](https://pressbooks.pub/linearalgebraandapplications/chapter/least-squares-and-svd/) 1. [37\.1. Set of solutions](https://pressbooks.pub/linearalgebraandapplications/chapter/least-squares-and-svd/#chapter-85-section-1) 2. [37\.2. Sensitivity analysis](https://pressbooks.pub/linearalgebraandapplications/chapter/least-squares-and-svd/#chapter-85-section-2) 3. [37\.3. BLUE property](https://pressbooks.pub/linearalgebraandapplications/chapter/least-squares-and-svd/#chapter-85-section-3) 5. [38\. LOW-RANK APPROXIMATIONS](https://pressbooks.pub/linearalgebraandapplications/chapter/low-rank-approximations/) 1. [38\.1. Low-rank approximations](https://pressbooks.pub/linearalgebraandapplications/chapter/low-rank-approximations/#chapter-86-section-1) 6. [39\. APPLICATIONS](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-4/) 1. [39\.1 Image compression](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-4/#chapter-87-section-1) 2. [39\.2 Market data analysis](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-4/#chapter-87-section-2) 7. [40\. EXERCISES](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-6/) 1. [40\.1. SVD of simple matrices](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-6/#chapter-88-section-1) 2. [40\.2. Rank and SVD](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-6/#chapter-88-section-2) 3. [40\.3. Procrustes problem](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-6/#chapter-88-section-3) 4. [40\.4. SVD and projections](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-6/#chapter-88-section-4) 5. [40\.5. SVD and least-squares](https://pressbooks.pub/linearalgebraandapplications/chapter/exercises-6/#chapter-88-section-5) 8. VII. EXAMPLES 1. [Dimension of an affine subspace](https://pressbooks.pub/linearalgebraandapplications/chapter/examples/) 2. [Sample and weighted average](https://pressbooks.pub/linearalgebraandapplications/chapter/sample-and-weighted-average/) 3. [Sample average of vectors](https://pressbooks.pub/linearalgebraandapplications/chapter/sample-average-of-vectors/) 4. [Euclidean projection on a set](https://pressbooks.pub/linearalgebraandapplications/chapter/euclidean-project-on-a-set/) 5. [Orthogonal complement of a subspace](https://pressbooks.pub/linearalgebraandapplications/chapter/orthogonal-complement-of-a-subspace/) 6. [Power laws](https://pressbooks.pub/linearalgebraandapplications/chapter/power-laws/) 7. [Power law model fitting](https://pressbooks.pub/linearalgebraandapplications/chapter/power-law-model-fitting/) 8. [Definition: Vector norm](https://pressbooks.pub/linearalgebraandapplications/chapter/definition-vector-norm-2/) 9. [An infeasible linear system](https://pressbooks.pub/linearalgebraandapplications/chapter/an-infeasible-linear-system/) 10. [Sample variance and standard deviation](https://pressbooks.pub/linearalgebraandapplications/chapter/sample-variance-and-standard-deviation/) 11. [Functions and maps](https://pressbooks.pub/linearalgebraandapplications/chapter/functions-and-maps-2/) 1. [Functions](https://pressbooks.pub/linearalgebraandapplications/chapter/functions-and-maps-2/#chapter-727-section-1) 2. [Maps](https://pressbooks.pub/linearalgebraandapplications/chapter/functions-and-maps-2/#chapter-727-section-2) 12. [Dual norm](https://pressbooks.pub/linearalgebraandapplications/chapter/dual-norm/) 13. [Incidence matrix of a network](https://pressbooks.pub/linearalgebraandapplications/chapter/incidence-matrix-of-a-network-2/) 14. [Nullspace of a transpose incidence matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/nullspace-of-a-transpose-incidence-matrix/) 15. [Rank properties of the arc-node incidence matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/rank-properties-of-the-arc-node-incidence-matrix/) 16. [Permutation matrices](https://pressbooks.pub/linearalgebraandapplications/chapter/permutation-matrices/) 17. [QR decomposition: Examples](https://pressbooks.pub/linearalgebraandapplications/chapter/qr-decomposition-examples-2/) 18. [Backwards substitution for solving triangular linear systems.](https://pressbooks.pub/linearalgebraandapplications/chapter/backwards-substitution-for-solving-triangular-linear-systems/) 19. [Solving triangular systems of equations: Backwards substitution example](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-triangular-systems-of-equations-backwards-substitution-example/) 20. [Linear regression via least squares](https://pressbooks.pub/linearalgebraandapplications/chapter/linear-regression-via-least-squares-2/) 21. [Nomenclature](https://pressbooks.pub/linearalgebraandapplications/chapter/nomenclature-2/) 1. [Feasible set](https://pressbooks.pub/linearalgebraandapplications/chapter/nomenclature-2/#chapter-747-section-1) 2. [What is a solution?](https://pressbooks.pub/linearalgebraandapplications/chapter/nomenclature-2/#chapter-747-section-2) 3. [Local vs. global optimal points](https://pressbooks.pub/linearalgebraandapplications/chapter/nomenclature-2/#chapter-747-section-3) 22. [Standard forms](https://pressbooks.pub/linearalgebraandapplications/chapter/standard-forms-2/) 1. [Functional form](https://pressbooks.pub/linearalgebraandapplications/chapter/standard-forms-2/#chapter-749-section-1) 2. [Epigraph form](https://pressbooks.pub/linearalgebraandapplications/chapter/standard-forms-2/#chapter-749-section-2) 3. [Other standard forms](https://pressbooks.pub/linearalgebraandapplications/chapter/standard-forms-2/#chapter-749-section-3) 23. [A two-dimensional toy optimization problem](https://pressbooks.pub/linearalgebraandapplications/chapter/a-two-dimensional-toy-optimization-problem-2/) 24. [Global vs. local minima](https://pressbooks.pub/linearalgebraandapplications/chapter/global-vs-local-minima-2/) 25. [Gradient of a function](https://pressbooks.pub/linearalgebraandapplications/chapter/gradient-of-function/) 1. [Definition](https://pressbooks.pub/linearalgebraandapplications/chapter/gradient-of-function/#chapter-755-section-1) 2. [Composition rule with an affine function](https://pressbooks.pub/linearalgebraandapplications/chapter/gradient-of-function/#chapter-755-section-2) 3. [Geometric interpretation](https://pressbooks.pub/linearalgebraandapplications/chapter/gradient-of-function/#chapter-755-section-3) 26. [Set of solutions to the least-squares problem via QR decomposition](https://pressbooks.pub/linearalgebraandapplications/chapter/set-of-solutions-to-the-least-squares-problem-via-qr-decomposition-2/) 27. [Sample covariance matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/sample-covariance-matrix-2/) 1. [Definition](https://pressbooks.pub/linearalgebraandapplications/chapter/sample-covariance-matrix-2/#chapter-760-section-1) 2. [Properties](https://pressbooks.pub/linearalgebraandapplications/chapter/sample-covariance-matrix-2/#chapter-760-section-2) 28. [Optimal set of least-squares via SVD](https://pressbooks.pub/linearalgebraandapplications/chapter/optimal-set-of-least-squares-via-svd/) 29. [Pseudo-inverse of a matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/pseudo-inverse-of-a-matrix-2/) 30. [SVD: A 4x4 example](https://pressbooks.pub/linearalgebraandapplications/chapter/svd-a-4x4-example/) 31. [Singular value decomposition of a 4 x 5 matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/singular-value-decomposition-of-a-4-x-5-matrix/) 32. [Representation of a two-variable quadratic function](https://pressbooks.pub/linearalgebraandapplications/chapter/representation-of-a-two-variable-quadratic-function/) 33. [Edge weight matrix of a graph](https://pressbooks.pub/linearalgebraandapplications/chapter/edge-weight-matrix-of-a-graph-2/) 34. [Network flow](https://pressbooks.pub/linearalgebraandapplications/chapter/network-flow-2/) 35. [Laplacian matrix of a graph](https://pressbooks.pub/linearalgebraandapplications/chapter/laplacian-matrix-of-a-graph/) 36. [Hessian of a function](https://pressbooks.pub/linearalgebraandapplications/chapter/hessian-of-a-function-2/) 1. [Definition](https://pressbooks.pub/linearalgebraandapplications/chapter/hessian-of-a-function-2/#chapter-780-section-1) 2. [Examples](https://pressbooks.pub/linearalgebraandapplications/chapter/hessian-of-a-function-2/#chapter-780-section-2) 37. [Hessian of a quadratic function](https://pressbooks.pub/linearalgebraandapplications/chapter/hessian-of-a-quadratic-function-2/) 38. [Gram matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/gram-matrix/) 39. [Quadratic functions in two variables](https://pressbooks.pub/linearalgebraandapplications/chapter/quadratic-functions-in-two-variables-2/) 40. [Quadratic approximation of the log-sum-exp function](https://pressbooks.pub/linearalgebraandapplications/chapter/quadratic-approximation-of-the-log-sum-exp-function-2/) 41. [Determinant of a square matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/determinant-of-a-square-matrix-2/) 1. [Definition](https://pressbooks.pub/linearalgebraandapplications/chapter/determinant-of-a-square-matrix-2/#chapter-791-section-1) 2. [Important result](https://pressbooks.pub/linearalgebraandapplications/chapter/determinant-of-a-square-matrix-2/#chapter-791-section-2) 3. [Some properties](https://pressbooks.pub/linearalgebraandapplications/chapter/determinant-of-a-square-matrix-2/#chapter-791-section-3) 42. [A squared linear function](https://pressbooks.pub/linearalgebraandapplications/chapter/a-squared-linear-function-2/) 43. [Eigenvalue decomposition of a symmetric matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/eigenvalue-decomposition-of-a-symmetric-matrix-2/) 44. [Rayleigh quotients](https://pressbooks.pub/linearalgebraandapplications/chapter/rayleigh-quotients/) 45. [Largest singular value norm of a matrix](https://pressbooks.pub/linearalgebraandapplications/chapter/largest-singular-value-norm-of-a-matrix/) 46. [Nullspace of a 4x5 matrix via its SVD](https://pressbooks.pub/linearalgebraandapplications/chapter/nullspace-of-a-4x5-matrix-via-its-svd/) 47. [Range of a 4x5 matrix via its SVD](https://pressbooks.pub/linearalgebraandapplications/chapter/range-of-a-4x5-matrix-via-its-svd/) 48. [Low-rank approximation of a 4x5 matrix via its SVD](https://pressbooks.pub/linearalgebraandapplications/chapter/low-rank-approximation-of-a-4x5-matrix-via-its-svd/) 49. [Pseudo-inverse of a 4x5 matrix via its SVD](https://pressbooks.pub/linearalgebraandapplications/chapter/pseudo-inverse-of-a-4x5-matrix-via-its-svd/) 9. VIII. APPLICATIONS 1. [Image compression via least-squares](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-5/) 2. [Senate voting data matrix.](https://pressbooks.pub/linearalgebraandapplications/chapter/senate-voting-data-matrix-2/) 3. [Senate voting analysis and visualization.](https://pressbooks.pub/linearalgebraandapplications/chapter/senate-voting-analysis-and-visualization-2/) 4. [Beer-Lambert law in absorption spectrometry](https://pressbooks.pub/linearalgebraandapplications/chapter/beer-lambert-law-in-absorption-spectrometry-2/) 5. [Absorption spectrometry: Using measurements at different light frequencies.](https://pressbooks.pub/linearalgebraandapplications/chapter/absorption-spectrometry-using-measurements-at-different-light-frequencies/) 6. [Similarity of two documents](https://pressbooks.pub/linearalgebraandapplications/chapter/similarity-of-two-documents/) 7. [Image compression](https://pressbooks.pub/linearalgebraandapplications/chapter/image-compression-2/) 8. [Temperatures at different airports](https://pressbooks.pub/linearalgebraandapplications/chapter/temperatures-at-different-airports-2/) 9. [Navigation by range measurement](https://pressbooks.pub/linearalgebraandapplications/chapter/navigation-by-range-measurement-2/) 10. [Bag-of-words representation of text](https://pressbooks.pub/linearalgebraandapplications/chapter/bag-of-words-representation-of-text-2/) 11. [Bag-of-words representation of text: Measure of document similarity](https://pressbooks.pub/linearalgebraandapplications/chapter/bag-of-words-representation-of-text-measure-of-document-similarity/) 12. [Rate of return of a financial portfolio](https://pressbooks.pub/linearalgebraandapplications/chapter/rate-of-return-of-financial-portfolio/) 1. [Rate of return of a single asset](https://pressbooks.pub/linearalgebraandapplications/chapter/rate-of-return-of-financial-portfolio/#chapter-834-section-1) 2. [Log-returns](https://pressbooks.pub/linearalgebraandapplications/chapter/rate-of-return-of-financial-portfolio/#chapter-834-section-2) 3. [Rate of return of a portfolio](https://pressbooks.pub/linearalgebraandapplications/chapter/rate-of-return-of-financial-portfolio/#chapter-834-section-3) 13. [Single factor model of financial price data](https://pressbooks.pub/linearalgebraandapplications/chapter/single-factor-model-of-financial-price-data/) 14. [The problem of Gauss](https://pressbooks.pub/linearalgebraandapplications/chapter/the-problem-of-gauss/) 15. [Control of a unit mass](https://pressbooks.pub/linearalgebraandapplications/chapter/control-of-unit-mass/) 16. [Portfolio optimization via linearly constrained least-squares.](https://pressbooks.pub/linearalgebraandapplications/chapter/portfolio-optimization-via-linearly-constrained-least-squares-2/) 10. IX. THEOREMS 1. [Cauchy-Schwarz inequality proof](https://pressbooks.pub/linearalgebraandapplications/chapter/cauchy-schwarz-inequality-proof/) 2. [Dimension of hyperplanes](https://pressbooks.pub/linearalgebraandapplications/chapter/dimension-of-hyperplanes/) 3. [Spectral theorem: Eigenvalue decomposition for symmetric matrices](https://pressbooks.pub/linearalgebraandapplications/chapter/spectral-theorem-eigenvalue-decomposition-for-symmetric-matrices-2/) 4. [Singular value decomposition (SVD) theorem](https://pressbooks.pub/linearalgebraandapplications/chapter/singular-value-decomposition-svd-theorem-2/) 5. [Rank-one matrices](https://pressbooks.pub/linearalgebraandapplications/chapter/rank-one-matrices-2/) 6. [Rank-one matrices: A representation theorem](https://pressbooks.pub/linearalgebraandapplications/chapter/rank-one-matrices-a-representation-theorem-2/) 7. [Full rank matrices](https://pressbooks.pub/linearalgebraandapplications/chapter/full-rank-matrices/) 8. [Rank-nullity theorem](https://pressbooks.pub/linearalgebraandapplications/chapter/rank-nullity-theorem-2/) 9. [A theorem on positive semidefinite forms and eigenvalues](https://pressbooks.pub/linearalgebraandapplications/chapter/a-theorem-on-positive-semidefinitive-forms-and-eigenvalues/) 10. [Fundamental theorem of linear algebra](https://pressbooks.pub/linearalgebraandapplications/chapter/fundamental-theorem-of-linear-algebra-2/) 11. [X. AI Learn](https://pressbooks.pub/linearalgebraandapplications/part/ai-learn/) # [Linear Algebra and Applications](https://pressbooks.pub/linearalgebraandapplications/) [Buy](https://pressbooks.pub/linearalgebraandapplications/buy/) # 23 ORDINARY LEAST-SQUARES - [Definition](https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/#chapter-66-section-1) - [Interpretations](https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/#chapter-66-section-2) - [Solution via QR decomposition (full rank case)](https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/#chapter-66-section-3) - [Optimal solution (general case)](https://pressbooks.pub/linearalgebraandapplications/chapter/ordinary-least-squares/#chapter-66-section-4) ## # 23\.1. Definition The Ordinary Least-Squares (OLS, or LS) problem is defined as ![\\\[ \\min\_x \\\\|A x - y\\\\|\_2^2 \\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-5bd0471818070b19f7583c98d8b55ec8_l3.png) where ![A \\in \\mathbb{R}^{m \\times n}, \\; y \\in \\mathbb{R}^m](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-cf13aa7cfaa4d59b455f8b49c9f800c1_l3.png) are given. Together, the pair ![(A, y)](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-58f05f4a7a2602759ede7a7aa013defb_l3.png) is referred to as the problem data. The vector ![y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) is often referred to as the ‘‘measurement” or “output” vector, and the data matrix ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) as the ‘‘design‘‘ or ‘‘input‘‘ matrix. The vector ![r:=y-A x](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-f095a0905d773a419f8e00325d188c66_l3.png) is referred to as the residual error vector. Note that the problem is equivalent to one where the norm is not squared. Taking the squares is done for the convenience of the solution. # 23\.2. Interpretations ## Interpretation as projection on the range \| \| \| \|---\|---\| \| [![](https://pressbooks.pub/app/uploads/sites/12458/2023/09/Image14b.svg)](https://pressbooks.pub/app/uploads/sites/12458/2023/09/Image14b.svg) \| We can interpret the problem in terms of the columns of ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png), as follows. Assume that ![A=\\left\[a\_1, \\ldots, a\_n\\right\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-85063cf6b2a138d9eac87d4d665d3190_l3.png), where ![a\_j \\in \\mathbb{R}^m](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-bf42b2f5b7d6bb663c613821a8326585_l3.png) is the ![j](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-43c82d5bb00a7568d935a12e3bd969dd_l3.png)\-th column of ![A \\; (j=1, \\ldots, n)](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-d8445476ac0186f1504b5590f6640870_l3.png). The problem reads \| \| ![\\\[\\min \_x\\left\\\\|\\sum\_{j=1}^n x\_j a\_j-y\\right\\\\|\_2 .\\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-9991a188d7e7374b521755f7b245a599_l3.png) \| \| \| In this sense, we are trying to find the best approximation of ![y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) in terms of a linear combination of the columns of ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png). Thus, the OLS problem amounts to project (find the minimum Euclidean distance) the vector ![y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) on the span of the vectors ![a\_j](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-e205fc6b45b906f467e64c8105604f2e_l3.png) ‘s (that is to say: the range of ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png)). \| \| \| As seen in the picture, at optimum the residual vector ![A x-y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-759e1f254cc24999796dca49b37126d8_l3.png) is orthogonal to the range of ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png). \| \| See also: [Image compression via least-squares.](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-5/) ## Interpretation as minimum distance to feasibility The OLS problem is usually applied to problems where the linear ![Ax=y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-ef9b2318eeb07429fb9de17d2fb480d2_l3.png) is not feasible, that is, there is no solution to ![Ax=y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-ef9b2318eeb07429fb9de17d2fb480d2_l3.png). The OLS can be interpreted as finding the smallest (in Euclidean norm sense) perturbation of the right-hand side, ![\\delta y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-d03ae29673df3c029780366f946c6365_l3.png), such that the linear equation ![\\\[A x=y+\\delta y\\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-d3a4458cd9e22689e0e143d24a6438cc_l3.png) becomes feasible. In this sense, the OLS formulation implicitly assumes that the data matrix ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) of the problem is known exactly, while only the right-hand side is subject to perturbation, or measurement errors. A more elaborate model, total least-squares, takes into account errors in both ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) and ![y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png). ## Interpretation as regression We can also interpret the problem in terms of the rows of ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png), as follows. Assume that ![A^T=\\left\[a\_1, \\ldots, a\_m\\right\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-12b4c958054b51d74064f4eece5d232a_l3.png), where ![a\_i \\in \\mathbb{R}^{n}](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-719241859d7e38a4b8d47c5c3a6b89b9_l3.png) is the ![i](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-695d9d59bd04859c6c99e7feb11daab6_l3.png)\-th row of ![A \\;\\; (i=1, \\ldots, m)](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-c93ea69bac711703213ed1a70b7cbd95_l3.png). The problem reads ![\\\[\\min \_x \\sum\_{i=1}^m\\left(y\_i-a\_i^T x\\right)^2 .\\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-4a5056353ea968efd379e142105e803a_l3.png) In this sense, we are trying to fit each component of ![y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) as a linear combination of the corresponding input ![a\_i](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-f91083f3035e5168a6f0b3e6335d6858_l3.png), with ![x](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-ede05c264bba0eda080918aaa09c4658_l3.png) as the coefficients of this linear combination. See also: - [Linear regression.](https://pressbooks.pub/linearalgebraandapplications/chapter/linear-regression-via-least-squares/) - [Auto-regressive models for time series prediction](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-3/#chapter-69-section-2). - [Power law model fitting.](https://pressbooks.pub/linearalgebraandapplications/chapter/power-law-model-fitting/) # 23\.3. Solution via QR decomposition (full rank case) Assume that the matrix ![A \\in \\mathbb{R}^{m \\times n}](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-ab44a6e443678ee25caa73021f1128e5_l3.png) is tall (![m \\geq n](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-0f33e5f65664c8347d9ff5217b0362ba_l3.png)) and full column rank. Then the solution to the problem is unique and given by ![\\\[x^\=\\left(A^T A\\right)^{-1} A^T y .\\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-e9b9d24d2820056a4a50245137bc6906_l3.png) This can be seen by simply taking the [gradient](https://pressbooks.pub/linearalgebraandapplications/chapter/gradient-of-function/) (vector of derivatives) of the objective function, which leads to the optimality condition ![A^T(A x-y)=0](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-62b1fdaa42792ca372332edfbb350b93_l3.png). Geometrically, the residual vector ![A x-y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-759e1f254cc24999796dca49b37126d8_l3.png) is orthogonal to the span of the columns of ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png), as seen in the picture above. We can also prove this via the [QR decomposition](https://pressbooks.pub/linearalgebraandapplications/chapter/qr-decomposition-of-a-matrix/) of the matrix ![A: A=Q R](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-ed471941fd66aa841ad8410d218cd07e_l3.png) with ![Q](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-2c758bec4c272382411b95fc0e7ee250_l3.png) a ![m \\times n](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-7d0c5e4f172b3def521d0e4c97406eed_l3.png) matrix with orthonormal columns ( ![Q^T Q=I\_n](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-171ecf61e543ebd2ace369733aa2f26a_l3.png) ) and ![R](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-dae6bae3dcdac4629730754352c5e329_l3.png) a ![n \\times n](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-d5c3031c3a485131fabc8d0972e90f76_l3.png) upper-triangular, invertible matrix. Noting that ![\\begin{align\} \\\\|A x-y\\\\|\_2^2 & = x^T A^T A x - 2 x^T A^T y + y^T y \\\\ & = x^T R^T R x - 2 x^T R^T Q^T y + y^T y \\\\ & = \\left\\\\|R x-Q^T y\\right\\\\|\_2^2 + y^T\\left(I-Q Q^T\\right) y \\end{align\}](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-983b7a914be8e937ef2d04c0fa2b9de0_l3.png) and exploiting the fact that ![R](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-dae6bae3dcdac4629730754352c5e329_l3.png) is invertible, we obtain the optimal solution ![x^\=R^{-1} Q^T y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-d22d352a05a3a1b01b456d7d27a53c4e_l3.png). This is the same as the formula above, since ![\\\[\\left(A^T A\\right)^{-1} A^T y=\\left(R^T Q^T Q R\\right)^{-1} R^T Q^T y=\\left(R^T R\\right)^{-1} R^T Q^T y=R^{-1} Q^T y .\\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-026a92d4e54e678b6058e0bf35b41dd5_l3.png) Thus, to find the solution based on the QR decomposition, we just need to implement two steps: 1. Rotate the output vector: set ![\\bar{y}=Q^T y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-d3952cd3ed4956c497c4b6b23e38bdbd_l3.png). 2. Solve the triangular system ![R x=\\bar{y}](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-aa9c97f23d5499bb1ccde4164324347b_l3.png) by [backward substitution](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-triangular-systems-of-equations-backwards-substitution-example/). # 23\.4. Optimal solution and optimal set Recall that the [optimal set](https://pressbooks.pub/linearalgebraandapplications/chapter/global-vs-local-minima/) of a minimization problem is its set of minimizers. For least-squares problems, the optimal set is an affine set, which reduces to a singleton when ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) is full column rank. In the general case (![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) is not necessarily tall, and /or not full rank) then the solution may not be unique. If ![x^0](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-7c37f91603121b8b9fd93982223c4445_l3.png) is a particular solution, then ![x=x^0+z](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-9aae2736ef0c40a87dd2feb50cfb15a9_l3.png) is also a solution, if ![z](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-4586e340cb83d5b642972e97a288fec2_l3.png) is such that ![A z=0](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-295470b91e02b1abda500af0ecb4e2ec_l3.png), that is, ![z \\in \\mathbf{N}(A)](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-19a4654afae0e46f3dd5c8d6c72027ae_l3.png). That is, the nullspace of ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) describes the ambiguity of solutions. In mathematical terms: ![\\\[ \\underset{x}{\\text{argmin}} \\\\|A x-y\\\\|\_2 = x^0 + \\mathbf{N}(A) . \\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-9c1906c394f150dd46f4a4e5f402f9eb_l3.png) The formal expression for the set of minimizers to the least-squares problem can be found again via the QR decomposition. This is shown [here](https://pressbooks.pub/linearalgebraandapplications/chapter/set-of-solutions-to-the-least-squares-problem-via-qr-decomposition/). Previous/next navigation [Previous: LEAST-SQUARES](https://pressbooks.pub/linearalgebraandapplications/part/chapter-iv-least-squares/) [Next: VARIANTS OF THE LEAST-SQUARES PROBLEM](https://pressbooks.pub/linearalgebraandapplications/chapter/variants-of-the-least-squares-problem/) Back to top ## License ![Icon for the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License](https://pressbooks.pub/app/themes/pressbooks-book/packages/buckram/assets/images/cc-by-nc-nd.svg) [Linear Algebra and Applications](https://pressbooks.pub/linearalgebraandapplications) Copyright © 2023 by VinUiversity is licensed under a [Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License](https://creativecommons.org/licenses/by-nc-nd/4.0/), except where otherwise noted. ## Share This Book [Pressbooks](https://pressbooks.com/) [Powered by Pressbooks](https://pressbooks.com/) - [Pressbooks User Guide](https://guide.pressbooks.com/) - \\|[Pressbooks Directory](https://pressbooks.directory/) - \\|[Contact](https://pressbooks.com/support/) [Pressbooks on YouTube](https://www.youtube.com/user/pressbooks) [Pressbooks on LinkedIn](https://www.linkedin.com/company/pressbooks/?originalSubdomain=ca)
Readable Markdown	The Ordinary Least-Squares (OLS, or LS) problem is defined as ![\\\[ \\min\_x \\\\|A x - y\\\\|\_2^2 \\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-5bd0471818070b19f7583c98d8b55ec8_l3.png) where ![A \\in \\mathbb{R}^{m \\times n}, \\; y \\in \\mathbb{R}^m](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-cf13aa7cfaa4d59b455f8b49c9f800c1_l3.png) are given. Together, the pair ![(A, y)](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-58f05f4a7a2602759ede7a7aa013defb_l3.png) is referred to as the problem data. The vector ![y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) is often referred to as the ‘‘measurement” or “output” vector, and the data matrix ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) as the ‘‘design‘‘ or ‘‘input‘‘ matrix. The vector ![r:=y-A x](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-f095a0905d773a419f8e00325d188c66_l3.png) is referred to as the residual error vector. Note that the problem is equivalent to one where the norm is not squared. Taking the squares is done for the convenience of the solution. ## Interpretation as projection on the range See also: [Image compression via least-squares.](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-5/) ## Interpretation as minimum distance to feasibility The OLS problem is usually applied to problems where the linear ![Ax=y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-ef9b2318eeb07429fb9de17d2fb480d2_l3.png) is not feasible, that is, there is no solution to ![Ax=y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-ef9b2318eeb07429fb9de17d2fb480d2_l3.png). The OLS can be interpreted as finding the smallest (in Euclidean norm sense) perturbation of the right-hand side, ![\\delta y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-d03ae29673df3c029780366f946c6365_l3.png), such that the linear equation ![\\\[A x=y+\\delta y\\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-d3a4458cd9e22689e0e143d24a6438cc_l3.png) becomes feasible. In this sense, the OLS formulation implicitly assumes that the data matrix ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) of the problem is known exactly, while only the right-hand side is subject to perturbation, or measurement errors. A more elaborate model, total least-squares, takes into account errors in both ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) and ![y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png). ## Interpretation as regression We can also interpret the problem in terms of the rows of ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png), as follows. Assume that ![A^T=\\left\[a\_1, \\ldots, a\_m\\right\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-12b4c958054b51d74064f4eece5d232a_l3.png), where ![a\_i \\in \\mathbb{R}^{n}](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-719241859d7e38a4b8d47c5c3a6b89b9_l3.png) is the ![i](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-695d9d59bd04859c6c99e7feb11daab6_l3.png)\-th row of ![A \\;\\; (i=1, \\ldots, m)](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-c93ea69bac711703213ed1a70b7cbd95_l3.png). The problem reads ![\\\[\\min \_x \\sum\_{i=1}^m\\left(y\_i-a\_i^T x\\right)^2 .\\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-4a5056353ea968efd379e142105e803a_l3.png) In this sense, we are trying to fit each component of ![y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-0af556714940c351c933bba8cf840796_l3.png) as a linear combination of the corresponding input ![a\_i](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-f91083f3035e5168a6f0b3e6335d6858_l3.png), with ![x](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-ede05c264bba0eda080918aaa09c4658_l3.png) as the coefficients of this linear combination. See also: - [Linear regression.](https://pressbooks.pub/linearalgebraandapplications/chapter/linear-regression-via-least-squares/) - [Auto-regressive models for time series prediction](https://pressbooks.pub/linearalgebraandapplications/chapter/applications-3/#chapter-69-section-2). - [Power law model fitting.](https://pressbooks.pub/linearalgebraandapplications/chapter/power-law-model-fitting/) Assume that the matrix ![A \\in \\mathbb{R}^{m \\times n}](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-ab44a6e443678ee25caa73021f1128e5_l3.png) is tall (![m \\geq n](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-0f33e5f65664c8347d9ff5217b0362ba_l3.png)) and full column rank. Then the solution to the problem is unique and given by ![\\\[x^\=\\left(A^T A\\right)^{-1} A^T y .\\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-e9b9d24d2820056a4a50245137bc6906_l3.png) This can be seen by simply taking the [gradient](https://pressbooks.pub/linearalgebraandapplications/chapter/gradient-of-function/) (vector of derivatives) of the objective function, which leads to the optimality condition ![A^T(A x-y)=0](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-62b1fdaa42792ca372332edfbb350b93_l3.png). Geometrically, the residual vector ![A x-y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-759e1f254cc24999796dca49b37126d8_l3.png) is orthogonal to the span of the columns of ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png), as seen in the picture above. We can also prove this via the [QR decomposition](https://pressbooks.pub/linearalgebraandapplications/chapter/qr-decomposition-of-a-matrix/) of the matrix ![A: A=Q R](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-ed471941fd66aa841ad8410d218cd07e_l3.png) with ![Q](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-2c758bec4c272382411b95fc0e7ee250_l3.png) a ![m \\times n](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-7d0c5e4f172b3def521d0e4c97406eed_l3.png) matrix with orthonormal columns ( ![Q^T Q=I\_n](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-171ecf61e543ebd2ace369733aa2f26a_l3.png) ) and ![R](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-dae6bae3dcdac4629730754352c5e329_l3.png) a ![n \\times n](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-d5c3031c3a485131fabc8d0972e90f76_l3.png) upper-triangular, invertible matrix. Noting that ![\\begin{align\} \\\\|A x-y\\\\|\_2^2 & = x^T A^T A x - 2 x^T A^T y + y^T y \\\\ & = x^T R^T R x - 2 x^T R^T Q^T y + y^T y \\\\ & = \\left\\\\|R x-Q^T y\\right\\\\|\_2^2 + y^T\\left(I-Q Q^T\\right) y \\end{align\}](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-983b7a914be8e937ef2d04c0fa2b9de0_l3.png) and exploiting the fact that ![R](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-dae6bae3dcdac4629730754352c5e329_l3.png) is invertible, we obtain the optimal solution ![x^\=R^{-1} Q^T y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-d22d352a05a3a1b01b456d7d27a53c4e_l3.png). This is the same as the formula above, since ![\\\[\\left(A^T A\\right)^{-1} A^T y=\\left(R^T Q^T Q R\\right)^{-1} R^T Q^T y=\\left(R^T R\\right)^{-1} R^T Q^T y=R^{-1} Q^T y .\\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-026a92d4e54e678b6058e0bf35b41dd5_l3.png) Thus, to find the solution based on the QR decomposition, we just need to implement two steps: 1. Rotate the output vector: set ![\\bar{y}=Q^T y](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-d3952cd3ed4956c497c4b6b23e38bdbd_l3.png). 2. Solve the triangular system ![R x=\\bar{y}](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-aa9c97f23d5499bb1ccde4164324347b_l3.png) by [backward substitution](https://pressbooks.pub/linearalgebraandapplications/chapter/solving-triangular-systems-of-equations-backwards-substitution-example/). Recall that the [optimal set](https://pressbooks.pub/linearalgebraandapplications/chapter/global-vs-local-minima/) of a minimization problem is its set of minimizers. For least-squares problems, the optimal set is an affine set, which reduces to a singleton when ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) is full column rank. In the general case (![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) is not necessarily tall, and /or not full rank) then the solution may not be unique. If ![x^0](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-7c37f91603121b8b9fd93982223c4445_l3.png) is a particular solution, then ![x=x^0+z](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-9aae2736ef0c40a87dd2feb50cfb15a9_l3.png) is also a solution, if ![z](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-4586e340cb83d5b642972e97a288fec2_l3.png) is such that ![A z=0](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-295470b91e02b1abda500af0ecb4e2ec_l3.png), that is, ![z \\in \\mathbf{N}(A)](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-19a4654afae0e46f3dd5c8d6c72027ae_l3.png). That is, the nullspace of ![A](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-25b206f25506e6d6f46be832f7119ffa_l3.png) describes the ambiguity of solutions. In mathematical terms: ![\\\[ \\underset{x}{\\text{argmin}} \\\\|A x-y\\\\|\_2 = x^0 + \\mathbf{N}(A) . \\\]](https://pressbooks.pub/app/uploads/quicklatex/quicklatex.com-9c1906c394f150dd46f4a4e5f402f9eb_l3.png) The formal expression for the set of minimizers to the least-squares problem can be found again via the QR decomposition. This is shown [here](https://pressbooks.pub/linearalgebraandapplications/chapter/set-of-solutions-to-the-least-squares-problem-via-qr-decomposition/).
Shard	45 (laksa)
Root Hash	2282000291418685645
Unparsed URL	pub,pressbooks!/linearalgebraandapplications/chapter/ordinary-least-squares/ s443