What is an eigenvector of a covariance matrix?

One of the most intuitive explanations of eigenvectors of a covariance matrix is that they are the directions in which the data varies the most.

(More precisely, the first eigenvector is the direction in which the data varies the most, the second eigenvector is the direction of greatest variance among those that are orthogonal (perpendicular) to the first eigenvector, the third eigenvector is the direction of greatest variance among those orthogonal to the first two, and so on.)

Here is an example in 2 dimensions [1]:

Each data sample is a 2 dimensional point with coordinates x, y. The eigenvectors of the covariance matrix of these data samples are the vectors u and v; u, longer arrow, is the first eigenvector and v, the shorter arrow, is the second. (The eigenvalues are the length of the arrows.) As you can see, the first eigenvector points (from the mean of the data) in the direction in which the data varies the most in Euclidean space, and the second eigenvector is orthogonal (perpendicular) to the first.

It's a little trickier to visualize in 3 dimensions, but here's an attempt [2]:

In this case, imagine that all of the data points lie within the ellipsoid. v1, the direction in which the data varies the most, is the first eigenvector (lambda1 is the corresponding eigenvalue). v2 is the direction in which the data varies the most among those directions that are orthogonal to v1. And v3 is the direction of greatest variance among those directions that are orthogonal to v1 and v2 (though there is only one such orthogonal direction).

[1] Image taken from Duncan Gillies's lecture on Principal Component Analysis
[2] Image taken from Fiber Crossing in Human Brain Depicted with Diffusion Tensor MR Imaging

  
Written 1 Aug, 2013 • 48,160 views
 
Given a set of random variables {x1,...,xn}, the covariance matrix A is defined so that Ai,j=Cov(xi,xj). We can represent a linear combination ∑bixi as a vector x=(b1,...,bn).

It turns out that the covariance of two such vectors x and y can be written as Cov(x,y)=xtAy. In particular, Var(x)=xtAx. This means that covariance is a Bilinear form.

Now, since A is a real symmetric matrix, there is an orthonormal basis for Rnof eigenvectors of A. Orthonormal in this case means that each vector's norm is 1 and they're orthogonal with respect to A, that is vt1Av2=0, or Cov(v1,v2)=0.

Next, suppose v is a unit eigenvector of A with eigenvalue λ. Then Var(v)=λ∥v∥2=λ.

There are a couple interesting conclusions we can draw from this. First, since the eigenvectors form a basis {v1,...,vn}, every linear combination of the original random variables can actually be represented as a linear combination of the independent random variables vi. Second, every unit vector's variance is a weighted average of the eigenvalues. This means that the leading eigenvector is the direction of greatest variance, the next eigenvector has the greatest variance in the orthogonal subspace, and so on.

So, sum up, eigenvectors are uncorrelated linear combinations of the original set of random variables.

The primary application of this is Principal Components Analysis. If you have n features, you can find eigenvectors of the covariance matrix of the features. This allows you to represent the data with uncorrelated features. Moreover, the eigenvalues tell you the amount of variance in each feature, allowing you to choose a subset of the features that retain the most information about your data.

  
Written 28 Nov, 2013 • 5,789 views
 
The largest eigenvector of a covariance matrix points into the direction of the largest variance. All other eigenvectors are orthogonal to the largest one.

Now, if this direction of the largest variance is axis-aligned (covariances are zero), then the eigenvalues simply correspond to the variances of the data:

It becomes a little more complicated if the covariance matrix is not diagonal, such that the covariances are not zero. In this case, the principal components (directions of largest variance) do no coincide with the axes, and the data is rotated. The eigenvalues then still correspond to the spread of the data in the direction of the largest variance, whereas the variance components of the covariance matrix still defines the spread of the data along the axes:

An in-depth discussion of how the covariance matrix can be interpreted from a geometric point of view (and the source of the above images) can be found on:A geometric interpretation of the covariance matrix

  
Written 10 Mar • 2,674 views
Shreyas Ghuge

3 upvotes by Sameer Gupta, Anonymous, and Ram Shankar
 
Finding the directions of maximum and minimum variance is the same as looking for where the orthogonal least squares best fit line and plane of the data. The sums of squares for that line and plane can be written in terms of covariance matrix.The connections between them can be worked out to get the Eigen vectors of this covariance matrix.

  
Written 22 Aug, 2013 • 4,598 views
Julius Bier Kirkegaard, physics, computers, 'n' stuff

4 upvotes by David Joyce (Professor of Mathematics at Clark University), Andrei Kucharavy (PhD student in Bioinformatics), Martin Andrews, and Laasya Alamuru
 
Finding the eigenvectors a covariance matrix is exactly the technique of Principal Component Analysis (PCA).

The eigenvectors are those variables that are linearly uncorrelated.

  
Written 1 Aug, 2013 • 4,307 views
 
 
Write an answer
Related Questions
 

What is an eigenvector of a covariance matrix?的更多相关文章

  1. A geometric interpretation of the covariance matrix

    A geometric interpretation of the covariance matrix Contents [hide] 1 Introduction 2 Eigendecomposit ...

  2. 方差variance, 协方差covariance, 协方差矩阵covariance matrix

    https://www.jianshu.com/p/e1c8270477bc?utm_campaign=maleskine&utm_content=note&utm_medium=se ...

  3. 方差variance, 协方差covariance, 协方差矩阵covariance matrix | scatter matrix | weighted covariance | Eigenvalues and eigenvectors

    covariance, co本能的想到双变量,用于描述两个变量之间的关系. correlation,相关性,covariance标准化后就是correlation. covariance的定义: 期望 ...

  4. covariance matrix 和数据分布情况估计

    how to get data covariance matrix: http://stattrek.com/matrix-algebra/covariance-matrix.aspx meaning ...

  5. 图Lasso求逆协方差矩阵(Graphical Lasso for inverse covariance matrix)

    图Lasso求逆协方差矩阵(Graphical Lasso for inverse covariance matrix) 作者:凯鲁嘎吉 - 博客园 http://www.cnblogs.com/ka ...

  6. 随机变量的方差variance & 随机向量的协方差矩阵covariance matrix

    1.样本矩阵 如果是一个随机变量,那么它的样本值可以用一个向量表示.相对的,如果针对一个随机向量,那么就需要利用矩阵表示,因为向量中的每一个变量的采样值,都可以利用一个向量表示. 然后,一个矩阵可以利 ...

  7. A Beginner’s Guide to Eigenvectors, PCA, Covariance and Entropy

    A Beginner’s Guide to Eigenvectors, PCA, Covariance and Entropy Content: Linear Transformations Prin ...

  8. Ill-conditioned covariance create

    http://www.mathworks.com/matlabcentral/answers/100210-why-do-i-receive-an-error-while-trying-to-gene ...

  9. 协方差(Covariance)

    统计学上用方差和标准差来度量数据的离散程度 ,但是方差和标准差是用来描述一维数据的(或者说是多维数据的一个维度),现实生活中我们常常会碰到多维数据,因此人们发明了协方差(covariance),用来度 ...

随机推荐

  1. [转载]SQL语句中的日期计算

    1. 本月的第一天SELECT  DATEADD(mm,  DATEDIFF(mm,0,getdate()),  0) 2. 本月的最后一天SELECT  dateadd(ms,-3,DATEADD( ...

  2. springMVC--@requestBody

    springMVC支持将前端传来的json字符串直接自动解析,注意点如下: 后台 1,在需要自动解析的参数前加上“@requestBody”,例如“public boolean updateOnePd ...

  3. Cookie——Javascript

    基本的cookie操作有三种:读取.写入.删除. var CookieUtil = { //根据cookie的name获取相应的value.它会在document.cookie字符串中查找cookie ...

  4. OpenJudge计算概论-找最大数序列

    /*===================================== 找最大数序列 总时间限制: 1000ms 内存限制: 65536kB 描述 输入n行(n 不大于 30),每行不超过10 ...

  5. linux静态与动态库创建及使用实例

    一,gcc基础语法: 基本语法结构:(由以下四部分组成) gcc -o 可执行文件名 依赖文件集(*.c/*.o) 依赖库文件及其头文件集(由-I或-L与-l指明) gcc 依赖文件集(*.c/*.o ...

  6. table表格中单击添加动态编辑框

    var $newNode=$("<input type='text' style='width:250px; height:20px; maxlength='20' id='texti ...

  7. PHP 5.3中的命名空间使用方法

    PHP的命名空间(namespace)是php5.3之后才有的.这个概念在C#中已经很早就有了,php中的namespace其实和c#的概念是一样的. 为什么php中要使用namespace? 假设如 ...

  8. PSexec以及xcopy的简单使用

    1. 远程执行命令. 有时候不想远程但是想执行一些命令, 比较简单的方法是: 下载systeminternals 然后解压缩后可以讲目录放到path环境变量中 然后打开命令行工具 输入 如下的命令 p ...

  9. git的使用基础

    /*游戏或者运动才能让我短暂的忘记心痛,现如今感觉学习比游戏和运动还重要——曾少锋*/ 在Git-Bash中配置自己的名字和Email: git config --global user.name & ...

  10. boost.python编译及演示样例

    欢迎转载,转载请注明原文地址:http://blog.csdn.net/majianfei1023/article/details/46781581 linux编译boost的链接:http://bl ...