What is an eigenvector of a covariance matrix?

One of the most intuitive explanations of eigenvectors of a covariance matrix is that they are the directions in which the data varies the most.

(More precisely, the first eigenvector is the direction in which the data varies the most, the second eigenvector is the direction of greatest variance among those that are orthogonal (perpendicular) to the first eigenvector, the third eigenvector is the direction of greatest variance among those orthogonal to the first two, and so on.)

Here is an example in 2 dimensions [1]:

Each data sample is a 2 dimensional point with coordinates x, y. The eigenvectors of the covariance matrix of these data samples are the vectors u and v; u, longer arrow, is the first eigenvector and v, the shorter arrow, is the second. (The eigenvalues are the length of the arrows.) As you can see, the first eigenvector points (from the mean of the data) in the direction in which the data varies the most in Euclidean space, and the second eigenvector is orthogonal (perpendicular) to the first.

It's a little trickier to visualize in 3 dimensions, but here's an attempt [2]:

In this case, imagine that all of the data points lie within the ellipsoid. v1, the direction in which the data varies the most, is the first eigenvector (lambda1 is the corresponding eigenvalue). v2 is the direction in which the data varies the most among those directions that are orthogonal to v1. And v3 is the direction of greatest variance among those directions that are orthogonal to v1 and v2 (though there is only one such orthogonal direction).

[1] Image taken from Duncan Gillies's lecture on Principal Component Analysis
[2] Image taken from Fiber Crossing in Human Brain Depicted with Diffusion Tensor MR Imaging

  
Written 1 Aug, 2013 • 48,160 views
 
Given a set of random variables {x1,...,xn}, the covariance matrix A is defined so that Ai,j=Cov(xi,xj). We can represent a linear combination ∑bixi as a vector x=(b1,...,bn).

It turns out that the covariance of two such vectors x and y can be written as Cov(x,y)=xtAy. In particular, Var(x)=xtAx. This means that covariance is a Bilinear form.

Now, since A is a real symmetric matrix, there is an orthonormal basis for Rnof eigenvectors of A. Orthonormal in this case means that each vector's norm is 1 and they're orthogonal with respect to A, that is vt1Av2=0, or Cov(v1,v2)=0.

Next, suppose v is a unit eigenvector of A with eigenvalue λ. Then Var(v)=λ∥v∥2=λ.

There are a couple interesting conclusions we can draw from this. First, since the eigenvectors form a basis {v1,...,vn}, every linear combination of the original random variables can actually be represented as a linear combination of the independent random variables vi. Second, every unit vector's variance is a weighted average of the eigenvalues. This means that the leading eigenvector is the direction of greatest variance, the next eigenvector has the greatest variance in the orthogonal subspace, and so on.

So, sum up, eigenvectors are uncorrelated linear combinations of the original set of random variables.

The primary application of this is Principal Components Analysis. If you have n features, you can find eigenvectors of the covariance matrix of the features. This allows you to represent the data with uncorrelated features. Moreover, the eigenvalues tell you the amount of variance in each feature, allowing you to choose a subset of the features that retain the most information about your data.

  
Written 28 Nov, 2013 • 5,789 views
 
The largest eigenvector of a covariance matrix points into the direction of the largest variance. All other eigenvectors are orthogonal to the largest one.

Now, if this direction of the largest variance is axis-aligned (covariances are zero), then the eigenvalues simply correspond to the variances of the data:

It becomes a little more complicated if the covariance matrix is not diagonal, such that the covariances are not zero. In this case, the principal components (directions of largest variance) do no coincide with the axes, and the data is rotated. The eigenvalues then still correspond to the spread of the data in the direction of the largest variance, whereas the variance components of the covariance matrix still defines the spread of the data along the axes:

An in-depth discussion of how the covariance matrix can be interpreted from a geometric point of view (and the source of the above images) can be found on:A geometric interpretation of the covariance matrix

  
Written 10 Mar • 2,674 views
Shreyas Ghuge

3 upvotes by Sameer Gupta, Anonymous, and Ram Shankar
 
Finding the directions of maximum and minimum variance is the same as looking for where the orthogonal least squares best fit line and plane of the data. The sums of squares for that line and plane can be written in terms of covariance matrix.The connections between them can be worked out to get the Eigen vectors of this covariance matrix.

  
Written 22 Aug, 2013 • 4,598 views
Julius Bier Kirkegaard, physics, computers, 'n' stuff

4 upvotes by David Joyce (Professor of Mathematics at Clark University), Andrei Kucharavy (PhD student in Bioinformatics), Martin Andrews, and Laasya Alamuru
 
Finding the eigenvectors a covariance matrix is exactly the technique of Principal Component Analysis (PCA).

The eigenvectors are those variables that are linearly uncorrelated.

  
Written 1 Aug, 2013 • 4,307 views
 
 
Write an answer
Related Questions
 

What is an eigenvector of a covariance matrix?的更多相关文章

  1. A geometric interpretation of the covariance matrix

    A geometric interpretation of the covariance matrix Contents [hide] 1 Introduction 2 Eigendecomposit ...

  2. 方差variance, 协方差covariance, 协方差矩阵covariance matrix

    https://www.jianshu.com/p/e1c8270477bc?utm_campaign=maleskine&utm_content=note&utm_medium=se ...

  3. 方差variance, 协方差covariance, 协方差矩阵covariance matrix | scatter matrix | weighted covariance | Eigenvalues and eigenvectors

    covariance, co本能的想到双变量,用于描述两个变量之间的关系. correlation,相关性,covariance标准化后就是correlation. covariance的定义: 期望 ...

  4. covariance matrix 和数据分布情况估计

    how to get data covariance matrix: http://stattrek.com/matrix-algebra/covariance-matrix.aspx meaning ...

  5. 随机变量的方差variance & 随机向量的协方差矩阵covariance matrix

    1.样本矩阵 如果是一个随机变量,那么它的样本值可以用一个向量表示.相对的,如果针对一个随机向量,那么就需要利用矩阵表示,因为向量中的每一个变量的采样值,都可以利用一个向量表示. 然后,一个矩阵可以利 ...

  6. A Beginner’s Guide to Eigenvectors, PCA, Covariance and Entropy

    A Beginner’s Guide to Eigenvectors, PCA, Covariance and Entropy Content: Linear Transformations Prin ...

  7. Ill-conditioned covariance create

    http://www.mathworks.com/matlabcentral/answers/100210-why-do-i-receive-an-error-while-trying-to-gene ...

  8. 协方差(Covariance)

    统计学上用方差和标准差来度量数据的离散程度 ,但是方差和标准差是用来描述一维数据的(或者说是多维数据的一个维度),现实生活中我们常常会碰到多维数据,因此人们发明了协方差(covariance),用来度 ...

  9. 深度学习课程笔记(十二) Matrix Capsule

    深度学习课程笔记(十二) Matrix Capsule with EM Routing  2018-02-02  21:21:09  Paper: https://openreview.net/pdf ...

随机推荐

  1. 创建外网 ext_net - 每天5分钟玩转 OpenStack(104)

    虽然外部网络是已经存在的网络,但我们还是需要在 Neutron 中定义外部网络的对象,这样 router 才知道如何将租户网络和外部网络连接起来. 上一节我们已经为创建外部网络配置了ML2,本节将通过 ...

  2. 2015年ACM长春网络赛(准备做掉7道:已经更新到6道)

    总结汇总:模板 int getmax_min(char s[]) {//字符串的最大表示法:返回最小数组下标 , j = , k = ; while(i < len && j & ...

  3. 论ubuntu的作死技巧

    此处记录自己弄崩系统的几大杀器,长期更新. 1. sudo apt-get autoremove

  4. linux提取锁和信号灯经常使用

    1.信号( 这两个过程之间的同步) struct semaphore power_sem; sema_init(&pdata->power_sem,1); down(&pdata ...

  5. Android 文字过长TextView如何自动截断并显示成省略号

    当用TextView来显示标题的时候,如果标题内容过长的话,我们不希望其换行显示,这时候我们需要其自动截断,超过的部分显示成省略号. 如下图所示,标题过长,自动换行了,显示不是很好看. 这时候我们需要 ...

  6. C#版 - Leetcode 65. 有效数字 - 题解

    版权声明: 本文为博主Bravo Yeung(知乎UserName同名)的原创文章,欲转载请先私信获博主允许,转载时请附上网址 http://blog.csdn.net/lzuacm. Leetcod ...

  7. 【JavaScript】轮播图

    代码: <!DOCTYPE html> <html> <head> <meta charset="utf-8" /> <tit ...

  8. 线性求第k大

    快排变种. 快排每次只进行部分排序,进入左边或者右边或者当前mid就是答案. 据说期望值是O(n) 然后STL中的 nth_element也是用这个思想. #include <cstdio> ...

  9. [数据库]Sqlite使用入门

    官网的文档结构十分恶劣,大概翻了一下,提供入门指引. 0. sqlite的安装 根据自身情况,在官网下载32位/64位的dll文件以及sqlite-tools-win32-x86-3240000.zi ...

  10. Awake()跟Start()差在哪?

    刚开始学Unity的时候,最难搞定的就是这两个functions的差异,依照官方文件所描述的: Awake(): Awake is called when the script instance is ...