LazyProgrammer - Unsupervised Deep Learning in Python: Master Data Science and Machine Learning with Modern Neural Networks written in Python and Theano

There are 2 components to their description: 1) describing what PCA does and how it is used 2) the math behind PCA. So what does PCA do? Firstly, it is a linear transformation. If you think about what happens when you multiply a vector by a scalar, youll see that it never changes direction. It only becomes a vector of different length. Of course, you could multiply it by -1 and it would face the opposite direction, but it cant be rotated arbitrarily. 2(1, 2) = (2, 4) If you multiply a vector by a matrix - it CAN change direction and be rotated arbitrarily. Ex. [[1, 1], [1, 0]] [1, 2] T = [3, 1] T So what does PCA do? Simply put, a linear transformation on your data matrix: Z = XQ It takes an input data matrix X, which is NxD, multiplies it by a transformation matrix Q, which is DxD, and outputs the transformed data Z which is also NxD. [[1, 1], [1, 0]] [1, 2] T = [3, 1] T So what does PCA do? Simply put, a linear transformation on your data matrix: Z = XQ It takes an input data matrix X, which is NxD, multiplies it by a transformation matrix Q, which is DxD, and outputs the transformed data Z which is also NxD.

If you want to transform an individual vector x to a corresponding individual vector z, that would be z = Qx. Its in a different order because when x is in a data matrix its a 1xD row vector, but when we talk about individual vectors they are Dx1 column vectors. Its just convention. What makes PCA an interesting algorithm is how it chooses the Q matrix. Notice that because this is unsupervised learning, we have an X but no Y, or no targets. On Rotation Another view of what happens when you multiply by a matrix is not that you are are rotating the vectors, but you instead are rotating the coordinate system in which the vectors live.

Which view we take will depend on which problem we are trying to solve. Dimensionality Reduction One use of PCA is dimensionality reduction. When youre looking at the MNIST dataset, which is 28x28 images, or vectors of size 784, thats a lot of dimensions, and its definitely not something you can visualize. 28x28 is a very tiny image, and most images these days are much larger than that - so we either need to have the resources to handle data that can have millions of dimensions, or we could reduce the data dimensionality using techniques like PCA. We of course cant just take arbitrary dimensions from X - we want to reduce the data size but at the same time capture as much information as possible. Ex. info(1st col of Z) > info(2nd col of Z) > ... info(1st col of Z) > info(2nd col of Z) > ...

How do we measure this information? In traditional PCA and many other traditional statistical methods we use variance. If something varies more, it carries more information. You can imagine the opposite situation, where a variable is completely deterministic, i.e. it has no variance. Then measuring this variable would not give us any new information, because we already knew what it was going to be. So when we get our transformed data Z, what we want is for the first column to have the most information, the second column to have the second most information, and so on.

Thus, when we take the 2 columns with the most information, that would mean taking the first 2 columns. De-correlation Another thing PCA does is de-correlation. If we find correlations, that means some of the data is redundant, because we can predict one column from another column.