PCA or LSA on a very large dataset

I am trying to run LSA or PCA on a very large dataset, 50k docs by 300k terms, to reduce the dimensionality of the words.

My system runs out of memory and grinds to a halt. I am using this code on the TFIDF matrix to compute the LSA, and it gets stuck on SVD:

   % SVD decomposition of tf-idf matrix
   [ U S V ] = svd(tfidfmatrix);

   % Generate new rank reduced matrix of rank k
   Sk = S(1:K,1:K);
   Uk = U(:,1:K);
   output = inv(Sk)*Uk';

and using PRINCOMP for PCA. In both cases, it seems I have too many terms for my system to handle.

Is there a more efficient way to do dimensionality reduction for two words? The end goal is to visualize the documents in 2d or 3d where they are grouped by similarity to each other.

Answer 1

[arushi]

Hi Eugene,

When dealing with very large datasets, such as your 50k documents by 300k terms matrix, traditional methods like full Singular Value Decomposition (SVD) and Principal Component Analysis (PCA) can be computationally expensive and memory-intensive. Instead, you can use more efficient methods designed for large-scale data.Efficient Methods for Dimensionality Reduction

1. Truncated SVD (also known as Latent Semantic Analysis - LSA):

• Instead of computing the full SVD, you can compute only the top K singular values and vectors using methods like svds in MATLAB.

2. Incremental PCA:

• Incremental PCA is designed to handle large datasets by processing data in chunks.

Hope this helps.