Optimal way to create a large matrix from smaller chunks

I'm writing a function that is supposed to create quite large 2D matrices (up to billions of elements). Because of mathematical structure, I generate data for the matrix as smaller 2D matrices of varying size that are then put in proper places of predefined output matrix. The data generation itself is pretty well optimized, but putting these smaller chunks into the output array takes a lot of time.

Here's what the code looks like after deleting unnecessary elements:

function Znlm = matrixGenerator(phiV, psiV, nmax, lmax, mmax)

%% create output matrix of proper size

N_nlm = 0;
for n=0:nmax
   for l=0:min([n, lmax])
      N_nlm = N_nlm + 1 + 2*min([l, mmax]);
   end
end

N_ang = length(phiV);

Znlm = zeros(N_ang,N_nlm);

%% fill the matrix with HSHs

% define row position iterator
N_psi = length(psiV);
rowZ = 1:N_psi;

for psi = psiV'

   % reset column position iterator
   colZ_start = 1;
   
   tic
   for n = 0:nmax
      lmax_temp = min(n,lmax);
      for l = 0:lmax_temp
         mmax_temp = min(l,mmax);

         % determine column position iterators
         colZ = colZ_start:colZ_start + 2*mmax_temp;
         colZ_start = colZ(end) + 1;
         
         Znl = rand(N_ang,2*mmax_temp+1);  % here goes my data generating function, irrelevant for this example
         Znlm(rowZ,colZ) = Znl; % the time consuming line, after commenting it the function executes in close to no time
         
         % increment row position iterator
         rowZ = rowZ + N_ang;
      end
   end
   toc
end

If I run this function with some random input vectors I get following results:

matrixGenerator(rand(5,1),rand(5,1),30,10,10);
Elapsed time is 0.668320 seconds.
Elapsed time is 1.945542 seconds.
Elapsed time is 3.050079 seconds.
Elapsed time is 4.174644 seconds.
Elapsed time is 5.566776 seconds.

As you can see, the time needed to put consecutive parts of the matrix is significantly increasing. I run this example with only 5-element input vectors and eventually I want to use ~500-element ones, so it'd take hours to create the matrix. Do you have any idea how to make it faster?

[KSSV]

REad about cat. Or you can make your matrices a cell array and use cell2mat.

[Adam Szwajcowski]

Ok, turns out I'm just dumb and put row position iterator in a wrong loop (in fact in 2 wrong loops). Anyway, I tested the cat and it's much easier to use, but several times slower than predefined matrix, so I guess at least I learned something. I'll leave this thread, maybe somebody will benefit from this observation.

[Sara Boznik]

Time increasing because of for. Maybe would be better if you make matrixs with 1 column for every cases and then just put it together in larger matrix.

[Adam Szwajcowski]

Sara, no, this was not an issue. I had a bug in defining indices for the matrix and it was much larger than needed. I fixed it, looks like my method all in all was fine, I just brainfarted with its execution.

Answer 1

[Matt J]

One way is to do a dry run to see how big the matrix is supposed to be:

Znlm=matrixGenerator(rand(5,1),rand(5,1),30,10,10);

function Znlm = matrixGenerator(phiV, psiV, nmax, lmax, mmax)

    %% create output matrix of proper size

    N_nlm = 0;
    for n=0:nmax
        for l=0:min([n, lmax])
            N_nlm = N_nlm + 1 + 2*min([l, mmax]);
        end
    end

    N_ang = length(phiV);



    %% fill the matrix with HSHs

    % define row position iterator
    N_psi = length(psiV);

    
    
    %%Dry run
    rowZ = 1:N_psi;
    for psi=psiV.'
        colZ_start = 1;
        for n = 0:nmax
            lmax_temp = min(n,lmax);
            for l = 0:lmax_temp
                mmax_temp = min(l,mmax);
                
                % determine column position iterators
                colZ = colZ_start:colZ_start + 2*mmax_temp;
                colZ_start = colZ(end) + 1;
                
                % increment row position iterator
                rowZ = rowZ + N_ang;
            end
        end
    end
 
    
    Znlm = zeros(max(rowZ)-N_ang,max(colZ));
    
    rowZ = 1:N_psi;
        
    for psi = psiV.'

        
        % reset column position iterator
        colZ_start = 1;

        tic
        for n = 0:nmax
            lmax_temp = min(n,lmax);
            for l = 0:lmax_temp
                mmax_temp = min(l,mmax);

                % determine column position iterators
                colZ = colZ_start:colZ_start + 2*mmax_temp;
                colZ_start = colZ(end) + 1;

                Znl = rand(N_ang,2*mmax_temp+1);  % here goes my data generating function, irrelevant for this example
                Znlm(rowZ,colZ) = Znl; % the time consuming line, after commenting it the function executes in close to no time

                % increment row position iterator
                rowZ = rowZ + N_ang;
            end
        end
        toc
    end

end

Elapsed time is 0.009185 seconds.
Elapsed time is 0.006094 seconds.
Elapsed time is 0.006132 seconds.
Elapsed time is 0.005855 seconds.
Elapsed time is 0.005694 seconds.