How to import the Number of rows from the csv or xls file at a time depending upon the memory.

Hi,

Iam facing one issue (i.e., Memory out of range) while loading the large no of data from csv file. Now I would like to determine how much data (i.e., Number of rows can be imported at a single instance) to avoid out of mememory range issue. Please provide the example with code.

Regards, S.S.Kiran

Answer 1

[Sean de Wolski]

Well figure out how much memory your computer has and subtract 10-15% maybe for overhead. Assuming no other programs are killing your RAM, figure out the number of bytes free.

So on a 4GB machine:

availRAM = 3.5e9-bytes;

where bytes is this function:

function Nbytes = bytes
%Function to sum the number of bytes used in the caller workspace
%SCd 02/04/2011
%
%Usage:
%   >>Nbytes = bytes;
%
%Input Arguments:
%   -None!
%
%Output Arguments:
%   -Nbytes: number of bytes used
%
%See Also: whos who
%  

      %Engine: (Yeah big engine!)
      A = evalin('caller','whos');
      Nbytes = sum([A(:).bytes]);

end

Then, each number in your csv file will take up as many bytes as the class it is. Let's assume for now that each value is a double - thus it'll take up 8bytes/element.

Figure out how many elements you can acquire:

elems = availRAM/bytes_per_elem

That's how many elements you can read in. Divide that by the number of colums:

rows_allowed = floor(elems/ncolumns); %floor so you get a round number

import and have fun.