How to solve problem with smaller number of records read from datastore ?

I am using datastore to read data from csv file which has over 7 million records. The problem occurs when I set ReadSize field to 500000 or 1000000 and after making single read I get only about 100000 records. I get this issue in Matlab 2014b and in 2015a. Where could be the problem?

Answer 1

[Omar Qallaa]

I have a very similar problem. Similar to this case, I'm reading a very large single csv file that has ~100M records of just one feature. Setting "ReadSize" does not guarantee that the number of returned records after "read" is constant. I solved this as follows:

while hasdata(ds)
  data = read(ds);
  if (size(data,1) ~= requiredSamples) && hasdata(ds)
      % Change read size to number of missing samples
      ds.ReadSize = requiredSamples - size(data,1);
      tmp = read(ds);
      data = vertcat(data,tmp);
      % Set read size back to requiredSamples
      ds.ReadSize = requiredSamples;
  end
end

[thomassimm]

you may want to put

data = read(ds)

outside the while statement otherwise you're just overwriting it?

Pre-allocation of variable data would speed up too.

requiredSamples=85184;

data = table('size',[requiredSamples,3],'variabletypes',{'double';'double';'double'});

tmp = read(dat);
lena=height(tmp);
posa=lena+1;

data(1:lena,:)=tmp;

while hasdata(dat)
           
      tmp = read(dat);
      
      lena=height(tmp)-1;
      data(posa:posa+lena,:)=tmp;

      posa=posa+lena+1;
  
end

[Walter Roberson]

The person had wanted to get a full batch of size ReadSize, and process that batch, and then move on to the next batch as long as there was data. The person did not want to accumulate all of the data for the entire file. The assignment of the read(ds) is overwriting, but overwriting in the context of being the start of a batch, at a time when they want the entire batch to be overwritten with ReadSize new records.

[thomassimm]

Ok,

The 100M confuses me because the max data I can get is ~30k lines, presumably due to the 32 MB mentioned below. So if requireedSample>30000*2 then it won't work for me. Mine are 80k+ lines.

[Wesley]

Thanks so much, Omar! This seems to be working for me pretty well! I am running a large batch process and the reads started to reduce in size. I am hoping this will hold. If not, I may need to clear multiple variables from the workspace as a backup.

Answer 2

[Aaditya Kalsi]

I believe that datastore has an upper limit of the amount of data read from a file at once. I believe you are running up against this. Also, the 'ReadSize' property is an upper limit, so technically, this could be expected behaviour.

Is there a reason you require the exact number of rows? Does your algorithm depend on this?

[Camilius]

I just want to test how dimensionality reduction algorithm will work with large amount of specific data. It would be nice to see such limit documented somewhere. I will make some additional tests with datastore and post more comments about that.

[Camilius]

More interesting issue occured. When I set ReadSize to eg. 10000 and perform reading the datastore in while loop, the size of read chunks varies. One time it is 10000 but another oscillates around 1000 (1230,1234,1189 and so on)

[Aaditya Kalsi]

I believe that is expected. My suspicion is that you are reading multiple files and each file has only up to about a 1000 records, or you have a large file where each record is very large.

I quote the documentation below. The link is here .

I'm quoting the documentation here:

ReadSize — Amount of data to read
20000 (default) | positive scalar | 'file'
Amount of data to read in a call to the read function, specified as a positive scalar or the string, 'file'.

If ReadSize is a positive integer, then each call to read reads up to the specified number of rows from the datastore.
If ReadSize is 'file', then each call to read reads all of the data in one file.
When you change ReadSize from a numeric scalar to 'file' or vice versa, MATLAB resets the datastore to the state where no data has been read from it.

[Camilius]

Recently I examined source code of datastore, and there is one hardcoded feature which describes limit for a chunk of data. The limit is 32 MB. I am reading single file, that has about 7 million records, one record has 43 features. I think this 100000 record limit is caused by this 32 MB limit.