fprintf for ~million row string matrix

Hi,

I'm trying to write data to csv files, but using fprintf (or the way I am using it) seems to be extremely slow

The data is formatted in columns containing fields such as: {ID (string), date (string), numeric data, flag (string), ...}.

I get the data as a cell array, with each cell having a column of data.

What I've done is cast everything into a string and take the transpose of the data (it seems MATLAB does things in a column-major order).

Also note I've added a bunch of commas without data inbetween. That's the expected format of the data, so that has to stay.

fid = fopen(fullfile(output, replace(file.name, '.dat', '.csv')), 'w');

fprintf(fid, [repmat('%s,', 1, nCols), repmat(',', 1, 86-nCols), '\n'], T);

fclose(fid);

It looks as if saving a single file takes several minutes (haven't managed to save even one file yet - and I need to do this for 365 files).

Is there a way to speed up this process?

Kind Regards,

Erik

Answer 1

[Jon]

You could try writecell and see if that is more efficient.

[Erik Johannes Loo]

Thanks. I don't have writecell on my version (R2018b) but I tried the code on a different computer (with R2020b) and my original approach (fprintf) worked much faster without modification.

I'm assuming fprintf was improved from R2018b to R2020b which has led to dramatic performance improvements.

[Jon]

Or maybe the computer with R2020b was just faster? Also I have found that MATLAB is sometimes very slow writing to network drives. If possible write it to a file on your local hard drive.