Output very large matrix into text files

Dear friends

Thanks for viewing my problem. I am new to Matlab.

I have got a 2D matrix(please see exmaple.mat, node_matrix), in the form of:

1,0.5,0.8,0.9

2,0.1,0.2,0.7

I would like to output this into a text file in the form of :

N,1,0.5,0.8,0.9

N,2,0.1,0.2,0.7

As you can see I add a letter 'N' in the first column. To do this, as there are numbers and letters, I created a cell containing all this info and output the whole cell into text file using

B=transpose(myCell);

filename = 'elementdata.txt';

fid = fopen(filename, 'w');

fprintf(fid, '%s,%d,%d,%d,%d,%d,%d,%d,%d\n', B{:});

fclose(fid);

Q1: As my dataset would be very large, sometimes it could contain a billion elements,so to convert it into a cell add extra memory burden. I am wondering if I can output my matrix directly in the form I want (with letter 'N' ahead) without convert it into a cell?

Q2: If it can without creating the Cells that would be great. But I am afraid if this would still needs a lot of memory. So I would like to know if there is a way to output the first 100 rows in the Matrix and output it in the text file and then clean the memory and then load the next 100 rows and then output this in the same text file right after the first 100 rows ....and so on.

Please refer to the attached file and you will know the stupid way I am doing...:)

Many thanks

Yuan Chen

Answer 1

[Ashish Gudla]

Instead of converting the matrix into cells and adding an extra column with ‘N’ in it you can directly write the matrix to file with an ‘N’ at the beginning of each line as follows:

>> filename = 'nodedata.txt';
>> fid = fopen(filename, 'w');
>> B=transpose(Node_Matrix);
>> fprintf(fid, 'N,%d,%d,%d,%d\n', B(:));
>> fclose(fid);

In case you have large amounts of data in your “mat” file, you can load the matrix partially each time and append it to the text file.

>> my_file = matfile(‘example.mat’);
>> temp_data = m.Node_Matrix(1:100, :) %load first 100 rows
>> filename = 'nodedata.txt';
>> fid = fopen(filename, 'a'); %open the file in append mode
>> B=transpose(temp_data);
>> fprintf(fid, 'N,%d,%d,%d,%d\n', B(:));
>> fclose(fid);

For more information on efficient memory handling in MATLAB follow this doc page