Code to split and save a big file according to the experiment number

I have a big .txt file where I have numerical data and text. It begin with text "# Data from the experiment1" and three columns of numerical data (almost 1500 rows or even more) and then again text "# Data from the experiment2" followed with numerical data with several rows. This repeats several times (depends on how many experiments I have). I want to split the text file and save it as separate files named Exp1, Exp2...

What I am doing now is, open the file and find the last number in each section and use that number in the matlab code to split up the files. Is there any smarter way to split the files just by looking for the text begin with "# Data from" and split the textfile ?

Answer 1

[Star Strider]

If the number of header lines in the various sections of your file are all the same, something like this should work (it has for me in the past):

fidi = fopen('aneps Big_Text_File.txt', 'rt');
k1 = 1;
while ~feof(fidi)
    data{k1} = textscan(fidi,'%f%f%f%f%f', 'HeaderLines',9, 'Delimiter',' ', 'CollectOutput',1);
    fseek(fidi, 1, 0);
    k1 = k1 + 1;
end
fclose(fidi);

The various sections will be in various cells of ‘data’, and may be different sizes, so you can save each to a separate text file later. Make appropriate changes in the textscan call for your data.

Note: Your file has to have a valid end-of-file indicator or my code becomes an infinite loop. In that event, I need to see the file to figure out an appropriate way to find an end to it.

[aneps]

I know the number of files (end of the file indicator) for each "big data file". How can I make it terminate at nth loop?

[Star Strider]

My code will read all the sections in the file and stop when it encounters a valid end-of-file indicator.

If you only want to read a specific number of sections in your files, and you know my code will not encounter an end-of-file indicator first, replace the while loop with a for loop to read only the number of sections in the file you want. The code should work without any other modification.

The easiest way would be to let the code read the entire file, then choose only the number of ‘data’ cells you want to write to the individual data files.

Instead of creating individual data files, I would save the ‘data’ cell to a .mat file.