Importing a single piece of data from a lot of txt files

I usually feel pretty comfortable with matlab but this one has me scratching my head.

I have about 500 text files. Each with a [4500x5] set of data. I need to get one point of data from each file like [2350,3] and put that in an array with that data point from all 500 text files.

I am having difficulty getting this to work without having to import each of the 500 text files and all of the data points.

I have tried combining all the text files into one and importing it and using a for loop to get the data point that I need and then iterating to the next one in the list. The problem here was that when combined all of the text files have over 10 million points of data. When I started the loop it crashed my computer.

Is there an easier way to gather this one point of data from each one of the 500 text files?

Answer 1

[Walter Roberson]

cols_per_file = 5;

wantrow = 2350;
wantcol = 3;

linepattern = [repmat('%*f',1,wantcol-1), '%f', repmat('%f',1,cols_per_file - wantcol)];

indata = zeros(numfiles,1);

for K = 1 : numfiles
  thisfile = filenames{K};
  fid = fopen(thisfile, 'rt');
  onevalue = textscan(fid, linepattern, 1, 'HeaderLines', wantrow-1);
  fclose(fid);
  indata(K) = onevalue{1};
end

You might want to dress this up a bit with try/catch just in case a file is not readable or somehow comes out too small.

[Derek]

Thanks for the quick response.

I just set up your code but ran into a snag.

cols_per_file = 5;

wantrow = 2846;

wantcol = 3;

linepattern = [repmat('%*f',1,wantcol-1), '%f', repmat('%f',1,cols_per_file - wantcol)];

indata = zeros(512,1);

for K = 1 : 512

thisfile = Final AVG PIC{K};

fid = fopen(thisfile, 'rt');

onevalue = textscan(fid, linepattern, 1, 'HeaderLines', wantrow-1);

fclose(fid);

indata(K) = onevalue{1};

end

The files are named Final AVG PIC001, Final AVG PIC002 etc. I don't believe this works with K=1. Also when I input the file names I get an error since it believes that AVG is the matlab function not the name of the file. Maybe it's just too late here and I am missing something.

Thank you again for the help

[Derek]

One more comment. They are not .txt files they are .jvc they are still readable by text editor though. Could that be the problem?

That is just the format that the computation fluids software puts the data in.

[Walter Roberson]

My code presumes that you have already created a cell array of strings that contains the names of the files to work with.

For example,

filenames = cellstr(num2str((1:numfiles).', 'Final AVG PIC%03d.jvc'));

Or if appropriate you could use dir() to scan the directory and find all of the 'Final AVG PIC*.jvc' files, and then assign numfiles as the number of entries returned by dir():

fileinfo = dir('Final AVG PIC*.jvc');

filenames = {fileinfo.name};

numfiles = length(filenames);