Import function; not able to index in for loop. How to save variables as filenames?

Hello,

I have this script:

folder_name = uigetdir();

filenameExtension = '.txt';

for i = 1:13
   filename = [folder_name, '\OD ', int2str(i), filenameExtension];
   tmpData(i) = readtable(filename)
end

I want to import raw data from within a folder. All file names are OD 1.txt, OD 2.txt,... up to OD 13.txt.

The file content looks like this (905 rows):

OD 1,Time,Celsius(°C),Humidity(%rh),Dew Point(°C),Serial Number
1,2018-01-23 13:00:00,37.5,44.0,23.2,010201120
2,2018-01-23 13:05:00,36.0,48.5,23.4
3,2018-01-23 13:10:00,34.5,51.5,23.0
4,2018-01-23 13:15:00,35.0,51.0,23.3
5,2018-01-23 13:20:00,35.0,51.5,23.5
6,2018-01-23 13:25:00,33.5,55.0,23.2
7,2018-01-23 13:30:00,35.0,53.0,24.0
8,2018-01-23 13:35:00,36.5,47.5,23.5
9,2018-01-23 13:40:00,37.0,47.5,24.0
10,2018-01-23 13:45:00,36.5,47.0,23.4
11,2018-01-23 13:50:00,36.5,46.5,23.2
12,2018-01-23 13:55:00,36.0,49.5,23.8
13,2018-01-23 14:00:00,37.0,47.5,24.0

So this means it's actually a csv file within a txt file.

I want to import as a table. And I want the tablevariables in my workspace to represent the original filename. So OD1...etc.

My problem is in the for loop.

I need to use this format specifier (I guess) (& I can skip the serial number column):

formatSpec = '%f%{yyyy-MM-dd HH:mm:ss}D%f%f%f;

Can somebody help me on this one?

I get an error, because I cannot index with (i)... my loop reads all the datafiles as tables, but doesnt save them as individual variables in the workspace. How to do it?

Thank you in advance!

Answer 1

[Stephen23]

"My problem is in the for loop."

Yes, it is.

"... my loop reads all the datafiles as tables, but doesnt save them as individual variables in the workspace. How to do it?"

That is your problem.

The simplest, neatest, and most efficient answer to this question is "don't".

Magically creating or accessing variables names in a loop is exactly how beginners force themselves into writing slow, buggy, pointlessly complex code that is hard to debug. Read this to know more about why:

https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

Actually the fastest, neatest, and most efficient way for you to import that data into the MATLAB workspace would be to use indexing and one array: either an ND numeric array if the number of rows and columns are the same, or into a cell array if these are different. Using indexing is trivally simple to understand, is extremely efficient (unlike what you are trying to do), and simple to debug. I would reccomend that you do that because the filenames that you gave indicate a sequence 1, 2, ..., and that is exactly what indexing efficiently encodes:

C = cell(1,numel(files));
for k = 1:numel(files)
    C{k} = load(...);
end

And then access the data simply using idexing:

C{1} % OD1
C{2} % OD2

So simple.

So efficient!

https://www.mathworks.com/help/matlab/math/matrix-indexing.html

As an alternative which is less efficient than simple indexing (but still much more efficient than magically creating variable names) is to use the fields of a structure: e.g.:

S = struct();
for k = 1:numel(files)
    fieldname = ... % generate fieldname from filename
    S.(fieldname) = load(...)
end

And you can then access your data using

S.OD1
S.OD2
...etc

https://www.mathworks.com/help/matlab/matlab_prog/generate-field-names-from-variables.html

I would recommend that you use indexing.

[Image Analyst]

If you need another opinion, I totally agree with Stephen. Having separately named variables is a BAD idea. Just read it into one variable and process that file's data within that iteration of the for loop. You might call a custom function in the loop if you're going to be doing a ton of operations to the data.

[Vuk]

Thank you for the answer, but I still dont understand what to put in your (...) placeholders.

C = cell(1,numel(files));
for k = 1:numel(files)
    C{k} = load(...);
end

How do I get the txtfiles in the variable files?

I'm sorry for being this stupid, but I'm just what everybody calls here a 'scriptkiddy'...

The reason why I want variables with the filenames (just like you would get from the manual import tool when importing) is that I have this script for processesing:

%%Temp/RH plot over dryer length and time

x1=OD1.Time;
x2=OD2.Time;
x3=OD3.Time;
x4=OD4.Time;
x5=OD5.Time;
x6=OD6.Time;
x7=OD7.Time;
x8=OD8.Time;
x9=OD9.Time;
x10=OD10.Time;
x11=OD11.Time;
x12=OD12.Time;
x13=OD13.Time;

Xdatetime=vertcat(x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13);
X=datenum(Xdatetime);

y1=ones(length(x1),1).*1;
y2=ones(length(x1),1).*2;
y3=ones(length(x1),1).*3;
y4=ones(length(x1),1).*4;
y5=ones(length(x1),1).*5;
y6=ones(length(x1),1).*6;
y7=ones(length(x1),1).*7;
y8=ones(length(x1),1).*8;
y9=ones(length(x1),1).*9;
y10=ones(length(x1),1).*10;
y11=ones(length(x1),1).*11;
y12=ones(length(x1),1).*12;
y13=ones(length(x1),1).*13;

Y=vertcat(y1, y2, y3, y4, y5, y6, y7, y8, y9, y10, y11, y12, y13);

z1=OD1.CelsiusC;
z2=OD2.CelsiusC;
z3=OD3.CelsiusC;
z4=OD4.CelsiusC;
z5=OD5.CelsiusC;
z6=OD6.CelsiusC;
z7=OD7.CelsiusC;
z8=OD8.CelsiusC;
z9=OD9.CelsiusC;
z10=OD10.CelsiusC;
z11=OD11.CelsiusC;
z12=OD12.CelsiusC;
z13=OD13.CelsiusC;

Z=vertcat(z1, z2, z3, z4, z5, z6, z7, z8, z9, z10, z11, z12, z13);

createTEMPgraph(X,Y,Z)

Now I know this code is horrible for you real programmers, but can you just tell me how to complete my import code section from above, to get the variable names?

You could of course always give me advice how to make it more elegant... the problem is for a non-programmer to understand and implement.

I like my step with the uigetdir function. But how to go from there to make the rest work?

[Stephen23]

@Vuk: I don't have a version of MATLAB with readtable, so here is a working example using textscan:

opt = {'Delimiter',','};
fmt = '%*d%s%f%*f%*f'; % keep 2nd and 3rd columns only.
D = uigetdir();
%D = '.'; % current directory.
S = dir(fullfile(D,'OD*.txt')); % get the file names.
N = numel(S); % count how many files there are.
C = cell(N,1); % preallocate the output cell array.
for k = 1:N % for each file...
    [fid,msg] = fopen(fullfile(D,S(k).name),'rt');
    assert(fid>=3,msg) % check if file exists.
    hdr = fgetl(fid); % read header line and discard.
    one = textscan(fid,[fmt,'%*f'],1,opt{:}); % read first data line.
    two = textscan(fid,fmt,opt{:}); % read remaining data lines.
    C{k} = cellfun(@vertcat,one,two,'uni',0); % join data together.
    fclose(fid);
end
C = vertcat(C{:}); % create Nx2 cell (or however many columns you read).
R = cellfun('size',C(:,1),1); % rows of data for each file.
Y = cell2mat(arrayfun(@(n,r)n*ones(r,1),(1:N)',R,'uni',0));
X = vertcat(C{:,1});
Z = vertcat(C{:,2});

% createTEMPgraph(X,Y,Z)

This reads the timestamp and Celsius columns into one cell array C (exactly as you were advised). The contents of C are then simply concatenated together using vertcat: notice how this is much simpler than that ugly script that you have been given with all of those hard-coded variable names (ugh!), and note also that this code will automatically scale to any number of input files (not a fixed number like that badly written script): easily scaling to any number of files is one of the many advantages of using arrays.

Reading the file is a little bit fiddlier than usual because the first line of data is longer than the others. I got around this by simply reading that line separately (into variable one) and then joining the required data together with the rest of the lines (variable two).

When run on the test files (attached to this comment) this code produces the following variables (which match exactly the values that are in the test files):

>> X
X =
  '2018-01-23 13:00:00'
  '2018-01-23 13:05:00'
  '2018-01-23 13:10:00'
  '2018-01-23 13:15:00'
  '2018-01-23 13:20:00'
  '2018-01-24 13:00:00'
  '2018-01-24 13:05:00'
  '2018-01-24 13:10:00'
  '2018-01-24 13:15:00'
  '2018-01-25 13:00:00'
  '2018-01-25 13:05:00'
  '2018-01-25 13:10:00'
  '2018-01-25 13:15:00'
>> Y
Y =
   1
   1
   1
   1
   1
   2
   2
   2
   2
   3
   3
   3
   3
>> Z
Z =
   17.500
   16.000
   14.500
   15.000
   15.000
   27.500
   26.000
   24.500
   25.000
   38.500
   38.000
   38.500
   38.000

It would be easy to adapt this to using readtable and datetime objects, as you require.