Read in 150+ space delimited .csv files as matrices and perform the same operation on all of them

1 view (last 30 days)
I have 150+ space delimited .csv files that I need to read in as separate matrices so that I can perform the same operation on all of them (converting them to vectors by only keeping one side of the diagonal), and then recombine them into one matrix. I know I can generate a script or function from the wizard to read in a space delimited file, but I don't know how to loop this so it can be done to many files. Essentially, I want to programmatically read in all of these files (with different names) as separate matrices. I included the wizard code, but I don't know how to loop it to read in all .csv files. (I'm a beginner). Thank you.
%% Set up the Import Options and import the data
opts = delimitedTextImportOptions("NumVariables", 379);
% Specify range and delimiter
opts.DataLines = [1, Inf];
opts.Delimiter = " ";
% Specify column names and types
opts.VariableNames = ["VarName1", "VarName2", "VarName3", "VarName4", "VarName5", "VarName6", "VarName7", "e05", "VarName9", "VarName10", "e05_1", "e05_2", "VarName13", "VarName14", "VarName15", "VarName16", "VarName17", "VarName18", "e05_3", "e05_4", "e05_5", "e05_6", "VarName23", "VarName24", "VarName25", "e05_7", "e06", "e05_8", "e06_1", "VarName30", "e06_2", "VarName32", "e05_9", "e06_3", "VarName35", "e05_10", "VarName37", "e05_11", "VarName39", "VarName40", "VarName41", "VarName42", "VarName43", "VarName44", "e06_4", "e05_12", "e05_13", "VarName48", "e06_5", "e05_14", "e06_6", "VarName52", "e05_15", "e06_7", "VarName55", "e06_8", "e05_16", "VarName58", "e06_9", "e05_17", "VarName61", "VarName62", "VarName63", "VarName64", "e05_18", "VarName66", "VarName67", "VarName68", "VarName69", "VarName70", "VarName71", "e05_19", "VarName73", "VarName74", "VarName75", "VarName76", "VarName77", "VarName78", "VarName79", "VarName80", "VarName81", "e05_20", "VarName83", "VarName84", "VarName85", "VarName86", "e05_21", "VarName88", "VarName89", "VarName90", "VarName91", "VarName92", "VarName93", "VarName94", "VarName95", "VarName96", "VarName97", "VarName98", "VarName99", "VarName100", "VarName101", "VarName102", "VarName103", "e05_22", "VarName105", "e05_23", "VarName107", "e05_24", "e06_10", "e05_25", "VarName111", "VarName112", "VarName113", "e05_26", "VarName115", "e06_11", "e05_27", "VarName118", "VarName119", "VarName120", "VarName121", "VarName122", "VarName123", "e05_28", "VarName125", "e05_29", "VarName127", "VarName128", "VarName129", "VarName130", "e05_30", "VarName132", "e05_31", "e06_12", "e05_32", "e05_33", "e05_34", "e05_35", "VarName139", "e05_36", "e05_37", "e05_38", "e05_39", "e05_40", "e05_41", "VarName146", "e05_42", "e06_13", "VarName149", "e06_14", "e06_15", "VarName152", "e05_43", "e05_44", "e05_45", "e05_46", "e05_47", "VarName158", "e05_48", "e05_49", "VarName161", "e05_50", "e05_51", "VarName164", "VarName165", "e06_16", "VarName167", "VarName168", "e05_52", "VarName170", "VarName171", "e05_53", "VarName173", "VarName174", "VarName175", "e05_54", "VarName177", "e06_17", "VarName179", "VarName180", "e05_55", "VarName182", "VarName183", "e06_18", "e05_56", "e05_57", "e05_58", "VarName188", "VarName189", "VarName190", "VarName191", "VarName192", "VarName193", "VarName194", "VarName195", "e05_59", "e05_60", "VarName198", "VarName199", "VarName200", "VarName201", "VarName202", "VarName203", "VarName204", "VarName205", "e05_61", "VarName207", "VarName208", "VarName209", "VarName210", "VarName211", "VarName212", "VarName213", "VarName214", "VarName215", "VarName216", "VarName217", "VarName218", "VarName219", "VarName220", "VarName221", "VarName222", "VarName223", "VarName224", "VarName225", "VarName226", "VarName227", "VarName228", "VarName229", "VarName230", "VarName231", "VarName232", "VarName233", "VarName234", "VarName235", "VarName236", "VarName237", "VarName238", "VarName239", "VarName240", "VarName241", "VarName242", "VarName243", "VarName244", "VarName245", "VarName246", "VarName247", "VarName248", "VarName249", "VarName250", "VarName251", "VarName252", "VarName253", "VarName254", "VarName255", "VarName256", "VarName257", "VarName258", "VarName259", "VarName260", "VarName261", "VarName262", "VarName263", "VarName264", "e05_62", "VarName266", "VarName267", "VarName268", "VarName269", "VarName270", "VarName271", "VarName272", "VarName273", "VarName274", "VarName275", "VarName276", "e05_63", "VarName278", "VarName279", "VarName280", "VarName281", "VarName282", "VarName283", "VarName284", "VarName285", "VarName286", "VarName287", "VarName288", "VarName289", "VarName290", "VarName291", "VarName292", "VarName293", "e06_19", "VarName295", "VarName296", "VarName297", "VarName298", "e05_64", "e05_65", "VarName301", "e05_66", "VarName303", "VarName304", "VarName305", "VarName306", "VarName307", "VarName308", "VarName309", "VarName310", "VarName311", "e06_20", "VarName313", "VarName314", "VarName315", "VarName316", "VarName317", "VarName318", "VarName319", "VarName320", "VarName321", "VarName322", "VarName323", "VarName324", "VarName325", "VarName326", "e06_21", "VarName328", "VarName329", "VarName330", "VarName331", "VarName332", "VarName333", "VarName334", "VarName335", "VarName336", "VarName337", "VarName338", "VarName339", "VarName340", "VarName341", "VarName342", "VarName343", "VarName344", "VarName345", "VarName346", "VarName347", "VarName348", "VarName349", "VarName350", "VarName351", "VarName352", "VarName353", "VarName354", "VarName355", "VarName356", "VarName357", "VarName358", "VarName359", "VarName360", "VarName361", "VarName362", "VarName363", "VarName364", "e05_67", "e05_68", "VarName367", "VarName368", "VarName369", "VarName370", "e05_69", "VarName372", "VarName373", "VarName374", "VarName375", "e05_70", "VarName377", "VarName378", "VarName379"];
opts.VariableTypes = ["double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double"];
% Specify file level properties
opts.ExtraColumnsRule = "ignore";
opts.EmptyLineRule = "read";
opts.ConsecutiveDelimitersRule = "join";
opts.LeadingDelimitersRule = "ignore";
% Import the data
filename = readtable("C:\Users\me\Downloads\filename.csv", opts);
%% Convert to output type
filename = table2array(filename);
%% Clear temporary variables
clear opts

Accepted Answer

Mathieu NOE
Mathieu NOE on 25 Jun 2021
hello
the first topic is the loop to open all csv file automatically ,
if you prefer your own specific code instead of csvimport as shown here, you can do that as well
% A simpler, neater, more efficient solution is to use the structure returned by DIR:
% P = 'C:\Users\salonsov\Desktop\Results_lowflows\seasons';
P = cd;
S = dir(fullfile(P, '*.csv'));
for k = 1:numel(S)
F = fullfile(P, S(k).name);
S(k).data = csvimport(F); % or READTABLE or whatever.
end
% Take a look in the structure S: it contains all of your file data and the corresponding filenames, just as you require.
% For example, the 2nd filename and its data:
S(2).name
S(2).data
  2 Comments
Jordan
Jordan on 25 Jun 2021
For some reason the structure S doesn't contain my data or file names. It says it's a 0x1 structure with six fields
val =
name
folder
date
bytes
isdir
datenum
I know i have .csv files in the folder though. Not sure why it isn't grabbing them. Do you have any suggestions? Thank you so much!
Stephen23
Stephen23 on 25 Jun 2021
Edited: Stephen23 on 25 Jun 2021
"Do you have any suggestions?"
Did you change P to be the absolute/relative path to the folder where the files are saved?
Note that a simpler and more efficient way to refer to the current directory is to use '.', i.e.
P = '.';

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!