Alternatives for dir command

8 views (last 30 days)
Linus Dock
Linus Dock on 10 Oct 2021
Commented: Linus Dock on 11 Oct 2021
Hello dear masterminds!
I have a working code for collecting strings from several txt.files (appr. 53000 files) in a network folder. I have one problem though. It is quite time consuming (17-19 seconds) using the dir command to identify the filenames. Is there a better way of doing this? I will attach a part of the code and some of the filenames.
Thank you!
addpath \\winfs\data\prod\maprod\arkivering\TAF;
DS = StartDTstr;
DE = StopDTstr;
TI = duration(timein,'InputFormat','hh:mm');
TO = duration(timeout,'InputFormat','hh:mm');
P = '\\winfs\data\prod\maprod\arkivering\TAF';
S = dir(fullfile(P,'*.txt'));
D = regexp({S.name},'\d{10}','match','once');
T = datetime(D, 'InputFormat','yyMMddHHmm');
X = isbetween(T,DS,DE);
tod = timeofday(T);
Y = tod>=TI & tod<=TO;
Z = S(X&Y);
Z.name;
numfiles = length(Z(:,1)); %create empty cell
for h=1:numfiles
filename=Z(h).name;
fileID=fopen(filename); %open filename to create fileID
Data{h}=textscan(fileID,'%s','delimiter','=','headerlines',1);
fclose(fileID); %close fileID
end
rmpath \\winfs\data\prod\maprod\arkivering\TAF;
Utcell = cell(1,length(Data)); %output data of all TAF
  6 Comments
Linus Dock
Linus Dock on 11 Oct 2021
Ok thanks!
Really great to have your help!
/Linus

Sign in to comment.

Accepted Answer

Jan
Jan on 10 Oct 2021
Edited: Jan on 10 Oct 2021
addpath \\winfs\data\prod\maprod\arkivering\TAF;
Why do you append this folder to Matlab's path? Are the files stored in this folder or some required Matlab functions? If this folder does not contain Matlab functions, there is no reason to append it to the path. Simply omit the addpath/rmpath commands. See below.
What is the purpose of this:
DS = StartDTstr;
DE = StopDTstr;
TI = duration(timein,'InputFormat','hh:mm');
TO = duration(timeout,'InputFormat','hh:mm');
Are this variables or functions?
What exactly takes 17-19 seconds?
Is Data pre-allocated? If the final size is known, allocate it instead of letting the array grow iteratively:
numfiles = size(Z, 1); % Better than: length(Z(:,1));
Data = cell(1, numfiles);
for h=1:numfiles
filename = fullfile(P, Z(h).name); % Absolute path
fileID = fopen(filename);
Data{h} = textscan(fileID, '%s', 'delimiter', '=', 'headerlines', 1);
fclose(fileID);
end
It is more efficient to use the absolute path name because with fopen('file.txt') Matlab's complete path is searched.
Reading 53'000 files over a potentially slow network connection takes time. How large are the files? Maybe the runtime is limited by the network connection. Then there is no way to accelerate it magically.
  4 Comments
Linus Dock
Linus Dock on 10 Oct 2021
I see that now, sorry :)
/Linus

Sign in to comment.

More Answers (0)

Tags

Products


Release

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!