output of evalc

Hi, I am trying to load the filenames of 10000+ files (on a network path).

dir & ls seem the obvious commands, but they are both slow (33s for 18k files).

evalc on the other hand takes less than a second. (evalc('dir(...)')) however, the output is in a 1-d char array consisting of all filenames. There are spaces in the filenames as well, so that makes seperating a bit troublesome.

is it possible to get a different output format, or does anyone know a different "solution"?

Thanks! Jaap

PS. It's on a 32bit winXP

[Daniel Shub]

How are you mapping the network drive? Is it possible to run the dir command directly on the "server" and create a text file? This could speed things up a lot. Also, the way a drive is mounted can also influence how long things take.

[Jaap]

that would certainly be a good approach. however, it's a company network drive, in which I would like to keep the traffic one-way.

[Jaap]

other people are going to use this as well, who possibly do not have write access.

Answer 1

[Daniel Shub]

It seems that

evalc(['dir(', FilePath. ')'])

should go faster than

D = dir(FilePath);

since the evalc method only gets the file names while the direct call gets additional information. On a local filesystem, it probably doesn't matter much, but over a network it could make a big difference.

What about using the system call. On Windows that would be

[~, filenameArray] = system('dir /b');

and on Linux it would be

[~, filenameArray] = system('ls -1');

(note it is a one and not an el). . If you have oddly named or hidden files you may need more parameters.

[Jaap]

your reasoning then would also suggest that "ls" is faster than "dir", which is not the case. I like the system call, but I'm getting curious as to why there is such a big difference. (is it OS dependent?)

[Daniel Shub]

I wouldn't extrapolate between what MATLAB does to get the directory information over a network share and how system level calls do it. My guess is it is both OS and mount type dependent. Is this a samba/windows share mount?

[Jaap]

I'm not quite sure, but it seems to be a regular network drive (visible under my computer etc).

Answer 2

[Jan]

Are you sure that Str = evalc(['dir(', FilePath. ')']) and D = dir(FilePath); have such a large speed difference?! Or did you just call dir at first, such that evalc('dir') can access the cached data?

Please post a copy of the used code.

[Jaap]

Thanks for your response.

The order in which I call the commands does NOT seem to matter.

This is the code I used (first time since OS startup, so no cache (I would assume). s{2} contains the path.

>> tic, blub = evalc('dir(s{2})'); toc

Elapsed time is 0.684967 seconds. (output is 582313x1 char)

>> tic, blub = dir(s{2}); toc

Elapsed time is 53.124096 seconds. (output is 19089x1 struct)

>> tic, blub = ls(s{2}); toc

Elapsed time is 53.377271 seconds. (output is 19089x28 char)

[Titus Edelhofer]

One of the problems is definetely the point "network" path. I just created a folder with 20000 files, and the command

blub = dir(pwd);

took 1.2 seconds. But for a network drive I get similar differences as you (about 20s for dir, 0.5s for evalc).

[Titus Edelhofer]

Hmm, when scanning the output of evalc you already pointed out the main obstacle: filenames with spaces will give you a hard time (probably making this approach unusable).

[Jaap]

Thank you titus, indeed the same experience I had. there is something possible with strfind and filename extentions to search for filenames in a large string, but then I think I would prefer the slower approach.

Answer 3

[Titus Edelhofer]

But you could try something similar:

   p = s{2};
  [a,b]=system(['dir /B "' p '"']);

It took about half the time of the dir() call ...

Titus