matlab file indexing problem with many file

Hi, i need to understand if matlab ca be integrated with windows indexing service and use his data for find new file position and then elaborate it. At this time i've build a matlab application who recursive find file in a directory but cause the great amount of data the directory grow every day and more fill, more time to elaborate..... Anybody can indicate a solution to solve this issue?? Or what is the correct way... best regards, Alessandro.

[Jan]

The question is not clear. What do you want to achieve? How should the indexing service be used to find "a new file position"? Why does your tool create a large data set?

[Alessandro]

Ok, i will try to clarify. This is a matlab app who elaborate data incoming from seismic sensor. We have more than 150 sensor on ground and every sensor, every day, send lot of data (txt about 2-3 mb) in folder. The application to analyze data, scan all the directory structure (at this time this directory is about 350gb) and after this search, get the latest event and start the analisys. When we start this system the operation was really fast (about 4-5 minutes) but now, when the directory encreased, the time to solve the analisys is passed to 30-35 minutes. We have thinking to change the discovery method to check only the latest created file (today or yesterday) but is the same because the program must scan all directory to find them. At this time this app was not usable for realtime data analisys (was the scope created for...). The question is. 1)There is a method to implement a database where is writen the position of latest file.. 2)Have matlab any integrated function for directory indexing.. 3)It is possible to integrate matlab with windows indexing service (ws2008). 4)or...there is a simple walkaround to solve this issue? Our experience in matlab is not deep and we're locked at this time.

Hope now is clear. Best regards. Alessandro.

[Jan]

It is not clear, which part of the analysis takes more time. If the processing of the data is the main factor, please use the debugger to find the bottlenecks and post the code. If searching the files is the problem, please post the code also, because there could be many different causes for a slow down. It is impossible to make meaningful suggestions for the acceleration of code when the code itself is not known.

Answer 1

[dpb]

"Indexing Service is no longer supported as of Windows XP and is unavailable for use as of Windows 8. Instead, use Windows Search for client side search and Microsoft Search Server Express for server side search."

http://msdn.microsoft.com/en-us/library/aa965362(v=vs.85).aspx

Answer 2

[Jan]

If you work with Windows, you can let the operating system search for files written after a certain date:

 Date   = '08.01.2014';  % Use a date in your local date format!
 Folder = 'D:\';
 [status, result] = system(['forfiles /P ', Folder, ' /C "cmd /c echo @path" /S /D +', Date]);
 FileList = regexp(result, char(10), 'split');
 FileList = strrep(FileList, '"', '');

For the 230.000 files on my D-disk this needs 13 seconds. But I've measured it the second time I've called this function. Perhaps the file system data are stored in the disk cache, so it might be slower for the first time.

I do not see a connection to any kind of "indexing", because the problem does not concern the contents of the files. The actual problem is the idea to store 350GB of data in text files and use the date of the file system to identify new files. This is simply not the job of a file system, but this requires a data base. It is like putting a pile of index cards in a bathtub and check each single card when a specific question is asked. When a course solution can be implemented efficiently in Matlab based on the current set of files, it might be much to slow when you get twice as much files. A dedicated data base would be much smarter, because this is exactly the job data bases are designed for.