Reading huge number of files quickly (I/O)
1 view (last 30 days)
Show older comments
For analysis of EOD data for stocks, I need to read 3650 .csv files, but using a for loop takes around 30 seconds for 15 files, which would mean around 2 hours just to read data. Is there any way I can do it faster?
0 Comments
Answers (1)
Walter Roberson
on 4 Jun 2012
Are you currently reading using csvread(), or import(), or load() ?
If each file has exactly the same structure (same number of fields, some number of headers, same fields are character vs numeric) then you might be able to do better by using textscan()
If you are currently using xlsread() on MS Windows then you can almost certainly do better than you are doing now (but textscan() might be better still). xlsread() opens and closes Excel each time; Image Analyst has posted code several times showing how to open Excel just once and do several operations with it.
See Also
Categories
Find more on Spreadsheets in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!