Fast Export Method
8 views (last 30 days)
Show older comments
I have always used the export command to export a dataset to a comma delimited text file. When the files get a little larger (50-100MB) the export function seems to run very slow. Are there other functions that are much faster than the export dataset function?
My dataset is simple (just large). Col 1 is text 2:4 are numeric.
MyDS = dataset(MyData(:,1),MyData(:,2),MyData(:,3),MyData(:,4));
export(MyDS,'file','R:\Equity_Quant\BrianB\Factor Rotation\BulkData.txt','Delimiter',',');
Thanks much, Brian
3 Comments
Walter Roberson
on 12 Jun 2012
export() is a method of the dataset class.
http://www.mathworks.com/help/toolbox/stats/dataset.export.html
Accepted Answer
per isakson
on 12 Jun 2012
The functions save, load and whos is an alternative
save( 'my_datasets.mat', 'MyDS' )
save( 'my_datasets.mat', 'MyDS_2', '-append' )
or even faster
save( 'my_datasets.mat', 'MyDS', '-v6' )
without guarantee. (Dataset does not overload save as far as I can see.)
--- Faster method to export to text file ---
Some data. I use three columns to avoid word wrap.
N = 1e1;
MyData = randn( N, 3 );
The variant with dataset
tic
ds = dataset( MyData(:,1), MyData(:,2), MyData(:,3) );
export( ds, 'file','c:\temp\test_ds.txt', 'Delimiter', ',' )
toc
produces this output
Var1,Var2,Var3
0.490621219889722,-0.64500446971637,1.6174852368957
0.0660146644635964,-0.408559384693175,0.445251540387096
...
A faster variant
tic
ms = permute( MyData, [ 2, 1 ] );
fid = fopen( 'c:\temp\test_fp.txt', 'w' );
fprintf( fid, '%s,%s,%s\n', 'Var1', 'Var2', 'Var3' );
fprintf( fid, '%f,%f,%f\n', ms );
fclose( fid );
toc
produces this output
Var1,Var2,Var3
0.490621,-0.645004,1.617485
0.066015,-0.408559,0.445252
...
I've run these two variants with different values for N. The faster variant is at least an order of magnitude faster.
With N=5e6 on my three years old vanilla desktop I get "Elapsed time is 16.478986 seconds." with the faster variant. That is 8 MB/s - something.
How many decimals do you need in the text file?
.
--- fprintf is hard to exceed ---
I added this test
tic
dlmwrite( 'c:\temp\test_dlm.txt', MyData )
toc
With N=5e6 I got the following elapsed times
- fprintf (A faster variant): Elapsed time is 16.507707 seconds.
- dlmwrite: Elapsed time is 202.905913 seconds. (without header)
- dateset.export: Elapsed time is 819.649789 seconds.
With plain Matlab I don't think there is a faster alternative. Maybe it is possible to do something faster with a MEX-function.
3 Comments
per isakson
on 13 Jun 2012
That's correct. I didn't realize the purpose. Maybe it is an option to write directly to the database.
More Answers (0)
See Also
Categories
Find more on Data Import and Export in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!