importing large 4GB+ tab-delimited ascii file makes MATLAB crash

i didnt have a problem until now the data files have been scaled up in complexity

i usually use

IMPORT = importdata('components.dat')

in this case components.dat is 4 GB+

it is a table of numerical values consisting of 12 columns and millions of rows

for smaller models this has worked but now MATLAB crashes when i try to import at this size.

I have 96GB of RAM and dual Xeons.

Do i need to change setting in MATLAB or is there better ways to import this data

this is an exmaple of the file

   1         0.01       -50.00         0.00     -3950.00 -2.0000E+04    0.000       -4000.        -1.409E-15  -2.138E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.9900E+04    0.000       -4000.        -1.410E-15  -2.193E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.9800E+04    0.000       -4000.        -1.411E-15  -2.249E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.9700E+04    0.000       -4000.        -1.412E-15  -2.306E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.9600E+04    0.000       -4000.        -1.411E-15  -2.364E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.9500E+04    0.000       -4000.        -1.411E-15  -2.424E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.9400E+04    0.000       -4000.        -1.409E-15  -2.485E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.9300E+04    0.000       -4000.        -1.407E-15  -2.547E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.9200E+04    0.000       -4000.        -1.405E-15  -2.611E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.9100E+04    0.000       -4000.        -1.401E-15  -2.676E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.9000E+04    0.000       -4000.        -1.397E-15  -2.742E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.8900E+04    0.000       -4000.        -1.392E-15  -2.810E-15    0.00E+00   0.00E+00
   1         0.01       -50.00         0.00     -3950.00 -1.8800E+04    0.000       -4000.        -1.386E-15  -2.879E-15    0.00E+00   0.00E+00

it is tab delimited

Answer 1

[Sean Phillips]

thank you for your answers. I found a very simple solution

using

M = dlmread('filename.dat')

as my data is asscii-delimited file of numerical data

Answer 2

[Steven Lord]

Consider making a tall array or perhaps using memmapfile. These tools are linked from this documentation page.

Please also send the crash log file to Technical Support so we can investigate why MATLAB crashed.

[Sean Phillips]

Hi Steven Thank you for your answer,

I have tried this so far using memmapfile

 datamem = memmapfile('components.dat', 'Format', 'double', 'Writable', true);
 D = datamem.data;

how then do i use mapped data for regualr MATLAB operations?

Answer 3

[per isakson]

That's strange. I use Win10 and R2018b (64bit). btw: What do you mean by "[...] up in complexity" ?

Anyhow try

>> num = cssm( 'components.dat' ); 
>> whos num
  Name       Size            Bytes  Class     Attributes
  num       13x12             1248  double  

where

function    num = cssm( ffs )

    fid = fopen( ffs );
    cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f'         ...
                ,   'Delimiter','\t', 'CollectOutput',true  ); 
    fclose( fid );
    num = cac{1};
end

If that doesn't work, I guess there is some strange stuff in the middle of your text file.

I added

,   'ReturnOnError',false

to the textscan call and the string, "an error", to the text file an run

>> tic, num = cssm( 'cssm.txt' );toc
Error using textscan
Mismatch between file and format character vector.
Trouble reading 'Numeric' field from file (row number 1005147, field number 1) ==> an error\n
Error in cssm (line 4)
    cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f'         ... 
>> 

textscan() found my string "an error" on the row 1005147.