slicing a char array seems to internally allocate far more memory than expected

3 views (last 30 days)
My laptop has 64G ram.
I am reading a large text file (> 8G) with s=fileread(filename).
The resulting char array is
K>> whos('s')
Name Size Bytes Class Attributes
s 1x8656516488 17313032976 char
Grabbing everything but the first, e.g. 200 characters results in this:
K>> v = s(201:end);
Requested 8656516288x1 (64.5GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive.
See array size limit or preference panel for more information.
It seems that slicing a char array allocates 8x(number of chars). Is this expected?
[[Yes, I know of plenty of workarounds -- datastores, reading it pieces, fscanfs, etc., but this code is already a workaround to deal with the glacial speed of readtable(), so, I don't have to have it working, it's just somewhat surprising]]

Accepted Answer

Walter Roberson
Walter Roberson on 16 Feb 2022
That indexing 201:end is not internally optimized as passing subsref 'type', '()', 'subs', {{201, ':', 8656516488}}) and expecting the internal implementation of subsref to handle the range.
What is done instead is that the colon operator 201:8656516488 is executed, producing a double precision array of length 8656516288 to hold the indices, and that array is passed to subsref. But that array of indices is 64 gigabytes...
Copy the first 200 elements out of the array, and then
s(1:200) = [] ;
which only needs creating a vector of length 200 of double precision indices.
  3 Comments
Dmitry Kaplan
Dmitry Kaplan on 17 Feb 2022
Thank you. That's exactly the road I was following. Matlab is getting better at the "tens of gigabytes", but isn't quite there with the "hundreds of gigs or more" yet. Multithreaded C (or typescript) shreds through that much data, but yes, that's an unfair comparison. Thank you again for your help.

Sign in to comment.

More Answers (1)

Sourav Karmakar
Sourav Karmakar on 16 Feb 2022
Hey Dmitri,
I've tried to create a char array of size 1x8656516488, but as it exceeds the maximum array size, it throws an error message. For example,
>> s = char(1:8656516488);
Requested 8656516488x1 (64.5GB) array exceeds maximum array size preference (48.0GB). This might cause
MATLAB to become unresponsive.
As you have 64G RAM , it should produce the same error. Try creating the 's' char array in your matlab workspace and check whether you are getting the same error message or not. Because, slicing only 200 chars from the array, does not change significantly in the memory( see difference between sizes of 's' and 'v' ).
You can refer to the following document for more reference:
Hope this helps!
  1 Comment
Dmitry Kaplan
Dmitry Kaplan on 16 Feb 2022
Edited: Dmitry Kaplan on 16 Feb 2022
Thank you. I belive that Walter's answer below is getting to the crux of the problem. The real issue is that I can easily allocate two 8G int8 arrays, I simply can't copy huge pieces of one into another without the internal allocation of huge double arrays to hold the indices.

Sign in to comment.

Products


Release

R2019a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!