creating a big matrix (2000x2000x2000)

I want to create 2 big 2000x2000x2000 matrices in Matlab 2009a.

currently, Matlab is limited in his memory, and I'm looking for a way to get past it.

Answer 1

[Walter Roberson]

If you are using a 32 bit version of MATLAB (which was still possible in the time of your R2009a release) then what you are asking for is not possible.

If you are using a 64 bit version of MATLAB, then what you need to do is configure yoiur system to support about 180 gigabytes of swap space, and turn off the MATLAB array size limit in preferences; https://www.mathworks.com/help/matlab/matlab_env/set-workspace-and-variable-preferences.html . Be aware that this is likely to be very very very slow.

In the special case that the great majority of the entries are zero, then you might be able to take advantage of sparse

If you were to upgrade to at least R2016b then you would be able to start using tall

[Or]

I'm using the 32-bit version.

Is there any other way to get past this? How does image and video processing perform otherwise?

the variables can be also unit8.

[Walter Roberson]

There is no hope for 32 bit matlab unless you can use sparse arrays. With 32 bit matlab it was sometimes possible to fudge the memory limit between 2 gigabytes (2*2^30) and 4 gigabytes, but each of your arrays requires 8 gigabytes.

[Walter Roberson]

How does image and video processing perform otherwise?

You have a 2000 x 2000 x 2000 array. That is not an image. It might be 2000 frames of a video in which each frame is 2000 x 2000 .

The key to processing videos that cannot entirely fit into memory, is to process a frame at a time, possibly creating summary variables as you go. Something like

videoobject = VideoReader('VideoNameHere.mp4');
numframes = 2000;
framesize = [2000, 2000];
totalframe = zeros(framesize, 'double');


for idx = 1 : numframes
    thisframe = readFrame(videoobject, idx);
    totalframe = totalframe + double(thisframe);
end

avg_frame = totalframe ./ numframes;

for idx = 1 : numframes
    thisframe = readFrame(videoobject, idx);
    this_minus_average = double(thisframe) - avg_frame;
    %now do something with this_minus_average
end

[Or]

basicelly i need 2000 mtrices of [2000,2000] size.

so maybe a better option for me will be saving the mtrices as txt files?

is there another way?

is there a way to save all of the files together?

[Rik]

I suspect mat files would be easier to store and retrieve.

If you can upgrade to R2016b or later you will have access to tall arrays. This datatype allows you to use arrays that are larger than your memory. But since you're using R2009a in 2023, you probably a have a good reason not to upgrade.

[DGM]

Storing images as text (e.g. CSV) is incredibly wasteful. It actually takes up far more space than writing raw uncompressed binary data. Consequently, reading the data from the text file and converting it back to numeric will require as much as 7x as much peak memory use compared to reading raw uncompressed uint8 data (assuming the text is read in at 2B/char).

% a 2000x2000 single-channel image
inpict = uint8(randi([0 255],200,200));
inpict = imresize(inpict,[2000 2000],'nearest');

% write a giant CSV file
writematrix(inpict,'test.txt')
fileInfo = dir('test.txt');
fileSize = round(fileInfo.bytes/1000) % kB per frame

fileSize = 14262

% write a MAT file 
save('test.mat','inpict')
fileInfo = dir('test.mat');
fileSize = round(fileInfo.bytes/1000) % kB per frame

fileSize = 266

% write a normal PNG file
imwrite(inpict,'test.png')
fileInfo = dir('test.png');
fileSize = round(fileInfo.bytes/1000) % kB per frame

fileSize = 93

If you're storing frames/slices as individual single-channel images in uint8, I don't see why you wouldn't just use PNG. If you wanted to do something like store multiple frames in a single file or use floating point data, then maybe TIFF or MAT would be the way to go. Which of those produces the smallest files depends heavily on image content. My main point is only that MAT files and image files will be much smaller than using text.

Mind, we still don't know what exactly you're trying to do with 2000 image frames/slices, so it's not clear if or how writing them to disk in any format helps achieve those goals. One would think that if the original data cannot exist in memory, then it must already be on disk to begin with.

[Or]

my purpose is to divide a group of parts into groups of 3.

using this 3D matrix, I wanted to find and save the best suitable groups. That is why I need a 3D matrix, since I have 2000 parts I will need a [2000,2000,2000] matrix.

I tried using unit8 variables, which can work for me, but it is still too big for 2009a 32bit.

[Rik]

How do find 'the best suitable groups'? Can you show an example with 25x25x2000 (which will only take 1.25MB of memory, instead of 8GB of contiguous memory)? We might be able to guide you to an implementation that will work for the full size.

data=uint8(ones(25,25,2000));whos

  Name         Size                   Bytes  Class    Attributes

  cmdout       1x33                      66  char               
  data        25x25x2000            1250000  uint8              

[Or]

the whole 2000x2000x2000 will be ones and zeros, which will tell if the threesome is good or not.

Then I can keep forward when having the threesomes that pass the threshold criteria.

[Steven Lord]

Can you say more about why you're using release R2009a and a 32-bit version of MATLAB? Moving to a 64-bit version of MATLAB wouldn't necessarily completely solve the problem, but it would remove one major limitation (the limit the OS imposes on address space.)