The requested array exceeds the maximum allowed variable size

Hi,

When I use meshgrid to generate 3D mesh, I got a error as below:

Error using repmat

Requested array exceeds the maximum possible variable size.

Error in meshgrid (line 80)

xx = repmat(xx, ny, 1, nz);

where my code is ab below and posx,posy and posz are the 1*83014 double.

[mesh_posx,mesh_posy,mesh_posz] = meshgrid(posx,posy,posz);

[Chunru]

The meshgrid would produce array of size 83014^3, which is too big (around 570T elements, 1T = 10^12)

83014^3

ans = 5.7208e+14

[Dyuman Joshi]

When you get this particular message - "Requested array exceeds the maximum possible variable size.", it means your computer does not have enough RAM to create the data you require.

The size of mesh_posx will be [83014 83014 83014], let's see how much RAM you need for just the one variable -

fprintf('%f Tera Bytes of RAM', 83014*83014*83014*8/1024^4) %8 bytes for double

4162.403542 Tera Bytes of RAM

I seriously doubt that you have this much RAM available, let alone 3x of that. Even some Super Computers don't have this amount of RAM.

I suggest you re-evaluate what you are trying to do.

[Steven Lord]

When you get this particular message - "Requested array exceeds the maximum possible variable size.", it means your computer does not have enough RAM to create the data you require.

No, that's a different error message. The message about which @Jiawei Xu asked is where the requested array is larger than MATLAB can theoretically create. The requested array would have more elements than the maximum number of elements an array can possibly have in MATLAB:

format longg
numElements = 83014^3

numElements = 
           572076386806744

[~, maxNumElements] = computer

maxNumElements = 
           281474976710655

tooBig = numElements > maxNumElements

tooBig = logical
   1

That's not a MATLAB specific limitation, but a limit of the x86-64 architecture address space.

[maxNumElements; 2^48-1]

ans = 2×1

1.0e+00 * 281474976710655 281474976710655

If you tried to allocate more memory than you have available but still less than the theoretical limit you may receive an error like the following (pasted as text not code because I don't want MATLAB Answers to even try creating such a huge array.)

>> x = zeros(83014, 83014, 1000);

Error using zeros

Requested 83014x83014x1000 (51344.4GB) array exceeds maximum array size preference

(63.6GB). This might cause MATLAB to become unresponsive.

or the more succint (if you don't have the maximum array size preference enabled)

>> x = zeros(83014, 83014, 1000);

Out of memory.

Answer 1

[chicken vector]

A double in Matlab occupies 8 bytes of memory and you are creating 3 cubic arrays with 83014^3 doubles.

This corresponds to more than 4 TB of space, which, even if you had, would require a lot of computational time for allocation only.

As far as I know, you have three ways of handling the problem.

You can also combine these methods to obtain a better solution.

1) Reduce size of the problem

I don't know what you are using this mesh for, but if you want to have a cubic mesh you are obligated to reduce the size of the problem.

You can probably find useful interp1, interp2 or interp3 for this purpose.

% Old and new problem size:
problemSize = 100;
subproblemSize = 10;

% Old problem:
x = linspace(-50, 50, problemSize); y = x; z = y;
[X, Y, Z] = meshgrid(x, y, z);
sphereValues = X.^2 + Y.^2 + Z.^2;
V = 1./(sphereValues/max(sphereValues, [], 'all') + 1);

% Interp to new grid with fewer points:
xq = linspace(-50, 50, subproblemSize); yq = xq; zq = yq;
[Xq,Yq,Zq] = meshgrid(xq, yq, zq);
Vq = interp3(X, Y, Z, V, Xq, Yq, Zq, 'Cubic');

% Show difference:
figure;
tiledlayout(1,2);

nexttile;
slice(X, Y, Z, V, 0, 0, 0);
shading flat;

nexttile;
slice(Xq, Yq, Zq, Vq, 0, 0, 0);
shading flat;

2) Change variable type

Another advice to avoid reducing the problem too much, is to change the precision, e.g. from double to single, or even half.

A = randi(1e3,1e4,1e4);
fprintf(['Variable A occupies ' num2str(whos('A').bytes/1.024e6) ' MB of memory']);

Variable A occupies 781.25 MB of memory

B = single(A);
fprintf(['Variable B occupies ' num2str(whos('B').bytes/1.024e6) ' MB of memory']);

Variable B occupies 390.625 MB of memory

C = uint16(A);
fprintf(['Variable C occupies ' num2str(whos('C').bytes/1.024e6) ' MB of memory']);

Variable C occupies 195.3125 MB of memory

Be careful as the precision you decide to use MUST always be based on the numbers you are dealing with and the interval of confidence you desire:

% Variable types have different intervals of number they are able to represent:
x = 65700;
fprintf('Double: %d\nHalf:   %d\nUint32: %d\n', x, half(x), uint32(x));

Double: 65700
Half:   9223372036854775807
Uint32: 65700

% Variable types store numbers with different levels of accuracy:
x = 1.4321000;
fprintf('Double: %.7f\nHalf:   %.7f\nUint32: %.7f\n', x, half(x), uint32(x));

Double: 1.4321000
Half:   1.4316406
Uint32: 1.0000000

3) Divide the problem into smaller subproblems

Finally, a third option, is to divide your cubic mesh into submeshes and work on the problem locally.

Your problem size is not very convenient for that since:

factor(83014)

ans = 1×2

2 41507

But the process to follow would be something like:

% Problem input:
problemSize = 83;
subproblemSize = 10;
posx = linspace(1, 10, problemSize);
posy = posx;
posz = posy;

% Divide into subproblems:
sub_posx = mat2cell(posx, 1, diff([0:subproblemSize:problemSize-1, problemSize]));
sub_posy = mat2cell(posy, 1, diff([0:subproblemSize:problemSize-1, problemSize]));
sub_posz = mat2cell(posz, 1, diff([0:subproblemSize:problemSize-1, problemSize]));

% Access subblock [2, 6, 4]:
x = 2; y = 6; z = 4;
[submesh_posx, submesh_posy, submesh_posz] = meshgrid(sub_posx{x}, sub_posy{y}, sub_posz{z});

% Plot:
c = 1:numel(submesh_posx);
sz = 75*cos(c/c(end)*3*pi) + 100;
figure;
scatter3(submesh_posx(:), submesh_posy(:), submesh_posz(:), sz, c, 'filled', 'pentagram');

You can use nested for loops to perform operations on every subblock:

% Loop over every sub-block:
for x = 1 : length(sub_posx)
    for y = 1 : length(sub_posy)
        for z = 1 : length(sub_posz)
            [submesh_posx, submesh_posy, submesh_posz] = meshgrid(sub_posx{x}, sub_posy{y}, sub_posz{z});
            % _______ [Do stuff here]
        end
    end
end

Depending on the simplicity of your problem, you might also be able to exploit vectorisation saving a lot of time.