No out of memory error but matlab crash

Hello,

I use matlab 2022a on ubuntu 20.04.

These two codes crash matlab without generating an error message :

ii=1;
while 1
img{ii}=rand(2048);
ii=ii+1;
end % until matlab crash

and

a = rand(2048,2048,2000); % ok
b=a;                      % still ok but if b is really allocated memory should not work on my system
b=a+1;                    % matlab crash

I would be expecting an out of memory error and not a crash. Is there a reason I do not get an error message ?

Thank you

[Rik]

The run tool on this website also seems to struggle (it is running Linux as well):

a = rand(2048,2048,2000);

Error using rand
Requested 2048x2048x2000 (62.5GB) array exceeds maximum array size preference (30.9GB). This might cause MATLAB to become unresponsive.

a = rand(2048,2048,950); % results in some sort of error

The reason why your second code only fails when doing +1 is that the second assignment doesn't require a new array in memory, since it is the same as a. Matlab will generally only use copy-on-write, meaning that a new array will only be created in memory when there is a need to write changes.

[Matt]

But is it not expected that matlab crash no ?

I am expecting simply an error message doing this +1 saying something like buy more ram or add some swape space. My (realistic) hope is that it crashes because something in misconfigured on my pc/matlab and somebody would know what :)

[Rik]

I would personally call this a bug. If you request too much memory you should get an error like I showed. Did you submit a bug report already?

I am not aware of any configuration you could change (other than adding more physical RAM and swap space). The fact that the run tool also fails indicates to me that it is unlikely to be a problem with your own configuration. (although it is of course possible Mathworks made some mistake when configuring the installation of the run tool)

[Matt]

No, I tend to believe that when something goes wrong on matlab it's because I misunderstood something rather then I hit a bug. And so far this assumption has proven valid !

I do get the warning you indicated, just for a larger array size because of the amount of ram of my pc. But if I try to ask for more memory then the pc has in severals or 2 attemps (so the first and second example of my last post) matlab crash.

Answer 1

[Steven Lord]

Did MATLAB crash or did the operating system kill it? Check if there's a message in the log file that the Out of Memory (OOM) killer terminated the MATLAB process. If it did then this is not a bug in either MATLAB or the operating system.

[Matt]

Hi,

Yes OOM did kill Matlab ! Thank you.

So I guess that in conclusion I must be caution of OOM kills Matlab and I lose my work ... It's not great but clear.

For information the logs where in var/log/kern.log not var/log/message, so to check the command is

grep -i -r kill /var/log

[Steven Lord]

I couldn't find an Ubuntu-specific page talking about the OOM killer, and the one I posted is a bit old, so I'm not surprised the information is in a different log file than the one named on that page.

Answer 2

[Jan]

It is expected, that creating a bunch of large arrays (1st example) crash an application. You cannot do much gainst it, because filling the available memory creates a situation, in which noch enough memory is available to display an error message: this would need memory to open a new figure and even updating the text of the command window costs RAM.

In the 2nd example, the duplication of the array is performed inside a+1. Again there is no reliable chance to predict in advance if the memory is exhausted after the allocation of the additional RAM.

I consider the behavior in both cases as a bug - not of Matlab, but of the operating system. There should be a method to tell the OS: "I need at least this amount of memory for an error message. If an operation does not leave this part free, stop it." But neither Linux, nor Windows, nor any other operating system I know is capable to do so. The same problem is unsolved for file systems: If you fill the drive of the OS until the last byte (or kB usually), the system falls into an undefied state and will show random behavior. Even posting a warning can request some bytes of the disk (caching or logging), such that the power to control the system is lost. For stablility the rule of thumb is: leave 5% of the fisk free - very coarse...

Conclusion: When the memory is exhausted, Matlab (and any other application including the OS) drives into an undefined state. This is a bug and there is no known method to catch the problem reliably in general. Therefore the responsibility remains by the programmer: Check the availability of RAM in advance.

Remember, that this problem does not concern Computers only: Even physical systems suffer from such undefined states due to exhausted resources: A fuel tank can become unusable, if you fill it completely without leaving any gas in it: Then trying to extract some fuel can be prevented by the force of the vacuum.

[Matt]

I understand your point but it seems weird to me that the OS cannot do this. If it was the case we would have computers running google chrome and a thousand tabs shutting down every day and it's not the case, isn't it ? Or is it because applications are designed with care and developers before doing a large malloc ask manually the os if enough memory is available ?

To add to the fuel tank metaphor, you have a system on the fuel pomp that prevents the tank from overflowing when filling, and it is usefull.

[Jan]

Chrome and Firefox control the memory actively, so they are a cery good example. In opposite to a programming language, the browsers can control, how much memory is requested to accomplish a job.

You could write your own memory controller and encapsulate all rand(), zeros(), ones(), ... functions. The wrapper must ask feature memstats if there is a matching free block of memory. But this costs runtime and you cannot go deep enough to catch a+1. Such a wrapper would be equivalent to a measurement of the filled-in fuel to avoid exhausting the resources.

Overflow is a serious problem in all fields of science and numerical maths. It is hard to catch unexpected exceptions.