We have been virtualizing our application servers on Ubuntu 12.04 under VMware ESXI. VM has four cores and 4GB ram. Also happened with 8 cores and 8GB of RAM, and on two separate VMs. We are at Matlab 2013A, mexext=mexa64
$ uname -a
Linux fukushima 3.5.0-51-generic #77~precise1-Ubuntu SMP Thu Jun 5 00:48:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
$ free
total used free shared buffers cached
Mem: 4049176 3884400 164776 0 153980 2794844
-/+ buffers/cache: 935576 3113600
Swap: 4191228 486932 3704296
$ htop
shows a flood of threads like this:
11002 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.00 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11003 matlab 20 0 3272M 236M 36904 S 0.0 6.0 1h07:59 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11004 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.17 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11006 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:02.39 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11007 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:02.27 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11008 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:02.58 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11009 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:02.60 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11010 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:12.03 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11011 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.04 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11012 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.06 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11013 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.00 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11014 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:09.73 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11015 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:11.50 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11016 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.00 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11017 matlab 20 0 3272M 236M 36904 S 0.0 6.0 2:22.43 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11018 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.04 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11019 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.12 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11020 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.05 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11021 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:39.21 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11023 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.00 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11024 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.02 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11025 matlab 20 0 3272M 236M 36904 S 0.0 6.0 1:35.60 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11027 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.00 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11028 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.32 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11029 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.00 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11030 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:00.00 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
11031 matlab 20 0 3272M 236M 36904 S 0.0 6.0 0:19.78 /usr/local/MATLAB/R2013a/bin/glnxa64/MATLAB
etc.
>> clear
>> clear functions
does not help
does not help
>> feature memstats
Error using feature
Feature memstats no found
>> version
ans =
8.1.0.64
So the story goes like this. No graphics used, but lots of data pushed through matlab, on the order of ~5000 17Mb chunks before the error occurs. Command prompt remains responsive, yet after 48 hrs of inactivity, this instance of matlab remains out of memory.
Although we do not make any java calls, we suspect Java garbage collection of course!
>> version -java
Java 1.6.0_17-b04 with Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM mixed mode
We disabled all other processing to focus on this bug and it repeats reliably. The same code on other physical machines (various Ubuntu 10.04/12.04, Matlab R2012a), works just fine, and the code worked just fine under previous virtualizations we experimented with. (Not new code) Also, despite the 'out-of-memory' condition of our back processing session (it is still up), I re-instated daily processing on this host yesterday, and it ran three dozen other heavy lifting (cron) matlab jobs last night without fail. (Probably swapped a lot though)
The code uses a mex file to read in data and mym:
Matlab<->MySQL connector v1.36
So 6 physical app servers, no problem. Two VM app servers, both problem.
I searched the Mathworks site and Matlab Central, but all the out-of-memory problems I read up on involved one of three elements (which are not us): 1) Windows; 2) Graphics; or 3) Attempt to allocate too big of an array. The failure we are having seems to build up like a slow memory leak.
How can I diagnose this? Is there another way to dump Matlab's memory usage during the job so we can see what's happening? How can I force matlab to free memory when running a big job? Would you suspect Java? Have others had similar problems with VMs?
Thanks!
Clark
