Memory Issues: Almost like clear doesn't fully work

11 views (last 30 days)
I am running 32b vista, Matlab 2009a. The machine has 4GB of RAM installed, PAE & 3G switch activated. the machine is dedicated to MATLAB and doesn't run anything else.
I have code that reads and writes data from disk. Data is stored in the form of txt files. There are thousands of text files around 200MB each in size. The code loads the text file into RAM processes it and then writes it back to disk.
Each load is done within a loop
for i = 1: numFiles
loadAndWriteData(file(i))
end
function loadAndWriteData(file)
load file
do stuff
save file
clear file (and all other variables I can see lurking around)
end
hence any RAM intensive processes are done in loadAndWriteData.m This returns on each call of the for loop.
I use memory.m to inspect the memomry availble to me. I see that after a few runs around the available memory starts dropping off. Then Matlab crashes with an out of RAM error.
I would have thought that as the function is returning each time around the loop, I would be nicely clearing everything in RAM.
What I am doing wrong? Any help gratefully received. thank you.

Accepted Answer

mathworks2011
mathworks2011 on 21 Apr 2011

More Answers (3)

mathworks2011
mathworks2011 on 21 Apr 2011
Since MATLAB uses a heap method of memory management, extended MATLAB sessions may cause memory to become fragmented. When memory is fragmented, there may be plenty of free space, but not enough contiguous memory to store a new large variable.
Just to append this. One possible solution is to quit matlab and to start again. This is clearly non-sensical though.
Another, which has the same effect, is to use the pack command. As this cant be used from the command line it is essentially useless too.
This seems to be the most ridiculous problem: essentially it means that matlab can not be used for any processes that require large amounts of data....??!!
  2 Comments
Jason Ross
Jason Ross on 21 Apr 2011
Your suggestion of quitting MATLAB and starting again isn't so nonsensical. If this program was amenable to using functions from the parallel computing toolbox, you can set the RestartWorker on the job object to do exactly that.
http://www.mathworks.com/help/toolbox/distcomp/restartworker.html
mathworks2011
mathworks2011 on 21 Apr 2011
noted and good point.
However, 1. I dont have the toolbox and (2) I feel such a basic issue shouldnt require specialist treatment.
I dont rememeber much of my C++, but fragmentation isnt an issue there or in C# etc. The problem being matlab doesnt allow you to ever allocate memory according to stack/heap (I dont think...)
http://www.go4expert.com/forums/showthread.php?t=9669

Sign in to comment.


Matt Fig
Matt Fig on 21 Apr 2011
How are you clearing the files? Are you using FCLOSE and checking the return variable?
EDIT
I am not certain that clearing the file identifier will close the file. I don't see it in the documentation...
fid = fopen('myfile')
clear fid % Is the file in memory or not?
fclose(fid) % Does using this instead of clear help???
  5 Comments
mathworks2011
mathworks2011 on 21 Apr 2011
Im not clear on what you mean when you say "Why not just use a script instead of functions? That way you can have access to the pack command".
You do sketch out some code for me?
The reason I never use scripts, is that the architecture of the code is actually very complex (much more so than in MWE I showed here).
Matt Fig
Matt Fig on 21 Apr 2011
A script is an M-file which has no keyword: function in it. For example, this is would be a function M-file:
function B = mysquare(A)
B = A.^2;
and this would be a script M-file:
B = A.^2;
The script will actually execute in the base workspace, and so can call PACK. To change a function to a script, simply copy and paste all of the code into a new M-file. Note that scripts can call function M-files, so just put the bare-bones in a script, then you can call PACK and other command line functions.

Sign in to comment.


aasim Azooz
aasim Azooz on 22 Apr 2011
Try this for i = 1: numFiles loadAndWriteData(file(i)) clear all end
if it does not work, you may have a virus in your computer. It happened with me. everything went fine after I scanned my PC for viruses. good Luck Aasim Azooz

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!