Dynamic allocation of mxArray structure

Hi everyone,

I have a mex function in C that internally generates an unpredictably large volume of data to be output back to Matlab. I would like the return to be a Struct Matrix of the 1 by N, where N is the unknown large number. The memory should be allocated by chunks, so that in case of insufficient memory the calculation can be stopped and the return that does fit into memory handled safely back to Matlab.

It seems that the combination of mxSetN() and mxRealloc() is a proper combination, but I can't find sufficient info on how to use mxRealloc to allocate space for additional elements of structural matrix.

Here is a minimal code example of what I'm trying to do

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) 
{
  mwSize i;
  mxArray *mxp;

    nlhs[0]=mxCreateStructMatrix(1,1,0,NULL); //structure with one element and no fields for now

    while (continue calculation)
    { 
      //one more round of calculations that creates a lot of output
      //stored in structure g with double* field1,..,K

      mxAddField(nlhs[0],"field1");
      mxp=mxCreateDoubleMatrix(appropriate,size,mxREAL);
      if (mxp==NULL) {mexErrMsgTxt("Failed to allocate memory!")}
      mxSetField(nlhs[0],i,"field1",mxp);

      ...

      mxAddField(nlhs[0],"fieldK");
      mxp=mxCreateDoubleMatrix(appropriate,size,mxREAL);
      if (mxp==NULL) {mexErrMsgTxt("Failed to allocate memory!")}
      mxSetField(nlhs[0],i,"fieldK",mxp);

      i++;

      mxSetM(nlhs[0],i)
      //What then?

    }
  }

[Fedor Iskhakov]

I think what I need is right here

Answer 1

[Jan]

Letting an array grow iteratively consumes a lot of resources. Therefore it is a good idea to create the new data in chunks and reply them separately.

As James said already, mxCreateDoubleMatrix stops the Mex function immediately if it is not successful. Then all formerly creatzed data are lost. A workaround would be to call the Mex function for each junk in a loop and collect the data in Matlab:

allChunk = cell(1, 1000);
iChunk   = 0;
while ~ready
  try
    newChunk = myMexfunction();
    iChunk   = iChunk + 1;
    allChunk{iChunk} = newChunk;
    if iChunk == numel(allChunk)      % Container exhausted
       allChunk{iChunk + 1000} = [];  % Expand
    end
  catch
    disp('Memory exhausted');
    ready = true;
  end
end

But if te MEX function could not allocate any memory anymore, Matlab is in danger to crash also. You cannot do a lot of computations anymore, most of all you cannot combine the chunks to a large array.

Perhaps it is better to write the junks to the disk (SSD preferrably), such that an exhaustion is less likely and the memory is not filled at all. This would be much better than a strategy which is expected to cause an out-of-memory crash.

[Fedor Iskhakov]

Thanks, saving to disk does sound like most reasonable thing I can do. Running the loop in Matlab would also be more problematic in my application..

Answer 2

[Philip Borghesani]

Although the code example you linked to will do the resizing operation on a structure and in general I like it better then the code you posted. I suggest not ever resizing the main structure but allocating it to some reasonable max size from the beginning. Resizing and reallocating operations are slow and tend to cause memory fragmentation issues. In addition structure arrays have quite a bit of memory overhead proportional to the dimensions of the structure.

Hopefully (and if not try to fix it) the amount of memory used by field1 is relatively large compared with the size of your overall structure. If size is a few thousand or larger then the memory used to store the basic structure is insignificant and it can be allocated to it maximum size initially. Your function can return the max "i" reached as a second output or the calling code can just check for empty fields when processing the data and stop.

[Fedor Iskhakov]

I take your point, thanks, I'm aware that I'm looking for a "second best" here. The output format can not be changed in the short term, but in the longer term we should reconsider this whole setting.

Answer 3

[James Tursa]

"... The memory should be allocated by chunks, so that in case of insufficient memory the calculation can be stopped and the return that does fit into memory handled safely back to Matlab. ..."

You can't do this in a mex routine.

In these lines (and other similar lines in your code):

      mxp=mxCreateDoubleMatrix(appropriate,size,mxREAL);
      if (mxp==NULL) {mexErrMsgTxt("Failed to allocate memory!")}

mxp will never be NULL and you will never get inside the if-block. Inside a mex routine, if there is an API call that results in an out of memory condition, the mex routine will immediately return control to the caller with an error. In that situation it will never execute the next line in your code (i.e., you will never return from the mxCreateDoubleMatrix function call). The only way you actually get a NULL value from API function calls in your code that allocate memory is if they are used in an Engine application ... it never will happen in a mex routine.

[Fedor Iskhakov]

Thanks for pointing this out. I did notice that mxRealloc() behaves exactly like this, but I was sure the check for NaNs returned from mxCreate** routines worked before to detect memory shortage. What a bummer!