What is the most memory efficient way to return large arrays in a C structure from a Matlab C mex function?

I have a program which creates a C structure which contains large arrays of various basic data types (ints doubles etc.). What is the most memory efficient way for me to return this data to Matlab from a C mexfunction, while also ensuring all of the memory deallocation is carefully taken care of? I would ideally like to return the whole structure, but methods for returning each array individually are also acceptable.

You may also assume I understand the basics of writing mexfunctions and returning arguments using the basic method of copying the data to an array pointed to by the plhs pointer. As I understand it, this will require creation of a duplicate of the memory, i.e. requiring double the memory, correct me if this is not right.

Answer 1

[James Tursa]

You cannot mix native C/C++ memory (i.e., local stack variables or allocated variables with malloc & cousins) into an mxArray for returning to the MATLAB workspace. That will eventually lead to crashing MATLAB when it tries to free this memory. So you are stuck with duplicating this memory. As I see it your options are:

1) Rewrite your code to create your C/C++ structure using MATLAB API functions mxMalloc & cousins instead of native C/C++ functions malloc & friends. Then this memory could be directly attached to an mxArray struct for returning to the MATLAB workspace ... no duplication or deallocation would be required.

2) Create your MATLAB struct piecemeal with mxMalloc & cousins as you deallocate the C/C++ memory piecemeal. This would still require you to duplicate the largest block temporarily, but saves you from duplicating everything in memory at the same time.

3) Ignore what I said about mixing native C/C++ memory and MATLAB API memory. Play games with hacking into the mxArray to mix them, keep shared data copies of them inside the mex routine to prevent MATLAB from attempting to free the memory. This is very tricky and is not recommended since you can easily leak memory and/or crash MATLAB if you don't manage everything correctly.

It doesn't save you any significant amount of memory returning several individual variables to MATLAB vs returning a struct or cell array, so just return whatever is easier to create and manage based on your intended use.

[Richard Crozier]

Thanks for answering, this is what I suspected. Is there a way of gradually copying memory from one array to another, gradually reducing the size of one and increasing the size of the other for instance?

[James Tursa]

Generally, no. When you use realloc (or mxRealloc) to decrease a block of memory by a little bit, it often results in copying the large remaining block to new memory. So you have nearly double the amount of memory allocated temporarily which is exactly what you are trying to avoid. There is no way to coax it into simply "thinking" that the current block is now smaller and freeing the trailing part.

[James Tursa]

Another (slower) option is to write your variable out to a file, then free your C/C++ memory, then read the data in on the MATLAB side.

[Richard Crozier]

Ok, thanks again for the help. I think I'll go for writing the data to a file when this is necessary.