Mex mxCreateStructMatrix and mxSetFieldByNumber massive Matlab workspace byte use.

I am trying to write a mex c file which creates a structure in matlab. 59mB of structure in C of 6 uint32's becomes 1.5GBs structure in matlab.

I don't know where I'm going wrong with this code.

       plhs[13] = mxCreateStructMatrix(1, gyro_time_mark.size(), gps_time_field_count, &gps_time_names_pointers[0]);
       for (int i = 0; i < gyro_time_mark.size(); i++) {
         uint32_t* offset = (uint32_t*)&(gyro_time_mark[i]);
        for (int j = 0; j < gps_time_field_count; j++){
          mxArray *field_value;
          field_value = mxCreateNumericMatrix(1, 1, mxUINT32_CLASS, mxREAL);
          uint32_t*  copy = (uint32_t*)mxCalloc(1, sizeof(uint32_t));
          std::memcpy(copy, offset, sizeof(int32_t));
          offset++;
          mxSetData(field_value, copy);
          mxSetFieldByNumber(plhs[13], i, j, field_value);
        }
       }

The data in Matlab is correct, its just that the structure in matlab is 1.5GB in size when it should be 50MB. The number of elements is correct and the structure field count and inside types are correct. However the data use is way way above what I would expect.

Instead of a creating a structure , I tried creating an mx6 array and that did come out to ~50mB which is exactly what I would expect. How am I using mxCreateStructMatrix and mxSetFieldByNumber incorrectly?

Answer 1

[James Tursa]

I will look at this some more tomorrow, but first, this line creates a permanent memory leak:

mxSetData(field_value, copy);

You already have a data pointer that is part of field_value since it was created as a 1x1 matrix, and that data pointer is NOT on the garbage collection list since it is already part of an mxArray. So when you call mxSetData you overwrite this data pointer and lose that memory permanently. The proper way to attach a data pointer to an mxArray is:

mxFree(mxGetData(field_value));
mxSetData(field_value, copy);

That being said, you are doing this the hard way. Instead of creating a separate memory block and attaching it to field_value you should be modifying the data area of field_value directly.

What are the values of the following?

gyro_time_mark.size()
gps_time_field_count

Also, realize that each of those mxArray field_value variables takes approx 100+ bytes of memory for overhead. I don't recall the exact number, but suppose it is 120 bytes. Then for each of your uint32 values of 4 bytes, you are adding 120 bytes for the mxArray header stuff. That is an increase of 30x memory. 50MB x 30 = 1500MB = 1.5GB. So this seems to match the memory increase you are seeing if each one of those uint32 values is inside an mxArray field value in a struct. Of course, this points out why you should not store these values in a struct like you are doing ... you should store them in a simple numeric array.

[ccasebeer]

Massive thank you for the help. I will proceed with numeric arrays.

However it does make me wonder if there is a way to reduce struct memory usage or if that is just the way it works.

Thank you for the memory leak tip. I will study it. This is the first mex I've written. The two memory pools is a bit touchy for me.

gyro_time_mark.size() is 2361565. gps_time_field_count is 6. Each field is a uint32, but it says that in the code.

Thanks! Again!

[James Tursa]

"... it does make me wonder if there is a way to reduce struct memory usage or if that is just the way it works. ..."

In general, that's just the way it works for struct and cell arrays in MATLAB. Each element is another mxArray, so each element has that 120 byte overhead. There is a way to share this mxArray header stuff across multiple elements if the data contained in the elements is exactly the same (called reference copies). But that is a special case and requires the use of an undocumented function that is not part of the official API.

"... The two memory pools is a bit touchy for me. ..."

Basically, every API function that allocates memory (with the exception of mxArrayToString) puts the address of the allocated memory on a garbage collection list. When you return from the mex function back to the caller, all of the memory that is on the garbage collection list and not part of any plhs[] variable is free'd. However, anything that is part of an mxArray data area (i.e., the pointers returned by mxGetPr, mxGetPi, mxGetIr, mxGetJc, mxGetData, mxGetImagData) is not in and of itself on the garbage collection list. That memory is bound to the mxArray it is part of and it will get free'd whenever its mxArray gets destroyed. So anything you do that detaches this data pointer from the mxArray risks a permanent memory leak unless you also manually free that memory yourself.