How to convert any Matlab variable into vector of bytes (decode/encode)

Hi!

I'm trying to store Matlab results/objects that I got in MongoDB. It's simple to do when you have data types such as Double / Char etc. But when you have an object (for example neural network) you need to make some tricks.

I want to create the simplest serializer from byte vector to object. What I can do, I can save a specific variable to *.mat file and then read it, and then write byte vector to the MongoDB. But it is an ugly solution, I need to make an intermediate write operation to a disk.

Do we have any conversion from Matlab variable to set of bytes and back?

x = network;
y = encodebytes(x) % <- generates from x to vector of bytes
z = decodebytes(y) % <- generates from vector of bytes to new object
isequal(x,z)

ans = 1

Thank you!

[Kirill]

Hi,

any ideas?)

[José-Luis]

Most databases can store blobs (binary large objects). That could be an alternative.

[Kirill]

Thank you.

Database Store is not a problem. I need to convert a Matlab object to a set of bytes. I know how to do it with save file / read file. But how to convert object to byte vector without saving data on a disk?

[José-Luis]

Sorry I misunderstood the question.

In that case, two alternatives come to mind:

1. Save and load from a RAM disk, would still fall under the umbrella of what you call an ugly solution but it would be faster.
2. Use the Matlab C++ API. In that case you'd need to run your Matlab code using the Matlab engine, but then you can easily manipulate your data as a bytestream.

[Guillaume]

???

3. See answer below?

[José-Luis]

That too.

Why the ???.

Redundant post is redundant.

[Guillaume]

The ??? because the bulk of the above conversation (Kirill's question and your reply) occurred after I posted my answer and seemed to completely ignore it.

[Steven Lord]

Why exactly are you trying to save MATLAB objects in a database? How do you plan to use those saved objects later?

[José-Luis]

@Guillaume: I had this open in a tab and your answer didn't show at the time.

[Guillaume]

@Steven,

One use case, I had in the past was tracability. I use (versioned) function objects to process my data and I wanted to save the exact function object used to obtain the results together with the results in a database. So when I look at the results a year later and find an anomaly, I can check the function object to see if it's the inputs that were wrong, or a buggy version of the code that was used (this used to be an immutable property of the object, but due to the limitations of loadobj, now just a private property unfortunately), or a problem with the original data.

A simple method for serialising / deserialising the class would have been very helpful.

I'm serialising using mat files saved alongside the database, which is far from ideal (and sometimes breaks because of other matlab limitations, in particular my old foe loadobj)

But the better question would be: Why can't we store matlab objects in a database (or anywhere else of our choosing)?

[Steven Lord]

What you're describing doesn't sound like a use case for a database (at least not a standalone database) to me. For what you're describing I would use (and have used) a source control system (which may have some sort of database inside.) I've done my fair share of archaeology in the source control system we use here at MathWorks to identify when a particular line of code was changed or added.

MATLAB has had the ability to interface with source control systems for at least as long as I've been at MathWorks, though the exact mechanism and level of functionality has changed over the years. If I recall correctly, the most recent major change was adding support for Git and Subversion in release R2014b. You can allow source control systems to invoke MATLAB to compare MAT-files as well.

[Kirill]

José-Luis, Guillaume, Steven Lord

Thank you guys! It helps.

[Guillaume]

@Steven,

There are two parallel things:

1. The versioning of the object (developed long before matlab knew how to talk to subversion), which indeed has nothing to do with a database. That works mostly fine, the object stores properly which version of the code was used to create it. The only time this partially breaks down is when you try to load some very old versions of objects with newer code because loadobj is not capable enough.
2. The objects are function objects used to batch process tons of engine tests. The results of the processing are stored in a database with test conditions, etc. Together with the results, I want to store all the parameters used to create them, that is: the state of the function objects. For that, I need a way to serialise/deserialise the object. That's not currently supported by matlab.

[Kirill]

For example, try this in Matlab 2014b:

x=network;
y=getByteStreamFromArray(x);
z=getArrayFromByteStream(y);

You will have an error. It's unsupported function, so we have to be aware of what we are doing ;)

Answer 1

[Guillaume]

There are no official method of serialising / deserialising objects in matlab, other than saving them in a mat file and storing the content of that file (sloooow)

There are the completely undocumented getByteStreamFromArray and getArrayFromByteStream functions. A much more thorough answer can be found on Yair Altman's blog.

Of course, since the functions are completely undocumented, there's always the risk they'll be removed in a future version. Use at your own peril.

[Walter Roberson]

https://www.mathworks.com/matlabcentral/fileexchange/34564-fast-serialize-deserialize can also be used.

[Kirill]

Guillaume and Walter.

Thanks. It will works.

What I'm thinking it's really using a RAM disk or just a cached file system and use standard and supported save / read / load functions. I will try to use getByteStreamFromArray and getArrayFromByteStream, but changing version of Matlab can be a problem.