Saving and loading matrices with vpa entries with save and load commands is extremely slow

I am generating matrices consisting of vpa entries. They are not extremely large (1000 by 500), and when stored, the file takes only 60 MB. However, the save command takes HOURS to save them, and the load command HOURS to load them back. Is there any way to speed up the process? I have try lots of configurations without success: -v6, -v7, -v7.3, -nocompression, etc. Any piece of advice would be highly appreciated.

Answer 1

[Walter Roberson]

You are correct, this can happen. The slowdown is inside the symbolic engine, when it is asked to convert into a character representation that can be saved. The process ends up expanding all internally shared sub-expressions, and converts each identifier into a unique identifier (you cannot just use the name of the identifier because identifiers have scope in the symbolic toolbox. X - X is not necessarily 0 because they might be referring to X generated by two different functions.)

Disk speed is not the limitation here: the problem is the in-memory processing to describe (for writing) or build (upon reading) the symbolic data structures.

My experience is that this is a distinct weakness in the symbolic toolbox. I have complained about the performance to Mathworks in the past.

In my situations, I was always looking for human readable representation so I never tested further than that, but if my memory of documentation is correct, it is possible to use direct symbolic engine commands to save an expression in binary, and this is not the same as a mat file. If what I have in mind works, it would save to a binary file inside the symbolic engine, using a format specific to that engine, and you would instruct the symbolic engine to read the file directly. This would not be compatible with load() /save() and would not in itself create links between your workspace and symbolic variables that live in the symbolic engine, but you can work around those. I do not know what the performance would be at the moment.

[Walter Roberson]

Oh, snap! I did testing and I wrote up the below, but then I looked a bit further and found that the files that got written do NOT contain the content desired!

Normally I would just cancel the posting and test further, but this time I will leave it to give you ideas of what you could look into.

Remember, the below did not actually work as-written: somehow my test slipped up and showed me results as-if it had worked, and I did not notice the failure until later.

=== not-quite-functioning ideas below ===

Okay, try this. For the purposes of discussion, suppose you have two symbolic variables named A and B:

%first make a connection between the workspace variable name and the symbolic
%engine. The expression lives inside the symbolic engine, but the symbolic name
%for it is a unique identifier rather than the workspace variable name.
%After this is done, if you query the symbolic engine directly for the name, it
%will know what you are talking about

feval(symengine, '_assign', 'A', A);
feval(symengine, '_assign', 'B', B);

%now tell the symbolic engine to write the file in binary.
%it is important to use a complete file name including directory
%as otherwise it will try to write to the directory MuPAD is installed into.
%the documentation is misleading when it suggests that sometimes it might write
%into the current directory: be safe and always supply the complete path!

%File extension .mb is suggested. Use double-quotes for the file name.

tic; evalin(symengine, 'write(Bin, "/Users/roberson/MATLAB/533403/test.mb", A, B)'); toc

After that you can destroy your symbolic session if you want to; the file should be saved.

To restore:

%when you read you can use full path, or you can rely on the fact that
%it DOES look in the current directory for the file.

%use double-quotes for the file name.

tic; evalin(symengine, 'read("/Users/roberson/MATLAB/533403/test.mb"'); toc

%the variables are now loaded and live inside the symbolic engine, but 
%there is no connection between those and the workspace. So build that:

A = evalin(symengine, 'A');
B = evalin(symengine, 'B');

In my test, it took about 0.01 second to write a file that save() takes slightly longer to write out, and it took less than 1 second to read the file back in. This is an expression that takes over 14 hours to char().

[Francisco de la Hoz Méndez]

Thank you very much!

I have used your ideas, and in my case, it SEEMS to work. However, it worries me the fact that you said that the files might not contain the content desired. What happened in your case? What are the differences between the content actually saved and the content that should have been saved? Thank you again. I think that yours is a very important contribution to the MATLAB community.

[Francisco de la Hoz Méndez]

So far the only problem I've found is that symengine does not save all the decimals specified in Digits beyond some position. Do you know how to force symengine to save all the decimals?