Save Structure files in les space consuming format.

I have many structure files (.mat) as output, whose size is around 1.9 GB each. Is there any way , by which i can save this structure file in low size or space format.

Or is there any way by which I can reduce the size of this structure file, so that it takes less space on my pc.

[Walter Roberson]

Maybe, but we would need more information about the structure contents to know. One thing we would need to know is whether for different indices, the arrays stored in a given field is always the same size:

size(S(K).Fieldname) == size(S(J).Fieldname) for all indices K, J ?

What are the characters of the arrays for any one field? Is the data very fast changing, or are there similarities, M(K) is "close to" M(K+1) ?

[Priya]

I have cell array, having both strings and integers. I have around 800 cell arrays as fields in a structure file and each cell array has order of around 400 X 15 . Size of fields vary.

[Walter Roberson]

A first-pass estimate of the storage requirements would be:

800 fields times 400 times 15 cells per field times Q doubles per cell times 8 bytes per double

which is (Q * 38.4) megabytes where Q is the average number of doubles per cell (you did not give us an estimate of that value)

You ask about fitting that into "kilobytes". For the purpose of discussion, I will take that as meaning you would like to have at most 500 kilobytes of storage, as after that one would round up to speak in terms of megabytes instead of kilobytes.

>> syms Q; double(solve(1000*(500*1000) = 38400000*Q))

13.02

so if there is an average of less than 13 doubles per cell, the storage requirements could potentially be "kilobytes".

If your cells average more than 13 doubles per cell, it would be necessary to use compression techniques.

There would be overhead in marking the boundaries of which part of memory goes with which cell, so you would not be able to achieve as small as Q*38.4 megabytes with varying length arrays; the amount of overhead would depend upon the maximum length of each array.

[Priya]

It takes more than an hour just to save the files. there is one structure file which is around 3GB.

Is there any way, by which I can convert structure file into some other type like double or text, which takes less space.

However whenever I import it into matlab, it should look like same structure file as before.

Answer 1

[Jan]

Try to save it in different Mat-formats:

data = load('YourFile.mat');
save('test1.mat', '-v6')
save('test2.mat', '-v7')
save('test3.mat', '-v7.3')

Is one of them significantly smaller?

Do you store large integer arrays in the type double? Then a conversion can reduce the memory.

[Priya]

I have cell array, having both strings and integers. I have around 800 cell arrays as fields in a structure file and each cell array has order of around 400 X 15 . Size of fields vary.

[Priya]

save('test2.mat', '-v7') reduces to 50 MB. Can the size be reduced to KB's ?

[Jan]

@Priya: It is impossible for us to guess how much your data can be compressed. You could save the data to a not compressed MAT-file (perhaps using '-v4', if it is still supported, because even -v6 applies a compression under certain cirsumstances) and let e.g. 7zip compress the data with the maximum level. The resulting size is an upper limit for the achievable compression ratio. Perhaps another method can squeeze some further percent out of the data, but this is not very likely.

I'm convinced that the wanted reduction from 50MB to some kB is not possible.

[Priya]

It takes more than an hour just to save the files. there is one structure file which is around 3GB.

Is there any way, by which I can convert structure file into some other type like double or text, which takes less space.

However whenever I import it into matlab, it should look like same structure file as before.

[Priya]

Thnks