Array exceeds maximum array size preference

Hi, given the following

v = [1 2 3 4 1 1 1 1 1 2 3 2];
P =  unique(perms(v),'rows');

I've obtained

Error using zeros

Requested 479001600x12 (42.8GB) array exceeds maximum array size preference. Creation of arrays

greater than this limit may take a long time and cause MATLAB to become unresponsive. See array

size limit or preference panel for more information.

Do you know how to fix the problem to get all the possible permutations? One idea was to divide in different cases. For example is the total nuber of permutations is 1000, is there a way to modify the command perms to get just the first 200 elements? Is there a more efficient way?

[Joel Handy]

If you look at the help for perms, it notes that the function is only practical for up to about 10 elements.

You'll likely need to write your own version of perms more suitable for your purposes. What are you trying to do with all 479001600 permutations of your vector?

[luca]

But how can I write my own permutation?

Anyway I'm generating an NP complete algorithm, and I need all the possible permutations.

[Guillaume]

Anyway I'm generating an NP complete algorithm, and I need all the possible permutations.

There's a big difference between what you want and what is possible.

As the error message tells you, generate all the (479001600) permutations of your vector with perms would require 42.8 GB of memory. It's unlikely that your computer has that much memory and even if it had, it would take a loooong time to generate.

Even if you were to generate all these 479001600, tomorrow you'll decide to extend to 13 elements (6227020800 elements = 603 GB), 14 elements (8.8 TB of memory), etc. You need to release that your current algorithm requires an exponential amount of memory and rethink it. Brute-force generating all the combinations is only going to work for up to 10 elements.

[Joel Handy]

There is a lot being asked here with very little info to go on.

We don't have any insight into this algorithm. We don't know what it's supposed to do or how it does it. This makes it hard to suggest ways of generating the permutations in useable chunks.

Regardless, processing 479 billion things is going to take a while. Even if it only takes a microsecond to process each permutation, it will take over 5 days to run.

Answer 1

[Hari Krishna Ravuri]

I understand that, you are trying to generate all permutations of the array ‘v’ resulting in generating a vector of very large size, which exceeds maximum array size preference. To know more about the variable size preferences, please refer https://www.mathworks.com/help/matlab/matlab_env/set-workspace-and-variable-preferences.html.

As you have mentioned that you are trying to solve NP-complete problem and need all the permutations, I suggest you write a small script which generates all the permutations by rearranging the elements in the array itself i.e. with using constant memory space. Now, in that script, for the first Iteration, create a .mat file and write the permutation to the file and for the remaining iterations, you just need to append every permutation to the file. For writing data to .mat file you can use save function with the syntax save(filename, variables,'-append'). For more information, please refer https://www.mathworks.com/help/matlab/ref/save.html

By writing data to .mat file, you can access all the data without bringing the entire file into the main memory.

“is there a way to modify the command perms to get just the first 200 elements?”

No, as of now, there is no such way mentioned in the documentation. Please refer https://www.mathworks.com/help/matlab/ref/perms.html

“Is there a more efficient way?”

As you are solving an NP-complete problem and need all the permutations for sure to get the optimal solution, the best way is to write all the permutations to a file and accessing them, as the entire array will not fit in the main memory. To know more about handling mat files, please refer to https://blogs.mathworks.com/loren/2011/10/14/new-mat-file-functionality-in-r2011b/

If you need a good solution but not the optimal one, you can use the problem heuristics and try to get a good solution in less time.

[Guillaume]

You could indeed generate all the permutations a few at a time (see for example, nextperm, nextvector, another nextperm, and probably more on the FileExchange) and save these to a file (mat or text), and then process them with tall arrays to avoid loading them in memory all at once, it doesn't change the fact that generating 479 millions permutations is going to take a while and processing them even longer.

And this will get exponentially worse, if the vector length is increased. 6,222 millions permutations for 13, 87,178 millions for 14, etc.

Answer 2

[Muniba Ashfaq]

I was also getting this error. I deleted all unnecessary variables from workspace to free up memory. I just keep those variables that was needed for my MATLAB code execution.

This worked amazingly.