Problem in loading a large script

Hi. I want to load a large script (67Mb) and I get a memory eror. What can I do to solve it? What is the limit about number of rows in script file? (I can run script with 16K rowes. Works slowly but work)

[Rik]

Sounds a bit odd that this should fail, but apparently it does.

Do you really need the script to be in 1 file? Why not split functions over multiple files? To avoid a folder with a massive number of files, you can also use a function database for functions that are only a few lines like this:

function varargout = f_database(fid, varargin)
fh = str2func(fid);
[varargout{1:nargout}] = fh(varargin{:});
end
%example:
function value_out=add_two(value_in)
value_out=value_in+2;
end

You can call inside functions with this syntax:

val=f_database('add_two',10);

(and yes, I see the contradiction of suggesting splitting up functions and making a function database)

[Stephen23]

@Ehud ikar: can you confirm that this is really a script, i.e. an Mfile containing MATLAB code?

It seems rather large for a script. I can't image that its McCabe complexity is very low...

[ehud ikar]

Hi I calculate performances of 30,000 markov cains together. I have a code generator that create the script file. I add an example of a little file that I can run.

[Stephen23]

@ehud ikar: Why bother generating all of those variables and lines of code? It would be much simpler to write this using some loops, some indexing, and a handful of variables. This could be done using just a few lines of MATLAB code, without requiring a "code generator" or any of the bad practices that this code contains (numbered variables, copy-and-paste similar code, repetition without loops, etc.).

https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

[Jan]

@ehud: Please post a complete copy of the error message instead of the lean rephrasing "I get a memory error". Details matter.

Answer 1

[Jan]

A huge script, generated automatically, global variables, numbered variables - a lot of bad programming practices. It is not clear to me, what causes the "memory error" (see above: please post the error message instead of this lean description), but of course there is a limit, where Matlab cannot parse or run such files anymore. Obviously you hit the limit. Then either install more RAM, or much better: use clean programming techniques. The fact, that this code generator works for small problems sufficiently does not allow to draw conclusions about using it for large scale problems.

I'm convinced that there are much easier methods using loops and multiple functions. It is a better approach to use arrays instead of automatically created code. Better use one tool (code) and vary the contents of the data (variables), instead of creating a code for each problem. Your problem is another proof for this good programming practice.

[ehud ikar]

I have a problem that it is impossible to solve it without a code generator. There is a system represented by Markov chains by two states. If I have 2 such systems in a parallel connection, their equivalent Markov chain will have three modes. If I have 3 such systems in a parallel connection, their equivalent Markov chain will consist of 4 modes. There is a kind of recursion here. For compute 3 systems in parallel, should calculate earlier, 2 parallel systems. In my problem there are 90 systems at the same time, but it does not end here. I have 290 systems in a queue, when each system is built from 90 parallel systems. (A type of system matrix size 290 * 90) That means I have to do 380 sets of calculations to get to the last position. I am not a software engineer, but in my opinion and in the opinion of my academic advisor, there are no unnecessary steps in this process. So I ask again: What is the limit of the number of lines allowed in the script? Thank you for your attention.

[Stephen23]

"...in the opinion of my academic advisor..."

I have read many papers and articles, and the quality of code coming from academics is often quite poor. It would be a mistake to assume that someone is competent in writing code, just because they are an "academic advisor". Both undergrad and grad students would do themselves a big favor if they gained some outside perspectives on their code.

"I have a problem that it is impossible to solve it without a code generator"

That is a very strong statement. How did you prove this? What other methods have you investigated or tried? Why did you reject those methods? What other methods are you knowledgeable about?

I still don't see anything in your description that requires generating huge, complex code. Simple loops would seem to be enough to do what you explained, or some simple recursive code perhaps. And this would resolve the script-loading issues that you claim that you are facing.

[Jan]

 in my opinion and in the opinion of my academic advisor, there are no
 unnecessary steps in this process

An experienced software engineer will not share this opinion. Automatic code generation is a very bad idea, if such huge scripts are created, which create a massive number of variables. It is not only the number of lines, but such programming technique let the local workspace and the lookup table for the variables explode.

Think twice: The description of your problem explains, that the used method fails. We tell you, that your method is known to cause such problems and it is deprecated in consequence. Now you answer, that is the opinion of you and your advisor, that the method is okay. No, as you see, the method fails.

Again: Please post a copy of the error message to let us know, what the problem is exactly.

And use a dynamic programming technique, which varies the data by using variables, instead of creating hard coded functions automatically. You are not the first programmer, who fails with this approach when the problem grows.