I would like to, respectfully, disagree with MATLABs' developers.
I have used MATLAB on a super-computer cluster, where i had dozens of jobs running and couldn't see what's happening till the end. Of course it depends on the type of HPC scheduler you're using. with LSF you can just use bpeek <job_name>. you dont need to flush the output
but with PBSprofessional and SGD, you just can't take a peek at the job, and the diary won't be written till the job is over.
For that, i came up with super duper solution; however, it is dependent on what is the function that you're running, which takes ages to finish.
In my case, i was running particleswarm and ga. these two functions have an option "outputFcn", which i was able to exploit.
1) everytime you call diary(<diary filename>.txt) the diary will be flushed.
2) the problem is that outputFcn and plotFcn expect an output of 0 or 1 (continue running or stop, respectively)
3) i like single-liners, i dont want to write a separate function for this. i want to do this with an anonymous function
the solution is to use the "evalc" function. inside the "evalc", you carry out these commands:
a) run the diary(<diary filename>.txt) command to flush the output; this won't stop recording the diary. it just tazes matlab into flushing the diary
b) ask matlab to output a good old '0'. a problem here is that "display(0)" and "disp(0)" will output a line-break and then our beloved zero, and another line-break. the way to do it is use "fprintf"
c) take the output from "evalc", which is a string '0' and convert it back to a numerical value
all in all, the magic trick is this:
flush_diary = @(optimValues,state) str2num((evalc(['diary([''',pathname,'diary.txt'']);fprintf(num2str(0))'])));
pso_options = optimoptions('particleswarm',...
where "pathname" is where you want to save the diary
it has to be a single liner; i know it looks ugly but you can't break it down to several nested anonymous functions
most big matlab processes have something similar to an "outputfcn" feature. try to make use of them and read their documentation.
if you have a sub-routine or function that you call repeatedly (i.e. the function where you evaluate the cost of a solution), in that function just write: diary([pathname,'diary.txt']);
if you are using parallel-computing, some unexpected behavior may occur; but i dont think it'll cause an error/failure