How specific lines in a text file can be updated many times ?

Dear all, Please how can I update specific lines or specific matrix which found in the middle of a text file. Actually, I have jacobian matrix needs to be updated at every iteration and I should save the changes in the same file.

Many thanks for your help in advance.

[Cedric]

Why not computing/updating the Jacobian matrix directly in Fortran?

[ND]

As my job should be done within Matlab, using Matlab just for calling and running other software packages.

Answer 1

[dpb]

Simple answer is "you can't, at least easily". Sequential text files are, well, "sequential". Even with the complex ways you could only make changes that were precisely the same length in the revised data as that changed.

A far better solution would be to make the changes in memory (which you have to do, anyway, to make anything happen) and then if you really, really, must save these data every iteration, use save to save the array in a more efficient manner than text. If you then must look at it at some point in text, then you can convert but there's really no need otherwise.

[Joseph Cheng]

got an example of what the file is. and how you are save it? can you move it to the end of the file? like where it would be in the middle say see appendix A of saved jacobian matrices?

[ND]

Actually, the file is a fortran file I can open it in Matlab ,however Ineed to do some changes in this file. I use it in Abaqus software as that file contains a constitutive model. I need to change just the jacobian matrix (which is calculated in Matlab) at every iteration as at each once the jacobian would be changed (different jacobian matrix) and run abaqus again with this loop.

[Walter Roberson]

Read the file into memory and divide it into three parts: the part before the jacobians, the part that is the jacobians, and the part after the jacobians. Store the first and last parts in memory including any line terminators.

Then when you need to produce a new version, fopen() an output file for writing, fwrite() the part before where the jacobians are to go, then output the jacobians in an appropriate format, then fwrite() the part after the jacobians, fclose() the file. This doesn't try to "edit" the file in-place, it just creates a new file each time in a relatively efficient way. fwrite() of a long array is about the second most efficient file writing operation there is (the most efficient involves binary files and byte-for-byte replacements of data.)

[ND]

Dear Walter, thank you for your comment, please you mean that I will not use just one file (original one) with updating it ? I will have new file at every iteration? Best regards,

[dpb]

Your choice, depending on how you code it. There's no reason it would seem in this scenario to not reuse the same file.

Slightly different tack on Walter's idea would be to create the two sections of the file as he mentions, but then only rewrite the new section each time and use the OS to copy w/ concatenation the three sections into the one. This could possibly be slightly faster than the three separate fwrite operations.

Presuming you have built these as, say, partA and partC and have those saved, then

fid=fopen('partB','rt');  % open the middle part
fwrite(fid,fmt,array)  % w/ appropriate format
fid=fclose(fid);       % close partB
!'copy partA+partB+partC yourFile.dat' % concatenate

[Walter Roberson]

When you are using one file as a template to create a new version, it is a poor idea to overwrite the original file. Suppose something goes wrong during the process? A bug in the code? A power glitch? Your cat knocks the drive cable so the write fails?

Therefore, you should do one of the following:

• create a new file each time, preserving the previous ones, leaving it for the user to clean them up as desired
• create a new file, rename the old file to ".bak" (or similar), rename the new file to the old name; in this way if the new file creation fails, there is always the .bak to go back to
• create a new file, rename it to the old name once it is fully created; keeping a backup like the above would be better, but at the very least, do not put the new file into the place of the old file until the new file has been completely created and verified to have worked
• in cases where preserving file dates and permissions (and alternative resource forks) is important and those cannot be easily copied from one file to another, then create a new file, write the data to it, verify it is all correct, then take the contents and use them to overwrite the old file in-place (because overwriting preserves dates and access permissions and resource forks.) In this set-up, there is always a file with valid data contents in existence in case something goes wrong while overwriting the file. Note: you cannot reliably do this in MATLAB, because MATLAB does not allow you to truncate existing files if they should shrink due to the new content being smaller
• in the more general case, use a "transactional database" that is able to roll-back failed or incomplete transactions; or if your file-system supports transactions (for example the native Mac OS-X file system does), find a way to make the update take only a single transaction for file-system purposes so that you can recover if something goes wrong.

Those of us who used to do tape backups know the rule of three. Always use three generations of tapes for doing backups. The first of the tapes is the one being backed up to, and that is going to be the one that is corrupted when the system fails during backup and fouls itself needing restoration. The second of the tapes is the one you go to do the restore from to recover from the failure during the previous backup. And the reason why you need a third tape is that you forgot to put the write protect ring on the second tape and it promptly got clobbered by the tape drive management system (or by you accidentally entering the wrong command) when what you wanted to do was restore from it. The third tape is the one you remember to protect before starting the restore. And if your tape drive mangles that one too, it's your own fault for not having fixed the drive before you risked your last remaining copy.

[dpb]

As a rule, agree, Walter, wholeheartedly!!! (DAMHIKT, BTDT, etc., etc., etc., ...)

But, if OP does my suggestion he'll have the original safely tucked away and will simply reuse the pieces he created from it--and the new one will be modified every time step anyway. Now, certainly, one will want/need to debug the code but unless there's a need to regenerate this simulation again, it seems like the intermediary file here is a consumable, not an output.

[ND]

Basically, I thought that I can open the fortran file (.for) and read it line by line and when reach to jacobian delete the privious one and write the new one which is already calculated in Matlab in the previous step, then run Abaqus using this file and go on with this loop. Note that The location of matrix is constant during the process. Best regards,

[dpb]

You could, except--it's going to be slower than either of the above and there's no need to reread constant data every pass plus the major factor that when you write into a sequential file it is truncated from that point on -- hence Walter's suggestion of writing the three pieces or my alternative of only writing the changing part.

It would be possible if Matlab supported formatted direct-access record files, but it doesn't unless you "roll your own" which is what I alluded to in my original response of the harder way--namely, yes, if the file is a fixed format file you could determine the offset into the file of the location you're interested in, read the whole thing into memory as Walter says as a character array, and then do an internal formatted equivalent of Fortran internal WRITE (sprintf or the like) into that buffer at the correct point in memory and then write the modified file back out.

All in all, is it really worth that effort over the simpler way? Are you going to be doing this more than just a time or two so it would be worth the extra effort?

Alternatively, can your other package read anything other than a text file? A stream (so-called 'binary') file could be modified more readily knowing only the size of the arrays as the locations then are simply a direct multiple of the size (REAL or DOUBLE) times the array size.

[Walter Roberson]

It is not possible to "delete" lines in a text file. It is only possible to overwrite with new text that is at least as long as before (not shorter), or else to copy the lines before into a new file, write the new lines, and then copy the lines after from the old file to the new file.

When you are overwriting in-place, the limitation that the final file cannot get any shorter is due to a short-coming in MATLAB; operating systems do provide a way to say "truncate the file here" but MATLAB does not provide access to that.

The limitation that you cannot write a shorter chunk into the middle of a file and have the rest of the file "fall down" in relative position, is a limitation on all filesystems that store files as streams or as blocks. The last operating system that I know of that supported storing files as a list of variable-length records, was VAX VMS when using the RMS (Record Management System) I/O interface. The internal structure of those kinds of files did not fit in to the C/Unix model of a file as a consecutive stream of bytes, and eventually VMS RMS became obsolete even though it had some very good ideas.

There was a point at which MATLAB was supported on VMS, but my understanding is that MATLAB never provided support or interface to the VMS RMS services.

I don't know if there are any modern operating systems that support line-oriented files. It isn't impossible, I suppose: IBM's VM operating system supported datasets with variable-length records, and it would not entirely surprise me if there are still Big Data customers on mainframes who demand them.

[ND]

Many thanks for your comments (dpb and Walter), Please Walter what about create new file do you mean this process will be done manually every time I need to name the new file or I can do that automatically? The process should be done in Matlab and everything would be done automatically. For dpb question\ the another package needs just to read the text file (.for) at every iteration as the model in the text file will be changed depending on the new result getting from that package.

[dpb]

I understand you're reading the modified file; just wondered if it could understand some other form for the input besides a text file.

And, sure, you can create names dynamically, see the FAQ How_can_I_process_a_sequence_of_files? written with the idea specifically of processing existing files but the same ideas (other than dir) work for new file names, too.

[ND]

Please I understand that I cannot update a text file in place. I have to create a new file, but how can I code this (copy the lines before jacobian into a new file, write the new jacobian, and then copy the lines after jacobian from the old file to the new file? Best regards,

[dpb]

I posted a suggested sample code snippet earlier with only the dynamic name for the output file omitted. It uses the previous idea of saving the invariant parts of the file externally.

[Walter Roberson]

Use an editor to divide the file into partA.f (fixed part before), partB.f (the part that will be changing), partC.f (fixed part after.) It isn't worth doing it by program since you only need to do it once and you would need to examine the file carefully to determine exactly how to detect the boundaries by program. Since you are saving a copy of the file in this way, my discussion about generations of backups does not apply.

Then take the suggested code but with very minor modifications:

fid=fopen('partB.f','rt');  % open the middle part

fprintf(fid,fmt,array)  % w/ appropriate format

fid=fclose(fid);       % close partB
!'copy partA.f+partB.f+partC.f yourFile.f' % concatenate

but change the word yourFile.f to the exact file name that is expected by the other package, and change "array" to be the name of your variable that holds the computed jacobian, and change "fmt" to be a format specifier that formats the expression in a way that Fortran understands.

Creating a new file this way is the easy part. The harder part is in rewriting the expression for Fortran, as you might need to rewrite vector notation and you might need to do "word wrap", and whether you use .f or .F or .f77 or .F77 as your file extension can matter as to whether the file is interpreted as Fixed Format (72 column) or Free Format (I do not recall the maximum line length for that.)

[ND]

Dear Walter, What I have exactly in the middle of the fortran file is a jacobian which is calculated and modified in Matlab to be used in fortran file as this

      DDSDDE(1,1) = X1**(-5.0D-1)*X5*X6**2*2.10055D-2+9.371D-1
      DDSDDE(1,2) = X2**5.0D-1*X3**2*X4**5.0D-1*4.8591D-1
      DDSDDE(1,3) = X2**1.5D0*X3*X4**5.0D-1*6.4788D-1
      DDSDDE(1,4) = X2**1.5D0*X3**2*X4**(-5.0D-1)*1.6197D-1
      DDSDDE(1,5) = X1**5.0D-1*X6**2*4.2011D-2

At every run I will have new equations like in DDSDDE(1,1)= New ...etc during the analysis just the equations will be changed and the rest of the file will be constant. Now I need to read the file line by line then when reached jacobian write the new equations in a new file. In addition, the equations will not be fixed in length depends on the result may start with zero, then one line length of particular equation and during running reaches to two or three lines.

[dpb]

Again we repeat--you do NOT need to read the file every time unless the previous ground rule that the rest is invariant with the exception of this section, and in fact, it's a waste of resources to do so unless that is so. Just open one of them that you currently have in your favorite text editor and save the first and last parts as outlined.

I (and I presume Walter as well) was presuming the array to be written was numeric not a functional form but I suppose it does makes sense as input to have it as above.

Now it becomes a question of what actually is going to change; the functional form or only the numeric constants? The former is potentially fairly difficult depending on just what the form could be; the latter is pretty simple; you simply use fprintf with the expression as part of the format string and substitute in an appropriate numeric format string where the numbers that change are.

If I presume it's only the floating point numbers and not the exponents in the first since the first term is really SQRT() it probably is fixed, then

fmt1='DDSDDE(1,1) = X1**(-0.5D0)*X5*X6**2*%.3E+%.3E\n';

Then you use this with the constants you need/want something like

   fprintf(fid,fmt1,coeff(1),coeff(2))

where coeff are the values you've decided you need next, from whatever method you have to generate those. Clearly, you can use whatever array features of Matlab that are convenient, I simply showed the two elements specifically to illustrate the idea clearly.

When you've done the appropriate modifications for all sections of PartB as above, fclose it and follow with the concatenation as illustrated.

[dpb]

ADDENDUM NB: that Matlab doesn't have a way to write the 'D' Fortran double precision exponent indicator so the above will be single precision constants when read by Fortran, not doubles. If this is important, probably the easiest thing to do is to do the write as above first to a string variable via sprintf and then do a character substitution of 'D' for 'E' in that variable before writing that string to the PartB file.

[ND]

Sorry, the equations above got from Matlab, as you can convert a jacobian that calculated from matlab to be used in fortran by using fortran function in Matlab. Also, for what is changed ,I will have e.g. different equation for DDSDDE(1,1) and so on. in another word, the variables ,exponents and parameters would be changed depends on the results at every iteration.

[dpb]

Well, I don't know what "the Fortran function" is in Matlab, but surely you've seen enough pieces of how to write whatever it is you want to write expounded upon here.

Try some code and see where you then actually fail and with a specific example of that problem we can answer a specific question. I don't really follow what you're lacking at this point...

[Walter Roberson]

Using the fortran() call makes it really simple. Instead of fopen/fprintf/etc to write out the new file, just use

fortran(DDSDDE,'file','partB.f');

Here, the variable that stores your symbolic expression should be a symbolic matrix, and it should be the same name as what you want to appear in the file. If you use a regular MATLAB matrix of symbolic expressions then there is the possibility that the output variable name will be "t0" instead of the name you want.

Side note: when you use fortran() in this form, there is the bonus that it will generate optimized code, using temporary variables to avoid computing the same thing multiple times. That's good for computation purposes, but does tend to make the code more difficult to understand.