Write matrix/table to text files without changing checksum
Show older comments
I am implementing a data processing workflow in matlab that will generate text/csv data files of the following structure:
header info (experiment time, description, annotations etc.)
=================
MD5 checksum of the data matrix
=================
actual data matrix in comma delimited format
I cannot do checksum of the entire file as people may go in and edit the annotations in text editors.
Now the problem with the writematrix and writetable function is apparently loss of precision when writing to text files. This is also noted in the help page of both functions. which I think it's a wired limitation, but it is what it is.
So my requirement is to append a matrix to a text file, exactly, and can be retrived unaltered. I assume some low level function like the fprintf can do this, but I am not familar with its syntax.
5 Comments
Walter Roberson
on 9 Nov 2026 at 18:04
Edited: Walter Roberson
on 9 Nov 2026 at 18:05
What does it mean to take an MD5 checksum of the data matrix?
- Is the data being internally considered as being double precision and the MD5 is being taken over the big-enden IEEE-754 representation of the data?
- Is the data being converted to text form and the MD5 of the text form being taken? If so then it would be sensitive to leading and trailing zeros and sensitive to whether the data was represented in floating point or fixed point. Does the MD5 take into account whether the line terminators are LF alone or are CR+LF ?
Stephen23
on 9 Nov 2026 at 18:08
"So my requirement is to append a matrix to a text file, exactly, and can be retrived unaltered."
This is not possible without some encryption: anyone altering the data matrix can also alter the checksum.
Yixiao
on 9 Nov 2026 at 18:18
Yixiao
on 9 Nov 2026 at 18:22
dpb
on 9 Nov 2026 at 23:30
"...alternation, either intentional or by accident, can be detected, hench the checksum."
But unless the checksum is somehow known externally, there's no way to prevent the person who edited the file from recomputing a checksum to match. You have no way to be able to know at a later time what the checksum originally was unless there's an independent database somewhere that isn't able to be accessed.
That person would, of course, have to know how the checksum was generated and be able to do so, but there's no way to be certain.
There is no way to write a sequential text file other than sequentially unless the length of the data fields is fixed; if the file is allowed to be edited manually, nothing prevents there being more or fewer records inserted/deleted as well as simply the length of comments, etc., being changed.
One would have to sequentially read each record from the file until finding the section divider (presuming that is what the rows of "=" signs are intended to be), but there would be no way to even preclude those being either added or deleted, either.
Answers (2)
If you need to be absolutely certain that the data is represented exactly:
writematrix(compose("%.751g", TheMatrix), FILENAME, 'WriteMode', 'append')
If you just need to write enough digits that all values will be exactly recreated in binary even if they are truncated in decimal then
writematrix(compose("%.17g", TheMatrix), FILENAME, 'WriteMode', 'append')
For example,
format long g
B = -0.10224244608549089285975952634544228203594684600830078125
double(compose("%.14g", B)) - B
double(compose("%.15g", B)) - B
double(compose("%.16g", B)) - B
double(compose("%.17g", B)) - B
double(compose("%.18g", B)) - B
double(compose("%.19g", B)) - B
compose("%.17g", B)
Although the decimal expansion with 17 digits is truncated compared to the exact decimal expansion, 17 digits is enough for the binary value to be exactly reproduced when the decimal is read back in, but 16 digits is not enough for exact reproduction.
3 Comments
writematrix(compose("%.17g", TheMatrix), FILENAME, 'WriteMode', 'append')
can be written also as
fmt=[repmat('%.17g,',1,width(TheMatrix)-1) '%.17g\n']; % format by array width
fid=fopen(FILENAME,'a+'); % open the file for append mode
fprintf(fid,fmt,TheMatrix.'); % write in row-major order w/ transpose
fid=fclose(fid); % close the file
I don't know how much faster it might be using the straight i/o library runtime to do the formatting translation over the in-memory conversion to the string(+), but it will save a whole bunch of memory if the array is of any size since the string is 19*2/8 bytes larger for each element of the array plus the inherent overhead of the string array class.
Note carefully that MUST transpose the array in order to write it out by row rather than by column since MATLAB stores arrays in column major order internally.
(+) I didn't try timing actual writing to file to avoid creating a huge file on the disk, but comparing sprintf(fmt,X.') to compose('%.17g',X) for X=rand(100); was about a 3X penalty for compose() on local machine.
ADDENDUM
For convenience, one could wrap the above in a little utiity function to hide the details from the higher level code.
function appendMatrix(FILENAME,TheMatrix)
% append a numeric 2D array to existing file as csv
fmt=[repmat('%.17g,',1,width(TheMatrix)-1) '%.17g\n']; % format by array width
fid=fopen(FILENAME,'a+'); % open the file for append mode
fprintf(fid,fmt,TheMatrix.'); % write in row-major order w/ transpose
fid=fclose(fid); % close the file
end
For robustness, error handling should be included, of course and an enhancement would be to add an (perhaps optional) flag variable for the write permissions so could either write new file or append to existing.
Steven Lord
on 10 Nov 2026 at 15:56
Nowhere does the code in your comment use the fmt variable, @dpb. I also think you're missing a repmat call on that line.
Good catch @Steven Lord, I pasted wrong pieces-parts and didn't check carefully enough. Patched...==dpb
"...missing a repmat..."
Why oh why couldn't have figured out a way to keep Fortran FORMAT to be able to write repeat specifiers??? At one time when still 32-bit so supported old DVF/CVF compiler I had built a mex function that would take a FORMAT descriptor and array and then use the Fortran runtime. Alas, I lost the source in an employment move and then when there were no compatible compilers that I had access to, just slipped away. W/ gfortran now and ISO_C_BINDING, mex is again beginning to look attractive although there's a lot needing done to build modern interface modules to get out of the F77 dark ages.
A fiew pieces put together could be turned into something that ought to serve the stated purpose with something else to do the reading. Undoubtedly more sophistication would be wanted, but the base idea of how to accomplish the orignially stated question.
Note the following will not distinguish shape --the checksum is purely content -- a 4x3 or 6x2 arrangement of the same values will generate the same checksum so if that were to be important would have to do something like encrypt the row,col dimensions into the checksum byte stream or otherwise distinguish. Disclaimer: I asked an AI engine for how to compute the checksum, that's the extent of my knowledge there.
file = 'matrix_data.csv';
x=rand(4,3);
writeMatrixCSV(x,file)
type(file)
function writeMatrixCSV(matrix,filename)
% create a CSV file with header, checksum and CSV data
header=makeMatrixHeader(matrix); % construct with the checksum
writematrix(header,filename,'QuoteStrings','none') % go ahead and write it to new file
appendMatrix(matrix,filename) % add the CSV formatted data
end
function header=makeMatrixHeader(matrix)
checksum=MD5checksum(matrix);
header=["info (experiment time, description, annotations etc.)";
"=================";
string(checksum);
"=================";
];
end
function checksum=MD5checksum(matrix)
mdEngine = java.security.MessageDigest.getInstance('MD5');
mdEngine.update(typecast(matrix(:),'uint8'));
checksum=sprintf('%02x',typecast(mdEngine.digest(),'uint8'));
end
function appendMatrix(TheMatrix,FILENAME)
% append a numeric 2D array to existing file as csv
fmt=[repmat('%.17g,',1,width(TheMatrix)-1) '%.17g\n']; % format by array width
fid=fopen(FILENAME,'a+'); % open the file for append mode
fprintf(fid,fmt,TheMatrix.'); % write in row-major order w/ transpose
fid=fclose(fid); % close the file
end
NOTA BENE: reversed to put the array before the file specification to match the MATLAB writeXXX syntax pattern from the prior illustration of appending.
Categories
Find more on Text Files in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!