Combine writematrix with a text header and safe as file

Hi guys,

I am stuck with this problem since many hours. I want have multible *.IV0 files (with headers). I attached one example AS A *.TXT! It was npt possible to upload *.IV0 files. So please rename it when you try to solve this problem. The fact that it is a *.IV0 file changes many things. Since with this filetype it is not possible to use some matlab codes.

I try to average all files in a specifc folder excluding the header (first 27 rows). This works great so far. Afterwards I wanted to save the averaged data by using writematrix. This works too by using following code:

folder   = 'C:\Users\--__aver\';
IV0Files = dir(fullfile(folder, '*.IV0'));  % Use absolute path names
numfiles = length(IV0Files);
data_sum = 0;
rows = [1,2,3,4];
for ci = 1:numfiles
    data_file = readmatrix(IV0Files(ci).name,"NumHeaderLines",27,"FileType","text");
    data_sum = data_sum+data_file(:,rows); % summation accross the files (removed 3rd column (time))
end
% divide by numfiles to have the average (and not the sum)

average = data_sum/numfiles;
averagecell=num2cell(average);
%this is to save the data in the new IV0 file
writematrix(average, fullfile(folder,'otherdirectionSK-ITOLspray16.IV0'),"Delimiter","tab", "FileType", "text");

But now comes the heavy part. I tried to add a header to my output file. The header should be the same as in one of the files I average. So basically the first 27 rows of the attached file. I tried soo much and I can´t figure out a way to do it. Since it is a *.IV0 file things get even more complicated. I have tried things using:

opts (does not work for *.IV0)
fgets (in a for loop (same as code above), got stuck, I think because of the tab delimiter maybe...)

[dpb]

The high-level writeXXX routines don't have the ability to be able to append data to or to be used to append data to existing text files, so that's a non-starter.

Probably the easiest way in coding would be to hold the header lines in a temporary file and then just use the OS copy command to concatenate the two.

There's no reason I can think of why the fgets/fprintf combination wouldn't work; show the code for it as you tried it...note, however, that you can't combine it with writematrix; you'll have to use all low-level i/o functions.

I think the restriction of not having an append option is a real limitation in the general usefulness of the new functions.

[Simon Keegan]

Hi dpb,

Thanks for the quick answer. I tried to store the data in a temporary file to, but i did not find a way to bring the two files (header + matrix) back together... Maybe you have a good code for that.

This was the code I tried:

fileID = fopen(IV0Files(ci).name,'r');
formatSpec = '%c';
pease = {fscanf(fileID,formatSpec)};
fclose(fileID);

This works quit well, since the output 'pease' is a 1*1cell and looks like this

'#Filename	Simon_R
#Sample	1
#Cell	1
#Pixel	R
#group	2
#description	SK-Cspray
#ActiveArea	0.133	cm2
#LI	0	mW/cm2
#Type	darkIV
#Geometry	inverted
#IlluminationDirection	front
#ScanDirection	backward
#Delay	0.01	s
#Integrationrate	1
#Repetition	1
#LightSoaking	0	s
#VoltageStabilization	5	s
#Atmosphere	N2
#Date	2020-12-04
#Time	12:35:31	s
#Filepath	D:\data\IV\2020-12-04\
#JSC	0.012728	mA/cm2
#Rsh	3.028	Ohm/cm2
#Rs	0.001733	Ohm/cm2
	
V	I	J	Time
V	mA	mA/cm2	s
-0.5	2.1549	16.2022	1546.35	
-0.49	2.11718	15.9187	1546.42	
-0.48	2.07964	15.6364	1546.49	
-0.47	2.04002	15.3385	1546.56	
-0.46	2.00048	15.0412	1546.63	
-0.45	1.96221	14.7535	1546.7	
-0.44	1.92393	14.4656	1546.78	
-0.43	1.88449	14.1691	1546.84	
-0.42	1.84523	13.8739	1546.92	
-0.41	1.80576	13.5772	1546.98	
-0.4	1.76632	13.2806	1547.06	
-0.39	1.72795	12.9921	1547.13	
-0.38	1.68799	12.6916	1547.19	
-0.37	1.64906	12.399	1547.27	
-0.36	1.60852	12.0941	1547.34	
-0.35	1.56847	11.793	1547.41	
-0.34	1.52787	11.4877	1547.48	
-0.33	1.48748	11.1841	1547.55	
-0.32	1.44739	10.8826	1547.63	
-0.31	1.40835	10.5891	1547.7	
-0.3	1.36908	10.2939	1547.77	
-0.29	1.32607	9.97044	1547.85	
-0.28	1.28513	9.66262	1547.91	
-0.27	1.246	9.36846	1547.98	
-0.26	1.20684	9.07398	1548.06	
-0.25	1.16507	8.75991	1548.13	
-0.24	1.12414	8.45222	1548.2	
-0.23	1.08273	8.14084	1548.28	
-0.22	1.04684	7.87098	1548.35	
-0.21	1.00523	7.55814	1548.4	
-0.2	0.965133	7.25664	1548.48	
-0.19	0.921994	6.93229	1548.55	
-0.18	0.879875	6.6156	1548.62	
-0.17	0.836516	6.28959	1548.7	
-0.16	0.795471	5.98098	1548.77	
-0.15	0.751063	5.64709	1548.84	
-0.14	0.706719	5.31367	1548.91	
-0.13	0.663744	4.99056	1548.99	
-0.12	0.618529	4.65059	1549.05	
-0.11	0.572994	4.30822	1549.12	
-0.1	0.52684	3.96121	1549.2	
-0.09	0.479441	3.60482	1549.27	
-0.08	0.431241	3.24242	1549.34	
-0.07	0.383092	2.88039	1549.41	
-0.06	0.332879	2.50285	1549.49	
-0.05	0.281743	2.11837	1549.56	
-0.04	0.229072	1.72234	1549.63	
-0.03	0.175464	1.31928	1549.71	
-0.02	0.119318	0.897129	1549.78	
-0.01	0.0621344	0.467176	1549.85	
0	0.00169287	0.0127283	1549.92	
0.01	-0.0630867	-0.474336	1549.99	
0.02	-0.131665	-0.989966	1550.06	
0.03	-0.204532	-1.53784	1550.14	
0.04	-0.280579	-2.10962	1550.21	
0.05	-0.361421	-2.71745	1550.28	
0.06	-0.447311	-3.36324	1550.36	
0.07	-0.538849	-4.0515	1550.43	
0.08	-0.63558	-4.7788	1550.5	
0.09	-0.740072	-5.56445	1550.57	
0.1	-0.850548	-6.3951	1550.65	
0.11	-0.968678	-7.28329	1550.72	
0.12	-1.10875	-8.33647	1550.79	
0.13	-1.24648	-9.37202	1550.87	
0.14	-1.39516	-10.49	1550.94	
0.15	-1.56075	-11.735	1551.01	
0.16	-1.73215	-13.0236	1551.09	
0.17	-1.91328	-14.3856	1551.23	
0.18	-2.10811	-15.8505	1551.31	
0.19	-2.31273	-17.389	1551.38	
0.2	-2.52926	-19.017	1551.45	
0.21	-2.75875	-20.7425	1551.53	
0.22	-2.99947	-22.5524	1551.6	
0.23	-3.25212	-24.452	1551.66	
0.24	-3.52486	-26.5027	1551.73	
0.25	-3.80248	-28.5901	1551.81	
0.26	-4.10054	-30.8311	1551.88	
0.27	-4.40517	-33.1216	1551.95	
0.28	-4.73676	-35.6147	1552.03	
0.29	-5.06977	-38.1186	1552.1	
0.3	-5.4176	-40.7338	1552.17	
0.31	-5.77922	-43.4528	1552.24	
0.32	-6.14209	-46.1812	1552.32	
0.33	-6.52297	-49.0449	1552.39	
0.34	-6.91095	-51.962	1552.46	
0.35	-7.32006	-55.038	1552.54	
0.36	-7.734	-58.1504	1552.61	
0.37	-8.16401	-61.3835	1552.68	
0.38	-8.60084	-64.6679	1552.75	
0.39	-9.0378	-67.9534	1552.83	
0.4	-9.52041	-71.582	1552.9	
0.41	-9.99943	-75.1837	1552.97	
0.42	-10.4854	-78.8375	1553.04	
0.43	-10.9657	-82.4486	1553.12	
0.44	-11.4615	-86.1769	1553.19	
0.45	-11.9704	-90.0033	1553.26	
0.46	-12.4891	-93.9031	1553.34	
0.47	-13.024	-97.9251	1553.41	
0.48	-13.555	-101.918	1553.47	
0.49	-14.1044	-106.048	1553.54	
0.5	-14.6541	-110.182	1553.62	
0.51	-15.2197	-114.434	1553.69	
0.52	-15.7724	-118.589	1553.76	
0.53	-16.3615	-123.019	1553.84	
0.54	-16.945	-127.406	1553.91	
0.55	-17.5292	-131.799	1553.98	
0.56	-18.1306	-136.32	1554.06	
0.57	-18.7288	-140.818	1554.13	
0.58	-19.3304	-145.342	1554.2	
0.59	-20.0293	-150.596	1554.27	
0.6	-20.6564	-155.312	1554.35	
0.61	-21.2736	-159.952	1554.4	
0.62	-21.9023	-164.679	1554.48	
0.63	-22.5602	-169.626	1554.55	
0.64	-23.1887	-174.351	1554.7	
0.65	-23.8945	-179.658	1554.77	
0.66	-24.5594	-184.657	1554.85	
0.67	-25.2029	-189.496	1554.92	
0.68	-25.8751	-194.55	1554.99	
0.69	-26.5599	-199.699	1555.06	
0.7	-27.2318	-204.75	1555.14	
0.71	-27.9294	-209.995	1555.21	
0.72	-28.6116	-215.125	1555.28	
0.73	-29.3129	-220.398	1555.34	
0.74	-30.015	-225.677	1555.49	
0.75	-30.7045	-230.861	1555.64	
0.76	-31.423	-236.264	1555.87	
0.77	-32.1582	-241.791	1556.1	
0.78	-32.879	-247.211	1556.17	
0.79	-33.5899	-252.556	1556.23	
0.8	-34.319	-258.038	1556.31	
0.81	-35.0479	-263.518	1556.38	
0.82	-35.7709	-268.954	1556.45	
0.83	-36.4986	-274.426	1556.53	
0.84	-37.2321	-279.94	1556.6	
0.85	-37.9626	-285.433	1556.67	
0.86	-38.7156	-291.095	1556.74	
0.87	-39.4725	-296.786	1556.82	
0.88	-40.2235	-302.432	1556.89	
0.89	-40.9935	-308.222	1556.96	
0.9	-41.7451	-313.873	1557.04	
0.91	-42.5085	-319.613	1557.11	
0.92	-43.2488	-325.179	1557.18	
0.93	-44.0205	-330.981	1557.25	
0.94	-44.8033	-336.867	1557.32	
0.95	-45.542	-342.421	1557.4	
0.96	-46.3169	-348.247	1557.47	
0.97	-47.083	-354.008	1557.54	
0.98	-47.8437	-359.727	1557.77	
0.99	-48.64	-365.714	1557.84	
1	-49.4225	-371.598	1557.91	
1.01	-50.194	-377.399	1557.98	
1.02	-50.9628	-383.179	1558.06	
1.03	-51.7674	-389.228	1558.13	
1.04	-52.5371	-395.016	1558.2	
1.05	-53.344	-401.083	1558.27	
1.06	-54.1276	-406.975	1558.35	
1.07	-54.9069	-412.834	1558.42	
1.08	-55.6912	-418.731	1558.49	
1.09	-56.4796	-424.658	1558.57	
1.1	-57.2606	-430.531	1558.64	
1.11	-58.0643	-436.574	1558.71	
1.12	-58.8262	-442.302	1558.78	
1.13	-59.63	-448.346	1558.86	
1.14	-60.408	-454.195	1558.93	
1.15	-61.1747	-459.96	1558.99	
1.16	-61.9803	-466.018	1559.06	
1.17	-62.7198	-471.578	1559.14	
1.18	-63.4747	-477.253	1559.21	
1.19	-64.2381	-482.994	1559.28	
1.2	-65.0449	-489.059	1559.36	
1.21	-65.8053	-494.777	1559.43	
1.22	-66.5218	-500.164	1559.5	
1.23	-67.2503	-505.642	1559.58	
1.24	-67.9354	-510.792	1559.65	
1.25	-68.6096	-515.862	1559.72	
'

But I don´t know how I get just the header and not the whole thing and I don´t know how I put this thing above the 'writematrix' code.

[Simon Keegan]

Sorry for the long code field...

I did change the program so many times that I couldn´t restore my version with 'fgets/fprintf'...Sorry for that too.

Answer 1

[Mathieu NOE]

hello simon

welcome back and happy new year !

I somehow recognize this code.... and now comes the upgraded version that appends the data to the headers

try this ( after adjusting the folder path) :

folder   = cd;
% folder   = 'C:\Users\--__aver\';

IV0Files = dir(fullfile(folder, '*.IV0'));  % Use absolute path names
numfiles = length(IV0Files);
data_sum = 0;
rows = [1,2,3,4];
for ci = 1:numfiles
    data_file = readmatrix(IV0Files(ci).name,"NumHeaderLines",27,"FileType","text");
    data_sum = data_sum+data_file(:,rows); % summation accross the files (removed 3rd column (time))
end
% divide by numfiles to have the average (and not the sum)
average = data_sum/numfiles;
% averagecell=num2cell(average); % not needed as we use writematrix 

% retrieve headers line (from first file)
lines= readlines(IV0Files(1).name);
header_lines = convertStringsToChars(lines(1:27,:));

%this is to save the data in the new IV0 file
writecell(header_lines, fullfile([folder '\out'],'otherdirectionSK-ITOLspray16.IV0'),"Delimiter","tab", "FileType", "text");
writematrix(average, fullfile([folder '\out'],'otherdirectionSK-ITOLspray16.IV0'),"Delimiter","tab", "FileType", "text","WriteMode","append");

[Simon Keegan]

And here we have the genius again :) Hello. Hope you had a great start so far.

Yeah you probably recognize the code, because I have already asked a question and you answered it. The code from there has not been changed that much. I thought I am done with this topic, but it seems I have to dig a litle bit deeper.

Your code is working again. Thanks a million again. PERFECT. Nearly perfect. Sorry.

The file in the end has now " at the begging and the ending. See:

"#Filename Simon_L"
"#Sample	1"
"#Cell	3"
"#Pixel	L"
"#group	13"
"#description	SK-Cspray"
"#ActiveArea	0.133	cm2"
"#LI	0	mW/cm2"
"#Type	darkIV"
"#Geometry	inverted"
"#IlluminationDirection	front"
"#ScanDirection	backward"
"#Delay	0.01	s"
"#Integrationrate	1"
"#Repetition	1"
"#LightSoaking	0	s"
"#VoltageStabilization	5	s"
"#Atmosphere	N2"
"#Date	2020-12-04"
"#Time	12:25:40	s"
"#Filepath	D:\data\IV\2020-12-04\"
"#JSC	0.0096779	mA/cm2"
"#Rsh	7.7869	Ohm/cm2"
"#Rs	0.0016047	Ohm/cm2"
"	"
"V	I	J	Time"
"V	mA	mA/cm2	s"
-0.5	4.55157925	34.2224233333333	1264.53466666667
-0.49	4.46741891666667	33.5895833333333	1264.6055

Do you know how I can delete those "?

[Simon Keegan]

There are so many ways to read lines from files, but I did not thnink about the easiest one 'readlines'...

And I would have never tried this 'convertStringsToChars' and then append those two files like you did. It looks like a really good idea. I have to remember this one.

[Mathieu NOE]

yeah, I find those extra " a bit annoying

I tried two versions of my code ,

first attempt was :

second is the one I sent you with

header_lines = convertStringsToChars(lines(1:27,:));
writecell(header_lines,....

but both gives the same unpleasant output with added "

maybe Walter (the genius of all geniuses in the entire universe - sorry for the others guys) as the trick for that

i'll be glad to learn that one too !

[Mathieu NOE]

sorry , sometimes my pc will not react on time

so my first attempt was :

header_lines = lines(1:27,:);
writematrix(header_lines,....

[Mathieu NOE]

and both results are the same

looking into writematrix and writecell was not successfull to get rid of those "

doc says only if original strings contain double quote , this can be converted optionnally to two single quote , but I don't think it will do any good there.

[Simon Keegan]

Ah ok. Thanks anyway. I will cut of the " in a really ugly way at the end of my process. Later on I need some data of the header.

Lets wait for the answer of the boss Walter.

[Mathieu NOE]

VICTORY

I was wrong and this works :

replace last lines with :

%this is to save the data in the new IV0 file
writecell(header_lines, fullfile([folder '\out'],'otherdirectionSK-ITOLspray16.IV0'),"Delimiter","tab", "FileType", "text","QuoteStrings",0);
writematrix(average, fullfile([folder '\out'],'otherdirectionSK-ITOLspray16.IV0'),"Delimiter","tab", "FileType", "text","WriteMode","append");

[Simon Keegan]

I knew you are the genius. It works. Thanks a million.

[Mathieu NOE]

you're welcome !!

pretty happy to start the new year on a good tempo ! at least that worked today !!

Answer 2

[Walter Roberson]

https://www.mathworks.com/help/matlab/ref/writematrix.html#namevaluepairarguments

'writemode', 'append'

[dpb]

Interesting new addition, Walter.

Which release introduced it? R2019b lacks it which is what have here...

[Alan Hoskins]

Yeah, 2019b lacks this functionality. I guess I need to upgrade.