Combine text files with continues time stamp

Hi

The equipment that I am logging with is separating the logged data into 2 text files. One file for the first 4000 sec and another file for the next 3000 sec. I would prefer that the time stamp in the 2nd file would start where the 1st file timestamp stopped.

Is there a metod to merge the two files. I need a merge, beacuce I will combine it with .Fig file afterworth

1st file (total 2000009 lines, one line for each 2 milli seconds )

"COMMENT","8861-50 MEM DATA"
"DATE","02-06-2023"
"TIME","08:33:05.470"
"NUM_SIGS",22
"INTERVAL",2,00000E-003
"HORZ_UNITS","s"
"VERT_UNITS","s","KW","KW","KW","KW","KW","KW","KW","KW","KW","KW","KW","KW","KW","A","A","A","V","V","V","V","V"
"SIGNAL","TIME","M101 POWER [KW]","M102 POWER [KW]","M103 POWER [KW]","M104 POWER [KW]","M105 POWER [KW]","M106 POWER [KW]","M107 POWER [KW]","M108 POWER [KW]","M109 POWER [KW]","M110 POWER [KW]","M111 POWER [KW]","M112 POWER [KW]","M113 POWER [KW]","STRING 1 [A]","STRING 2 [A]","DC CABINET DC OUTPUT [A]","VFD AC OUTPUT [VAC]","DC CABINET DC OUTPUT [VDC]","VFD DC INPUT [VDC]","STRING 1 [VDC]","STRING 2 [VDC]"
"DATA"
 +0,000000000E+000,-1,35000E-001,-1,20000E-001,-8,80000E-002,-1,05000E-001,+1,54000E-001,+9,37500E-002,+0,00000E+000,+1,68750E-001,-1,68750E-001,-1,87500E-001,-1,50000E-001,-1,31250E-001,-3,75000E-002,+7,50000E-001,-1,00000E+000,+5,00000E-001,+5,00000E-001,+5,69000E+002,-2,00000E+000,+5,70000E+002,+5,49500E+002
 +2,000000000E-003,+3,75000E-001,-1,80000E-001,-1,76000E-001,-1,65000E-001,+8,80000E-002,-7,50000E-002,+0,00000E+000,+3,75000E-002,+0,00000E+000,-1,87500E-002,+1,87500E-002,+5,62500E-002,+1,50000E-001,-2,00000E+000,-5,00000E-001,+5,00000E-001,+5,00000E-001,+5,60500E+002,-2,00000E+000,+5,13500E+002,+5,49500E+002
 +4,000000000E-003,+1,20000E-001,-6,00000E-002,+1,32000E-001,+1,50000E-002,-2,42000E-001,-1,50000E-001,-1,87500E-002,-1,50000E-001,+1,68750E-001,+1,50000E-001,+1,68750E-001,+1,31250E-001,+7,50000E-002,-2,00000E+000,-7,50000E-001,+5,00000E-001,+5,00000E-001,+5,62000E+002,-2,00000E+000,+5,14000E+002,+5,49000E+002
 +6,000000000E-003,-3,00000E-001,+1,80000E-001,+1,54000E-001,+1,80000E-001,-1,10000E-001,+0,00000E+000,+1,87500E-002,+0,00000E+000,+0,00000E+000,+0,00000E+000,+0,00000E+000,-3,75000E-002,-9,37500E-002,+3,50000E+000,-5,00000E-001,+7,50000E-001,+5,00000E-001,+5,59000E+002,-2,00000E+000,+5,69500E+002,+5,49000E+002
 +8,000000000E-003,-2,10000E-001,+4,50000E-002,-4,40000E-002,-4,50000E-002,+1,10000E-001,+1,50000E-001,+0,00000E+000,+1,50000E-001,-1,68750E

2nd file (total 691432 lines, one line for each 2 milli seconds )

"COMMENT","8861-50 MEM DATA"
"DATE","02-06-2023"
"TIME","09:39:45.470"
"NUM_SIGS",22
"INTERVAL",2,00000E-003
"HORZ_UNITS","s"
"VERT_UNITS","s","KW","KW","KW","KW","KW","KW","KW","KW","KW","KW","KW","KW","KW","A","A","A","V","V","V","V","V"
"SIGNAL","TIME","M101 POWER [KW]","M102 POWER [KW]","M103 POWER [KW]","M104 POWER [KW]","M105 POWER [KW]","M106 POWER [KW]","M107 POWER [KW]","M108 POWER [KW]","M109 POWER [KW]","M110 POWER [KW]","M111 POWER [KW]","M112 POWER [KW]","M113 POWER [KW]","STRING 1 [A]","STRING 2 [A]","DC CABINET DC OUTPUT [A]","VFD AC OUTPUT [VAC]","DC CABINET DC OUTPUT [VDC]","VFD DC INPUT [VDC]","STRING 1 [VDC]","STRING 2 [VDC]"
"DATA"
 +0,000000000E+000,+6,84000E+000,+5,41500E+000,+6,46800E+000,+5,61000E+000,+6,33600E+000,+5,60625E+000,+6,78750E+000,+6,22500E+000,-1,20000E+000,-1,06875E+000,-1,31250E+000,-1,40625E+000,-1,16250E+000,+5,95000E+001,+7,15000E+001,+1,26750E+002,-3,91500E+002,+4,00500E+002,+3,93000E+002,+4,02500E+002,+4,02000E+002
 +2,000000000E-003,+6,18000E+000,+5,49000E+000,+6,62200E+000,+5,61000E+000,+6,46800E+000,+5,51250E+000,+7,03125E+000,+6,13125E+000,-1,08750E+000,-1,21875E+000,-1,35000E+000,-1,23750E+000,-1,10625E+000,+5,97500E+001,+7,17500E+001,+1,27000E+002,-3,91000E+002,+4,01000E+002,+3,93000E+002,+4,02000E+002,+4,02000E+002
 +4,000000000E-003,+6,49500E+000,+5,43000E+000,+6,73200E+000,+5,37000E+000,+6,66600E+000,+5,73750E+000,+6,75000E+000,+6,28125E+000,-1,27500E+000,-1,40625E+000,-1,16250E+000,-1,06875E+000,-1,25625E+000,+5,97500E+001,+7,17500E+001,+1,26750E+002,-3,91000E+002,+4,00500E+002,+3,93500E+002,+4,02000E+002,+4,02000E+002
 +6,000000000E-003,+6,88500E+000,+5,44500E+000,+6,57800E+000,+5,49000E+000,+6,79800E+000,+5,73750E+000,+6,50625E+000,+6,35625E+000,-1,36875E+000,-1,25625E+000,-1

[Image Analyst]

Make it easy for people to help you, not hard. Attach the two text files with the paperclip icon.

[Steen]

The files are to big to be attached here. Limit for upload are 5MB. My files are 600 and 212MB.

I have posted a snip of each file.

[Jan]

@Steen: I'm not sure what this means: "I would prefer that the time stamp in the 2nd file would start where the 1st file timestamp stopped." Numerically? In the text file? What exactly do you call "time stamp"?

[Steen]

The first colomn is the timestamp. You can see in boths file that the timestamp begins with +0,000000000E+000 and that is to be added to the start time.

Starttime in the 1st file is "TIME","08:33:05.470" and in the 2nd file the start time is "TIME","09:39:45.470". I would have prefered that the first timestamp in 2nd file would be +4,000000000E+003 since the last timestamp in 1st file is +3,999998000E+003

So I am just asking for a metode to arrange this data from 1st and 2nd file into consecutive numbers

Answer 1

[dpb]

Can you clean up the malformatted file format first? It uses "," for both the field delimiter as well as the decimal place indicator. That makes parsing it more difficult than otherwise...

I had written a start and then got somewhat bogged down and didn't have the energy to fix the problem inherent in the bad files.

What's the output expected to be -- you want all the data sequentially in one file, just with the wraparound of the sample time fixed? Or, do you want to keep the header info in the second file and just fix it?

Frankly, there's really no reason to fix it in the file(s); when you read the data for processing and can manage to get the input format fixed where it will convert to the numeric values correctly, then when you read the second file, just add the ending time of the first to the numeric values of the second plus one delta-t.

Or, since it's uniform sample rate, you can compute the time completely independently of the input files; just read the two files in, concatenate them and then fill in the time colum from 0:end at the constant dt.

[Steen]

I would like to go for this solution.

"Frankly, there's really no reason to fix it in the file(s); when you read the data for processing and can manage to get the input format fixed where it will convert to the numeric values correctly, then when you read the second file, just add the ending time of the first to the numeric values of the second plus one delta-t"

Can you tell me how the script should be written to do the above?

[dpb]

Can you first fix the files to not use the comma for both the delimiter and then the decimal point as well?

[dpb]

opt=fixedWidthImportOptions('DataLines',10, ...
                            'NumVariables',22, ...
                            'VariableWidths',[18 repmat(14,1,21)], ...
                            'VariableTypes',repmat("string",1,22));
B=readmatrix('B.csv',opt);
B=strrep(B,',+','+'); B=strrep(B,',-','-'); B=strrep(B,',','.');
B=str2double(B);

finally gets the numeric data as numeric; the malformed files are going to be a problem if you can't fix the generating application to do something other than writing a delimited file with the same delimiter as your locale setting uses as the decimal indicator.