How do you import LABVIEW time stamps into MATLAB

Bug in h5info.

Command

 gr_inf=h5info(f_name, '/wfm_group0/axes/axis0'    );

Produces an error:

Warning: There is no MATLAB integer type that corresponds to an HDF5 integer type with 16 bytes.  The
data will be treated as an uninterpreted uint8 array. 
> In h5info (line 74)
  In read_scope_hws (line 17)

The file that is being read was created by the NI oscilloscope program.

Not surprisingly, a later statement

 ref_time1=gr_inf.Attributes(9).Value;

produces an array of 16 two-byte int8 numbers instead of a 64 byte integer.

The problem is not due to not understanding which group or which attribute to read; the information in Attributes(9).Value is a time stamp. It is just that MATLAB and LABVIEW creators are not talking to each other. The file in question contains four time stamps; each produces a warning.

Does anybody know how to either:

1. Fix the bug in h5info.m

2. How to covert the received 64 bytes in a time stamp by hand

[dpb]

2. How to co[n]vert the received 64 bytes in a time stamp by hand

Would seem NI would have the definition and probably some sample code?

h5info calls h5infoc which is a private mex function so there's not much of any way to patch it w/o TMW cooperation.

OTOH, there _are- both signed and unsigned 64-bit integers supported since at least R14 so seems like should be able to do the conversion from the array manually and then write a conversion function as a workaround.

I didn't search; mayhaps somebody else already has on File Exchange or elsewhere...

ERRATUM

_"...both signed and unsigned 64-bit integers supported..."

My bad, overlooked it's a 128-bit integer, not 64, sorry.

OK, from the NI site --

"The most significant 64 bits should be interpreted as a 64-bit signed two's complement integer. It represents the number of whole seconds after the Epoch 01/01/1904 00:00:00.00 UTC."

So, can you get access to the array as bytes? How about an example to see what actually do have?

"The least significant 64 bits should be interpreted as a 64-bit unsigned integer. It represents the number of 2-64 seconds after the whole seconds specified in the most significant 64-bits. Each tick of this integer represents 0.05421010862427522170... attoseconds."

Seems like if can get the full array as bytes then int64(ary(1:8)) will give the seconds that can be converted to Matlab datenum by the offset of epoch starting times + uint64(ary(9:16)) for the fractional seconds of the day (albeit with some loss of precision if stored as a default double).

"The absolute time represented by this 128-bit data type is the sum of the two 64-bit components."

[Benjamin Dolgin]

Thanks for the advice.

I am working with NI on the issue but they are not very forthcoming. I could not find a definition of the time-spamp other than that is a Little-endian signed 64byte (not bit) integer. It is to be translated into seconds with a decimal point placed between two 32 byte words. Their point of reference is 1904 Jan 1 00:00:00Z and the time interval is in seconds. Under normal circumstances that should have been enough but, when I tried this using last 8 bytes followed by next but when I applied it using bytes 17 through 24 as offset in seconds I got the year, the month and the day correctly but the hours minutes and seconds were wrong.

ref_time1 is what MATLAB thinks is written in the *.hws file.

ref_time is the time stamp translated into MATLAB reference

    gr_inf=h5info(f_name, '/wfm_group0/axes/axis0'    );
      ref_time1=gr_inf.Attributes(9).Value;
      hx=[]; 
        for i=1:16; 
            hx=[hx dec2hex(ref_time1(i),2)]; 
        end;
        date_st=double(hex2dec(hx(17:24)));
        ref_time(1)= datenum(1904,1,1,0,0,0)+date_st/3600/24;
        ref_time(2)= double(hex2dec([hx(9:16) hx(1:8)]))/2^64;

I am probably parsing the hexagonal string "hx" wrong

[dpb]

http://www.ni.com/tutorial/7900/en/ says "The LabVIEW timestamp is a 128-bit data type...". That's 16-bytes which is what the hfinfo call thought was there. I don't see that the NI doc actually states about endianess, that would be an issue.

There's also the issue of whether it's Gregorian or Julian calendar as I note one respondent had issues with the Gregorian as claimed in the above link.

I was asking if you can't provide the actual 16 bytes returned and do you know what the answer is supposed to be?

I always loved NI hardware but detested their software libraries...I've not used any of their gear nor LABVIEW in over 20 years now...and can't says as how's I've missed it! :)

[dpb]

OK, messing around w/ the example on the NI page, if I take their third example of fractional seconds of a string of '0xCCCCCCCCCCCCCCCD'

>> c='CCCCCCCCCCCCCCCD';  % ML character representation
>> atto=sscanf(c,'%lx')  % the atto-seconds field
atto =
14757395258967641293
>> magn=uint64(2^64);    % max 64-bit integer
>> f=double(atto)/double(magn)  % convert to decimal (approx)
f =
  0.8000
>>

That is what they say the value is after the 54 seconds in 12/31/1903 23:59:54.800.

If you've got the 16 bytes you're almost there it would seem.

Answer 1

[Benjamin Dolgin]

MATLAB h5info has a bug: it does not recognize int128 variables in a NI LABVIEW files. Time stamp is one of those variables MATLAB does not recognize. Time stamp actually consists of two int64 numbers; the first one is signed the second is unsigned (see http://www.ni.com/tutorial/7900/en/ ). When one attempts to read a time stamp using h5info one gets a warning

Warning: There is no MATLAB integer type that corresponds to an HDF5 integer type with 16 bytes. The data will be treated as an uninterpreted uint8 array.

Subsequently, when the actual attribute is read one gets sixteen int8 numbers as opposed to one 128 bit or two int64 numbers. For example the following code applied to a data file produced by NI Oscilloscope program produces a wrong result:

gr_inf=h5info(file_name, '/wfm_group0/axes/axis0' );
ref_time1=gr_inf.Attributes(9).Value;
ref_time1'
ans =

    138  113  222  106   31  103   60  142  210  196    4  197    0    0    0    0

For the actual file recorded on 19-Jan-2016 10:20:37.122669 the answer should be 3536061637 and 2262843920682704384

The following code recovers appropriate numbers and generates UTC time (with credits to "dpb" and Stephen Van Kooten)

warning('off','MATLAB:imagesci:hdf5dataset:integerButNoSuchMatlabClass');
gr_inf=h5info(file_name, '/wfm_group0/axes/axis0'    );
warning('on','MATLAB:imagesci:hdf5dataset:integerButNoSuchMatlabClass');
ref_time1=gr_inf.Attributes(9).Value;

wholeUint = sscanf(sprintf('%02X',[ref_time1(13:16) ref_time1(9:12)]),'%lx');
if wholeUint>=bitshift(uint64(1),63)
  %convert to negative with 2s complement
  whole =  -1*(int64(intmax('uint64') - wholeUint)+1);
else
  whole = int64( wholeUint );
end
frac = sscanf(sprintf('%02X',[ref_time1( 5: 8) ref_time1(1: 4)]),'%lx');
utc = datetime(double(whole)+double(frac)/2^64, ...
               'ConvertFrom','epochTime','Epoch','1904-01-01', ...
               'TimeZone','UTC');

[dpb]

Again, I'd make use of the internal i/o scanning routines instead of explicitly converting byte-by-byte...

>> timebytes =[138 113 222 106  31 103  60 142 210 196   4 197  0  0];
>> t(1)=sscanf(sprintf('%02X',[timebytes( 5: 8) timebytes(1: 4)]),'%lX');
>> t(2)=sscanf(sprintf('%02X',[timebytes(13:16) timebytes(9:12)]),'%lX')
t =
2262843920682704490           3536061637
>>

I've not worked on the conversion to the prime date, seems as though one could use the offset from the LabVIEW epoch origin to that in Matlab as datenumber and make the adjustment and then use builtin datenum assuming they did, indeed, use Julian calendar.

I've not got the newer release with the new date class for comparison; you have investigated it also, I presume, for any similar applicability as above?

[dpb]

Also, if you don't like sprintf, there's always

>> hx=reshape(dec2hex(timebytes).',1,[])
hx =
8A71DE6A1F673C8ED2C404C500000000
>>

and you can go ahead and do the endianess thingie inside there as well. I suspect that dec2hex may be a little faster than the more general sprintf but haven't timed it. Note the "trick" above is that have to transpose output first to get the two-byte characters in column-major order, then the subsequent reshape to orient as single row character string is the equivalent to the idiom (:).' but can't write that without an temporary so this saves that as can't use additional parens after a function result in Matlab.

[dpb]

"The following code recovers appropriate numbers and generates UTC time (with credits to "dpb" and Stephen Van Kooten)"

Glad you finally (!!!) got it worked out. I'd think this might be a worthwhile File Exchange submission if you're so inclined...

[Guru Subramani]

"Time stamp actually consists of two int64; the first one is signed the second is unsigned " - Benjamin Dolgin, I think this describes what is going on clearly. Thanks!

[Walter Roberson]

"MATLAB h5info has a bug: it does not recognize int128 variables"

That is not a bug. MATLAB does not have any int128 data type. It is a limitation. To handle int128 or uint128, Mathworks would have to create a complete extended-precision software mathematics facility: all other calculations in basic MATLAB are done with hardware instructions, but the x64 instruction set does not offer 128 bit integer operations.