Extract individual member fields from h5 file

I have some large h5 files that I am unable to efficiently bring into MATLAB. Each group in these large files (50+GB for some) turns into a structure with dozens of fields, each of which is a vector with millions of individual values. This overwhelms the RAM on my computer, so I am looking for a way to only extract the relative handful of fields that I actually need rather than importing the entire h5 group and then just deleting what I don't currently need.

So, for example, in the file indicated below I am looking for a way to extract perhaps just the timestamp and state vectors without importing the rest.

h5info([Source File{kk}],Group)

ans =

struct with fields:

Filename: 'D:\Data\Msg.h5'

Name: ''

Datatype: [1×1 struct]

Dataspace: [1×1 struct]

ChunkSize: 9000

FillValue: [1×1 struct]

Filters: []

Attributes: [2×1 struct]

Raw = h5read([Source File{kk}],Group)

Raw =

struct with fields:

Msg_msg_type: [1000000×1 uint8]

Msg_timestamp: {1000000×1 cell}

irig_time: {1000000×1 cell}

timestamp: [1000000×1 double]

log_msg: {1000000×1 cell}

state: {1000000×1 cell}

julian_date: [1000000×1 double]

Using the typical start and count options don't seem to work because, at the level they operate, there is only a 1x1 structure - i.e., a single unparseable item.

Any suggestions?

Answer 1

[Stephen23]

Inspect the HDF5 layout

What does Group actually point to?

info = h5info(filename, Group)

Look carefully at:

info.Datatype

If this is a compound datatype, you'll see something like 'H5T_COMPOUND' and member names corresponding to:

Msg_msg_type
Msg_timestamp
irig_time
timestamp
log_msg
state
julian_date

Read only selected fields

MATLAB's high-level h5read does not provide a way to select only certain members of a compound datatype. However, the HDF5 library does:

https://www.mathworks.com/help/matlab/hdf5-files.html

The usual approach is:

fileID = H5F.open(filename,'H5F_ACC_RDONLY','H5P_DEFAULT');
dsetID = H5D.open(fileID, Group);
dtypeID = H5D.get_type(dsetID);

Create a new memory datatype containing only the fields you need:

memtype = H5T.create('H5T_COMPOUND',16); % example size
H5T.insert(memtype, 'timestamp',0, H5T.copy('H5T_NATIVE_DOUBLE'));
% insert state definition similarly

then:

data = H5D.read(dsetID, memtype, 'H5S_ALL','H5S_ALL','H5P_DEFAULT');

This causes HDF5 to transfer only those selected members from disk.

[Stephen Hall]

When I use the h5read function on my smallest data set, I see the following structure:

Data.U1 =

struct with fields:

irig_times: [1678319×1 double]

c_x: [1678319×1 single]

c_y: [1678319×1 single]

f_x: [1678319×1 single]

f_y: [1678319×1 single]

bkg: [1678319×1 single]

p: [1678319×1 int32]

Time: [1678319×1 datetime]

f: [355535×1 double]

So I tried this code:

fileID = H5F.open([Source File{kk}],'H5F_ACC_RDONLY','H5P_DEFAULT');

dsetID = H5D.open(fileID, Group);

dtypeID = H5D.get_type(dsetID);

for ii=1:length(Members)

memtype = H5T.create('H5T_COMPOUND',16); % example size

if strcmpi(Members{jj},'f')

H5T.insert(memtype,Members{ii},0, H5T.copy('H5T_NATIVE_DOUBLE'));

else

H5T.insert(memtype,Members{ii},0, H5T.copy('H5P_DEFAULT'));

end

Raw = H5D.read(dsetID, memtype, 'H5S_ALL','H5S_ALL','H5P_DEFAULT');

try

Data.(U{jj}).(Members{ii}) = cell2mat(Raw.(Members{ii}));

catch

Data.(U{jj}).(Members{ii}) = Raw.(Members{ii});

end

end

And then I get:

Error using matlab.internal.sci.hdf5lib2

The memory type is incompatible with VLEN H5T_STRING. Consider using 'H5ML_DEFAULT' instead.

Error in

H5D.read (line 55)

buf = matlab.internal.sci.hdf5lib2('H5Dread',varargin{:});

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Error in

Analysis_1 (line 125)

Raw = H5D.read(dsetID, memtype, 'H5S_ALL','H5S_ALL','H5P_DEFAULT');

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

So it looks like there are mismatches between the datatypes in the h5 files and the H5P_DEFAULT. And yes, I tried using H5ML_DEFAULT instead, just gives me a slightly different error. Any guidance would be appreciated, since the documentation I've found on h5 files online have been slim.

[Stephen23]

AFAICT the key issue is that H5P_DEFAULT is a property-list identifier, not an HDF5 datatype, so it cannot be supplied to H5T.insert. Instead of guessing each datatype, copy the selected member’s datatype directly from the dataset’s compound datatype. Your file also appears to contain a scalar compound dataset whose members are variable-length arrays. That explains why h5info reports a 1x1 dataspace while h5read returns million-element vectors.

Inspect the actual member types

This utility should print the actual HDF5 class of every compound member (AI supported, untested code):

function inspectCompoundMembers(filename, datasetName)
fileID = H5F.open(filename, 'H5F_ACC_RDONLY', 'H5P_DEFAULT');
fileCleanup = onCleanup(@() H5F.close(fileID));
dsetID = H5D.open(fileID, datasetName);
dsetCleanup = onCleanup(@() H5D.close(dsetID));
typeID = H5D.get_type(dsetID);
typeCleanup = onCleanup(@() H5T.close(typeID));
fprintf('Dataset datatype: %s\n', H5T.get_class(typeID));
nMembers = H5T.get_nmembers(typeID);
for k = 1:nMembers
    name = H5T.get_member_name(typeID, k-1);
    memberTypeID = H5T.get_member_type(typeID, k-1);
    memberCleanup = onCleanup(@() H5T.close(memberTypeID));
    memberClass = H5T.get_class(memberTypeID);
    fprintf('%2d  %-30s  %s', k, name, memberClass);
    if strcmp(memberClass, 'H5T_VLEN')
        baseTypeID = H5T.get_super(memberTypeID);
        baseCleanup = onCleanup(@() H5T.close(baseTypeID));
        fprintf(' of %s', H5T.get_class(baseTypeID));
    elseif strcmp(memberClass, 'H5T_STRING')
        if H5T.is_variable_str(memberTypeID)
            fprintf(' (variable-length)');
        else
            fprintf(' (fixed-length)');
        end
    end
    fprintf('\n');
    clear memberCleanup baseCleanup
end
end

Call it like:

inspectCompoundMembers(SourceFile{kk}, Group)

This should reveal whether state is:

• a variable-length numeric array,
• a variable-length string,
• a fixed-length string,
• an enum,
• or some other datatype.

Read one compound member using its actual datatype

The following helper locates a member by name, copies its datatype, constructs a one-member compound memory datatype, and reads only that member (AI supported, untested code):

function value = h5readCompoundMember(filename, datasetName, memberName)
fileID = H5F.open(filename, 'H5F_ACC_RDONLY', 'H5P_DEFAULT');
fileCleanup = onCleanup(@() H5F.close(fileID));
dsetID = H5D.open(fileID, datasetName);
dsetCleanup = onCleanup(@() H5D.close(dsetID));
fileTypeID = H5D.get_type(dsetID);
fileTypeCleanup = onCleanup(@() H5T.close(fileTypeID));
% Verify that the dataset has a compound datatype.
if ~strcmp(H5T.get_class(fileTypeID), 'H5T_COMPOUND')
    error('Dataset "%s" is not an HDF5 compound dataset.', datasetName);
end
% Locate the requested compound member.
memberIndex = H5T.get_member_index(fileTypeID, memberName);
if memberIndex < 0
    error('Member "%s" was not found in dataset "%s".', memberName, datasetName);
end
% Copy the member's real datatype from the file.
memberTypeID = H5T.get_member_type(fileTypeID, memberIndex);
memberCleanup = onCleanup(@() H5T.close(memberTypeID));
% Create a compound memory datatype containing only this member.
memberSize = H5T.get_size(memberTypeID);
memTypeID = H5T.create('H5T_COMPOUND', memberSize);
memCleanup = onCleanup(@() H5T.close(memTypeID));
H5T.insert(memTypeID, memberName, 0, memberTypeID);
% HDF5 matches compound members by name. Members not present in the
% memory datatype are not returned.
raw = H5D.read(dsetID, memTypeID, 'H5S_ALL', 'H5S_ALL', 'H5P_DEFAULT');
value = raw.(memberName);
% A scalar VLEN member is commonly returned inside a one-element cell.
if iscell(value) && isscalar(value)
    value = value{1};
end
end

Use it like this:

filename = SourceFile{kk};

Data.U1.timestamp = h5readCompoundMember(filename, Group, 'timestamp');

Data.U1.state = h5readCompoundMember(filename, Group, 'state');

For your smaller dataset:

Data.U1.irig_times = h5readCompoundMember(filename, Group, 'irig_times');

Data.U1.c_x = h5readCompoundMember(filename, Group, 'c_x');

Data.U1.f = h5readCompoundMember(filename, Group, 'f');