File operations in parfor loop lead to file operations error randomly

I'm currently processing black & white images for a large number of experiments (datasets).

Therefore, a function called bwReadout is called for all datasets:

for ii = 1:nbr_datasets
    [status(1,ii),output_id,input_ids] = bwReadout(dataset_name,...
        input_subfolder_ids);
end

The relevant lines of bwReadout are:

function  [status,output_ids,input_ids] = bwReadout(dataset_name,...
    input_subfolder_ids)
% ...
parfor ii = 2:nbr_png
        % . open *.png file:
        fileID = fopen(fullfile(png_files(ii).folder,...
            png_files(ii).name),'r');
        % . skip 20 byte header and move to specified position i.e.
        % 20 bytes in file starting at beginning of file - 'bof':
        fseek(fileID,20,'bof');
        % . read *.png file:
        file_data = fread(fileID,'uint8=>uint8');
        % . close *.png file:
        fclose(fileID);
        % . save black & white image in PNG format to destination folder
        % '..\01-Preprocessed_Data\01-HSBW_Camera_Data\01-Pictures\':
        file_name = png_files(ii).name;
        file_path = fullfile(datasetOutputFolder,folder_010101,...
            subfolder_id_temp,file_name);
        fileID = fopen(file_path,'w');
        fwrite(fileID,file_data,'uint8');
        fclose(fileID);
        % . read and assign black & white image to Matlab matrix:
        bwdata(:,:,ii) = imread(file_path);
end
% ...
end

Doing that, I'm getting the following error randomly (0.5% of processed datasets):

'Error using bwReadout
Invalid fileID for worker. File operations must occur on the worker where the file was opened.
Error in run (line 417)
[status(1,ii),output_id,input_ids] = bwReadout(dataset_name,...'

Can anybody explain what causes the error and knows how to solve it?

Thanks in advance, Martin

Answer 1

[Harald]

Hi Martin,

are you using a thread-based or process-based pool? In any case, I would not have expected this to error.

Two ideas for addressing this:

a) Create a separate file ID in each loop iteration, i.e., use fileID(ii) instead of fileID.

b) Move the body of the parfor loop into a function.

I hope that either of these resolves the issue for you.

Best wishes,

Harald

[Martin Schinnerl]

Hi Harald,

thank you for your fast reply.

I do not start the paralell pool programmatically. It is started automatically using the processes profile by using parfor.

I'll try your first idea a) and let you know.

Best regards,

Martin