Data being scrambled during processing a sequence of files

Morning, I have a problem with the processing of CSV files during the processing of a sequence of files from a simulation that I am running (image 1), the number on the folders representing the saved time steps occurring with a coupled open to show the files contained within. If the total number of folders containing the CSV's is ~< 150 (I have not determined the precise threshold) from the initial output the code performs perfectly and extracts and orders the CSV files for me to analyse. However, if the output from the simulation is ~>150 then the order of the data arrangement deteriorates (graph 1) and gets progressively worse with the increasing number of folders.

I must stress it is NOTthe output data of the simulation because I have tested it via two independent means. The first, directly from the UI of the programme itself (graph 2), which reads the same output data as image 1 to form the graph but also I have used a secondary code to sequence the same files as this code using a looped xlsread and the data arrangement extract the data (graph 3). Equally, I am running two types of analysis per simulation that export into two separate subfolders (transient and coupled analysis), the mis-arrangment exists for both.

From what I can determine, the problem may lie in one of these two lines or the compiler itself.

data_struct_Node = [mydata_NODE{:}]; %Compiles all the files into a structured cell array 
D_mat_N = cat(3,data_struct_Node.data); %concatenate the cell array - everything ordered based on data field

Attached are graphs and a set of folders which contains trainsent data for the first 30 folders of a total of 193. Does anyone have a solution to this problem?

Thanks

close;clc;clear;
%%IMPORT DATA%%

topLevelFolder = cd;
cd (topLevelFolder);
% Get list of all subfolders.
allSubFolders = genpath(topLevelFolder);
% Parse into a cell array.
remain = allSubFolders;
listOfFolderNames = {};

while true
	[singleSubFolder, remain] = strtok(remain, ';');
	if isempty(singleSubFolder)
		break;
	end
	listOfFolderNames = [listOfFolderNames singleSubFolder];
end
numberOfFolders = length(listOfFolderNames);

fprintf('Searching folders for node.csv files...\n');
for k = 1 : numberOfFolders %timesteps
	% Get this folder and print it out.
	thisFolder = listOfFolderNames{k};		
	% Get desired file type
	filePattern = sprintf('%s/*node.csv', thisFolder);
	baseFileNames = dir(filePattern);
	numberOfFiles = length(baseFileNames);
	if numberOfFiles >= 1
		% Go through all those files.
		for f = 1 : numberOfFiles            
			fullFileName = fullfile(thisFolder, baseFileNames(f).name);
            mydata_NODE{k} = importdata(fullFileName);			
		end
	else
		fprintf('     Folder %s has no files in it.\n', thisFolder);
    end    
end
%%
fprintf('Processing NODE information...\n');
data_struct_Node = [mydata_NODE{:}]; % Compiles all the files into a structured cell array
D_mat_N = cat(3,data_struct_Node.data); %concatenate the cell array - everything ordered based on data field. 

PWP_check = D_mat_N (391,2,:);
RS_PWP_check = reshape(PWP_check,size(PWP_check,1),size(PWP_check,3)); %plot graph

fprintf('done \n');

[Guillaume]

What's the purpose of genpath and the subsequent while loop? Is it a convoluted way of listing the subfolders of toplevelfolder (which could be done in just one line with dir)?

Have you checked that listOfFolderNames contains all the folders you want and in the right order? (bearing in mind that genpath is not designed for listing arbitrary subfolders and will exclude folders with specific patterns).

[Richard Rees]

Thanks for the reply, I looked at at https://matlab.fandom.com/wiki/FAQ for the section on how to process a sequence of files and adapted a few bits accordingly.

Actually, I just checked the listOfFolderNames and it is jumping sequences. How do I fix this?

[Guillaume]

Be aware that this FAQ is quite old and predates many improvements to matlab. In any case, nowhere does it suggests to use genpath, which is the wrong tool for this.

I don't know if the problem is from the usage of genpath or not. What's the rule for ordering the paths (bearing in mind that most likely currently, they're ordered alphabetically)?

[Richard Rees]

I'm looking through the comments now trying to figure it outa the moment.

In image 1, that is how the data is generated. Numerically the folders go from 000 to 10000 for example and at the end there are a few other files (see attached image). All I need it to do is process extract the data from the node.csv files in each subfolder in the same numerical seqeunce as it is written i.e. 000 to end.

[Richard Rees]

Hi, I fixed the problem by using natsortfiles. Thank you very much for your time and help in this.