FileCopy does't work

Copying files from source directory to destination directory is NOT working.

Below is my source code. Please help me

WorkingDir = 'C:\Users\shastry\MathWorks\test_that_results_hatch_B1CYKd';
OutputFolder = fullfile(WorkingDir, 'Tmp');
if ~exist(OutputFolder, 'dir')
  mkdir(OutputFolder);
end
Param.Source = WorkingDir;            %'C:\Users...'; 
Param.Target = OutputFolder;   %'C:\Users...';
Files=dir(fullfile(Param.Source,'**\*.*'));
Files = Files(~[Files.isdir]);  % Remove folders from list;
for k = 1:length(Files)
    if contains(Files(k).folder,'debug')
        if contains(Files(k).name,'test_that')
            CurrentFile = Files(k).name;
            exist(CurrentFile,'file') % 1=exist 0=doesn't exist
            RenameFile= regexprep(CurrentFile,{'\.'},{''}); % Replace dot from Files.name
            exist(RenameFile,'file')  % 1=exist 0=doesn't exist
            copyfile(fullfile(Param.Source, CurrentFile), fullfile(Param.Target, RenameFile));% copy sourcefolder file to destnationfolder file
        end
    end
end

[Guillaume]

The copyfile function works perfectly well. Now, it's very possible that there are bugs in you code so that it doesn't do what you want but since you haven't told us what it's supposed to do nor what it actually does, it's difficult to help.

One thing that certainly doesn't look right is that you're looking in subdirectories of Param.Source for those that contain files with name test_that in a subfolder whose name contain debug, but then attempt to copy the same file from the Param.Source directory.

Your exist calls also don't make much sense since they check for the existence of files in the current directory which may be completely different from the one where the file is found. I assume you get 0 displayed in the command window.

[Matlab]

it doesn't do what you want but since you haven't told us what it's supposed to do nor what it actually does, it's difficult to help.

Goal :The goal is to move files from every Subfolder "debug" to a new folder "OutputFolder"

attempt to copy the same file from the Param.Source directory

I want to copy "CurrentFile" (test_that.DEBUG) from the Subdirectory (test_that_results_hatch_B1CYKd\debug) "debug" to new directory "OutputFolder" (test_that_results_hatch_B1CYKd\Tmp).

Note: CurrentFile (test_that.DEBUG) has dot in file name which I don't want to have ( Since filereading is not supported . I want to pass filename as input argument to a function[readtable] ). So I have to remove dot from CurrentFile name and rename CurrentFile to RenameFile (test_thatDEBUG).

I assume you get 0 displayed in the command window

I know it's NOT working . I added it for debug purpose :-)

RenameFile= regexprep(CurrentFile,{'\.'},{''}) % Replace dot from Files.name

Remove dot from the current file name

Many thanks for your kind help. For reference I have added *.mat file. I am sure now I can move forward.

[Matlab]

I have rewritten the code as below , it copies the ranamed file to OutputFolder directory

WorkingDir = 'C:\Users\shastry\MathWorks\test_that_results_hatch_B1CYKd';
OutputFolder = fullfile(WorkingDir, 'Tmp');
if ~exist(OutputFolder, 'dir')
  mkdir(OutputFolder);
end
Param.Source = WorkingDir;     %'C:\Users...'; 
Param.Target = OutputFolder;   %'C:\Users...';
Files=dir(fullfile(Param.Source,'**\*.*'));
Files = Files(~[Files.isdir]);  % Remove folders from list;
for k = 1:length(Files)
    %debug subfolder
    if contains(Files(k).folder,'debug')    % look for folder with debug name
        if contains(Files(k).name,'test_that') % look for file with test_that name
            RenameFile = regexprep(Files(k).name,{'\.'},{''}); % Replace dot Files.name
            [SUCCESS,MESSAGE,MESSAGEID] = copyfile(fullfile(Files(k).folder ,Files(k).name), fullfile(Param.Target, RenameFile));% copy sourcefolder file to destnationfolder file
        end
    end
%     result- subfolder
    if contains(Files(k).folder,'results-')    % look for folder with results-
        if contains(Files(k).name,'autoserv') % look for file with autoserv
            RenameFile = regexprep(Files(k).name,{'\.'},{''}); % Replace dot Files.name
            [SUCCESS,MESSAGE,MESSAGEID] = copyfile(fullfile(Files(k).folder ,Files(k).name), fullfile(Param.Target, RenameFile));% copy sourcefolder file to destnationfolder file
        end
    end
    %firmware_ subfolder
    if contains(Files(k).folder,'firmware_')    % look for folder with firmware_
        if contains(Files(k).name,'firmware_') % look for file with test_that name
            RenameFile = regexprep(Files(k).name,{'\.'},{''}); % Replace dot Files.name
            [SUCCESS,MESSAGE,MESSAGEID] = copyfile(fullfile(Files(k).folder ,Files(k).name), fullfile(Param.Target, RenameFile));% copy sourcefolder file to destnationfolder file
        end
    end
end

[Guillaume]

That looks much better, you're now using the actual folder where the file is found.

I assume your problem is solved?

I don't understand your comment about the . in the file name. Matlab IO functions don't care one bit about the format of the filename, you can have as many or as few . in the filename, it won't change anything.

Note that it would be clearer if you passed char arrays to to regexprep instead of scalar cell arrays:

RenameFile = regexprep(Files(k).name, '\.', '');
%or
RenameFile = strrep(Files(k).name, '.', '');

and the whole lot, if it's needed would be better written only once before the ifs.

Also the double ifs could be written as just one:

if contains(Files(k).folder,'debug') && contains(Files(k).name,'test_that')

Answer 1

[Guillaume]

I don't remember the details of your previous question, but clearly the code I gave you is meant to work with data imported a particular way. If you change the way you import the data, it's not going to work. Reading the code above Content must be a scalar structure where each field is itself a scalar structure with just one field which is a table.

In your mat file, Content is a cell array, not a structure as expected. So yes, you'll get an error but not the one you mention (it's fieldnames that would complain first). The cell array can trivially be converted into a structure but the substructures contain more than one table. As it is, the code would only use the first table. The code would have to be modified to cope with more than one table per structure, but it's unclear what you want to do.

Furthermore, the Content description you pasted above doesn't match your attached mat file. Above, Content is indeed a structure as expected, but the fields contain cell arrays, not structures. So, yes you'll get a struct2cell error from that.

I think I told you before, if the format of your input keep on changing don't expect the solutions you're given to work anymore. It's important to be consistant.

[Guillaume]

I want the code would have to be modified to cope with more than one table per structure

Well, that can mean anything. One way to cope is to discard everyhing but the first table. Probably not what you want, though. As far as I can tell (I've forgotten what it was about) the code I wrote joined some tables after doing some strange calculations on the time and discarding everything but the Message column. Should all the tables be joined regardless of which field and subfield they're in? Same strange time calculations for all? How should the Message column be named in the joined table?

[Matlab]

Well, that can mean anything. One way to cope is to discard everyhing but the first table. Probably not what you want, though

Yes, I don't want .

Should all the tables be joined regardless of which field and subfield they're in ?

No. They should contain - time, DebugLevel1,DebugLevel2, and Message

Same strange time calculations for all?

I can see all the file generated out of N test ( 90%) following same pattern barring time seires format.

How should the Message column be named in the joined table?

We can keep same name . No problem

Thanks a lot.

[Guillaume]

We can keep same name . No problem

It is a problem. You can't have multiple columns with the same name in the joined table. So, the Message variable of each table would have to be renamed to something unique. And now that you want to keep the other variables as well, they would also have to be renamed.

[Matlab]

Sure, I would preferably put coloumn according to filename, so that it's clear. Others no problem. We can keep same or make some meaningful. They are from debug log structures Or coloumn beginning with

test_thatfilename_Timeseries

Message can be

test_thatfilename_log1 test_thatfilename_log2 test_thatfilename_Messages

[Matlab]

any feedback here for me ? Thanks !

[Guillaume]

I'm sorry but I don't have the time to debug your code for you. You need to use the debugger to check that your code does what you meant it to do. Currently, it produces a Content that makes no sense at all.

Working on the assumption that your Content variable is a structure whose fields are all structures themselves and each field of these substructures is a table, the following may do more or less what you want. It's a minor modification to the code I already gave you, in order to handle the substructures.

Earlier, you posted a Content in filecopy.mat that is a cell array of structure. That can easily be changed int the structure required by the code below with, e.g.:

Content = cell2struct(Content, {'a', 'b', 'c', 'd'}, 2);

Of course, it would be better if your code created the structure with more meaningful field names.

contentfields = fieldnames(Content);
for fieldidx = 1:numel(contentfields)    %iterate over each field of Content
    substruct = Content.(contentfields{fieldidx});  %which are themselves structures containing tables
    subfields = fieldnames(substruct);    
    for subfieldidx = 1:numel(subfields)  %iterate over the subfields
        structtable = substruct.(subfields{subfieldidx});  %and get the table out of it      
        structtable.Date = duration(structtable.Date.Hour, structtable.Date.Minute, structtable.Date.Second);  %extract h/m/s and make that a duration.
        structtable.Date = structtable.Date - structtable.Date(1);  %elapsed time since start of log.        
        structtable.Properties.VariableNames{1} = 'WeirdDuration';  %rename the time column
        structtable.Properties.VariableNames(2:end) = strcat(structtable.Properties.VariableNames(2:end), sprintf('_%s_%s', contentfields{fieldidx}, subfields{subfieldidx}));  %append content field name and subfield name to remaining table variables
        structtimetable = table2timetable(structtable);  %convert to timetable
        structtimetable = rmmissing(structtimetable);  %remove invalid rows to avoid problems with synchronize
        structtimetable.Properties.RowTimes.Format = 'hh:mm:ss.SSS'; %duration format to support millisecond
        if fieldidx == 1 && subfieldidx == 1
            joinedtimetable = structtimetable;  %1st time, create output
        else 
            joinedtimetable = synchronize(joinedtimetable, structtimetable, 'union'); %subsequent times, synchronize. Choose whichever method is prefered
        end
    end
end

[Matlab]

Thanks a lot for the code.

I am wondering what is the code doing ?

Why it's behaving differently when structure with field has [ 1x1 struct] & structure with field has [ 1x4 struct].

file1.mat is using case -1 code. This works fine for when [1 x 1 ]

Code-1 start

[contentfields] = fieldnames(Content);
for fieldidx = 1:numel(contentfields)    %iterate over each field of Content
    structtable = struct2cell(Content.(contentfields{fieldidx}));  %extract the structure within the field by converting it to cell array (avoids having to work out what its name is)
    structtable = structtable{1}; %get the table out of the cell
    structtable.Date = duration(structtable.Date.Hour, structtable.Date.Minute, structtable.Date.Second);  %extract h/m/s and make that a duration.
    structtable.Date = structtable.Date - structtable.Date(1);  %elapsed time since start of log.
    structtable.Properties.VariableNames{1} = 'WeirdDuration';  %rename the time column {system time logs are wired}
    structtimetable = table2timetable(structtable(:, {'WeirdDuration','Message'}));  %convert to timetable, only keeping Date and Message
    structtimetable.Properties.VariableNames{1} = sprintf('Message_%s', contentfields{fieldidx});  %rename Message variable so we know where it came from
    structtimetable = rmmissing(structtimetable);  %remove invalid rows to avoid problems with synchronize
    structtimetable.Properties.RowTimes.Format = 'hh:mm:ss.SSS'; %duration format to support millisecond
    if fieldidx == 1
        joinedtimetable = structtimetable;  %1st time, create output
    else 
        fieldidx
        joinedtimetable = synchronize(joinedtimetable, structtimetable, 'union'); %subsequent times, synchronize. Choose whichever method is prefered
    end
end

Code-1 End

file2.mat is using case -2 code. This works fine for when [1 x 4 ] but doesn't work when [1 x 1 ]

Code-2 Start

contentfields = fieldnames(Content);

for fieldidx = 1:numel(contentfields)    %iterate over each field of Content
%     fieldidx = 3;
    substruct = Content.(contentfields{fieldidx});  %which are themselves structures containing tables
    subfields = fieldnames(substruct)    
    for subfieldidx = 1:numel(subfields)  %iterate over the subfields
        structtable = substruct.(subfields{subfieldidx});  %and get the table out of it      
        structtable = structtable{1};
        structtable.Date = duration(structtable.Date.Hour, structtable.Date.Minute, structtable.Date.Second);  %extract h/m/s and make that a duration.
        structtable.Date = structtable.Date - structtable.Date(1);  %elapsed time since start of log.        
        structtable.Properties.VariableNames{1} = 'WeirdDuration';  %rename the time column
        structtable.Properties.VariableNames(2:end) = strcat(structtable.Properties.VariableNames(2:end), sprintf('_%s_%s', contentfields{fieldidx}, subfields{subfieldidx}));  %append content field name and subfield name to remaining table variables
        structtimetable = table2timetable(structtable);  %convert to timetable
        structtimetable = rmmissing(structtimetable);  %remove invalid rows to avoid problems with synchronize
        structtimetable.Properties.RowTimes.Format = 'hh:mm:ss.SSS'; %duration format to support millisecond
        if fieldidx == 1 && subfieldidx == 1
            joinedtimetable = structtimetable;  %1st time, create output
        else 
            joinedtimetable = synchronize(joinedtimetable, structtimetable, 'union'); %subsequent times, synchronize. Choose whichever method is prefered
        end
    end
end

Code-2 End

I didn't understood why joinedtimetable doesn't work in case -2 when struct with [1 x 1 ] is used ? We don't get joinedtimetable data for structure [1 x 1 ] dimension.

synchronize doesn't workanymore here. structtimetable contains correct data.

[Guillaume]

Your second code does not work with file1 and file2. You've added an extra line compared to the code I gave:

 structtable = structtable{1};

which means that the substructure must be a cell array containing one table.

We're back to the same problem, nothing is ever consistent. Your containers keep varying in their structure.

The code I gave you works on a scalar structure whose fields are all scalar structures containing a table. Your file1 Content conforms to this. Your file2 Content has some structure arrays. My code will still work with them but only use the first element of the structure array.

The same code to cope with structure arrays:

contentfields = fieldnames(Content);
for fieldidx = 1:numel(contentfields)    %iterate over each field of Content
    substruct = Content.(contentfields{fieldidx});  %which are structure arrays containing tables
    subfields = fieldnames(substruct);    
    for subelem = 1:numel(substruct)  %iterate over each element of the array
        for subfieldidx = 1:numel(subfields)  %iterate over the subfields
            structtable = substruct(subelem).(subfields{subfieldidx});  %and get the table out of it      
            structtable.Date = duration(structtable.Date.Hour, structtable.Date.Minute, structtable.Date.Second);  %extract h/m/s and make that a duration.
            structtable.Date = structtable.Date - structtable.Date(1);  %elapsed time since start of log.        
            structtable.Properties.VariableNames{1} = 'WeirdDuration';  %rename the time column
            structtable.Properties.VariableNames(2:end) = strcat(structtable.Properties.VariableNames(2:end), sprintf('_%s%d_%s', contentfields{fieldidx}, subelem, subfields{subfieldidx}));  %append content field name and subfield name to remaining table variables
            structtimetable = table2timetable(structtable);  %convert to timetable
            structtimetable = rmmissing(structtimetable);  %remove invalid rows to avoid problems with synchronize
            structtimetable.Properties.RowTimes.Format = 'hh:mm:ss.SSS'; %duration format to support millisecond
            if fieldidx == 1 && subelem == 1 && subfieldidx == 1
                joinedtimetable = structtimetable;  %1st time, create output
            else 
                joinedtimetable = synchronize(joinedtimetable, structtimetable, 'union'); %subsequent times, synchronize. Choose whichever method is prefered
            end
        end
    end
end

[Matlab]

Sorry for typo. structtable = structtable{1}; This is not a part of code. But as you stated

My code will still work with them but only use the first element of the structure array.

Here when structures of 1x1 , joinedtimetable is not showing messages. It works when structures of 1x4 is there. I can see joinedtimetable displaying correct synchronization. I want all elements to be part of joinedtimetable even when structures is 1x1. That's time synchronization with other table and messages Thanks

[Guillaume]

Here when structures of 1x1 , joinedtimetable is not showing messages

If you're talking about the Content in your file1, this has nothing to do with the number of elements in the structure and everything to do with what's in the tables. In particular, most of the tables have Dummy columns containing just empty string. rmmissing removes rows which have empty values in any column, so you end up removing all the rows of the tables.

I have no idea why you now have empty Dummy variables in your tables (again, it's important to be consistent!) but you need to decide whether you want to rmmissing empty rows or not. If you do, make sure that the Dummies are not empty. Or possibly, only rmmissing on the Message:

 structtimetable = rmmissing(structtimetable, 1);

[Guillaume]

Please don't start new answers, you're not answering your own question. Keep on commenting on the answer.

I recommend you learn to use the debugger and step through the code to check whether or not it does what you want.

With regards to messages disappearing from Content.faft_client.faft and possibly other tables, it's the expected behaviour of synchronize when you have several rows with the exact same time. synchronize only keep one. See:

tt = table2timetable(Content.faft_client.faft);
synchronize(tt, tt, 'union')  %you only get one row per unique time

If you're expecting something else then it's not a synchronisation of the timetables and I'm not sure what it is. In particular you'd have to explain which row of a duplicated time in one timetable should be matched with which row of the duplicated time in the other table.

[Matlab]

Sorry to trouble you.

it's the expected behaviour of synchronize when you have several rows with the exact same time. synchronize only keep one.

OK.

In particular you'd have to explain which row of a duplicated time in one timetable should be matched with which row of the duplicated time in the other table

I need this implementation where Timeseries remains in current state but all row's message is visible.

[Guillaume]

given these two timeseries

t1 = timetable(hours([0;0;0;0]), {'A';'B';'C';'D'})
t2 = timetable(hours([0;0]), [pi; sqrt(2)])

What synchronized output would you want (and why?)

[Matlab]

With the example implemented in below fashion, we get

>> join = synchronize(t1, t2, 'union')

join =

  1×2 timetable

    Time    Var1_t1    Var1_t2
    ____    _______    _______

    0 hr      'A'      3.1416 

I don't know which synchronize method will give output but for my implementation I would like to have following output

   Time    Var1_t1    Var1_t2
    ____    _______    _______

    0 hr      'A'      3.1416 

    0 hr      'B'      1.4142 

    0 hr      'C'      "" 

    0 hr      'D'      ""

Thanks a lot for the pointer and guidence.

[Guillaume]

Why does A get matched with pi and not sqrt(2) or ""? Why does B get matched with sqrt(2) and not pi or ""? and so on...

The purpose of synchronize is to have all events that occur at the same time on the same row. Here you seem to want an arbitrary number of events all occuring at the same time on different rows, matched up against each other arbitrarily. Completely ignoring code, from a design point of view it's not a good design.

In any case, it's not something that synchronize can do.

[Matlab]

OK, Then how to proceed? Please help me. Thanks!

[Guillaume]

I don't know yet. It may not be simple.

In any case, it's a completely different question from what you originally asked, so you should start a new one.The two timetable examples I gave above would probably be a good start for that new question.

[Matlab]

Thanks a ton for all learnings i received from you!