Save files to folder loop won't save the .txt files

Hi there,

I'm currently trying to use a script to basically read in .txt files from a spectrometer (simply 2 columns, one wavenumbers and the other intensity counts)

It should loop through a specified folder to find every .txt file (a range of 10 samples), run the Savitsky-Golay smoothing filter on them with specified order (that can be changed to the user's desire) and then save the new smoothed sample data to a folder named the same, but with Savitsky-Golay added on to the end of the folder name. The new files will have '_SMOOTHED' added on to the oname of the original .txt files to distinguish between them.

The problem I have is that when I run the script, it creates this Savitsky-Golday folder, and performs everything as it should, but does not actually save the new .txt files into the folder. I did a full search to see if the new .txt files were being saved elsewhere, but it seems not. So the issue is, the folder is being created, but the data is not being saved in it after it has been smoothed. Any help would be greatly appreciated - I've provided the code below.

% location of data files
data_path = 'C:\....\Desktop\Fat-L100T10-785';

% create search string to pick up all txt files in folder
search_string = strcat(data_path, '\*.txt');
% find all files matching search string
files = dir(search_string);

% set the polynomial order for Savitzky-Golay
order = 5;

% specify where to save smoothed files
new_path = strcat(data_path, 'Savitzky-Golay/');
if ~exist(new_path, 'dir')
   mkdir(new_path);
end

for i = 1:numel(files)
    %% Raw Data Files
    %
    filename = files(i).name;
    % open raw data
    data = fopen(strcat(data_path, filename), 'r');
    % format of data to extract (%f is a floating point decimal/double)
    formatspec = '%f';
    % extract out the data and save into a single array
    % there probably IS a better way to do this but i couldn't figure it out
    D = fscanf(data, formatspec);

    % data saves as wave number in odd rows and intensity count in even rows

    % extract out the odd rows and save to separate array
    wave_number = flipud(D(1:2:end-1,:));
    % extract out the even rows and save to separte array
    count = flipud(D(2:2:end,:));
    
    %% Run Savitzky-Golay Algorithm
    % run the smoothign algorithm with the Savitzky-Golay method
    smoothed_data = smoothdata(count, 'sgolay', 'degree', order, 'SmoothingFactor', 0.03);
    
    % NB SmoothingFactor (0-1) specifies the window used for the algorithm.
    % a value closer to 0 makes the window smaller and allows you to reduce the
    % order of the polynomial. This is the only way I could figure out how to
    % control the frame length. If you increase the window to closer to one,
    % you need to increase the order of the polynomial also
    
    %% Save Data
    % Create table of data with variable names
    T = table(wave_number, count, 'VariableNames', { 'RamanWaveNumber', 'IntensityCount'} );
    % define Filename
    new_filename = strcat(new_path, erase(filename, '.txt'), '_SMOOTHED.txt');
    % save data to text file
    writetable(T, new_filename)
    
end

[Guillaume]

Have you checked the value of new_filename to make sure that it is what you expect?

I'd recommend using fullfile instead of strcat to build paths. With fullfile you don't have to worry about path separators. My first instinct was that you were missing the path separator when you were building new_filename and hence your files would have been saved in the directory above where you expected. But your path is terminated (with a unix terminator but windows is fine with that).

Of course, the one possible reason that no file is written is that the loop doesn't execute because files is empty. Have you checked that?

Completely unrelated:

    wave_number = D(end-1:-2:1, :);
    count = D(end:-2:2, :);

is simpler and faster than using flipud.

[Nicola Fairbairn]

Hi Guillaume,

So I checked and 'files' is definitely not empty. It has all the correct files there as expected.

I'm getting this error when I run:

>> sgolay

Error using fscanf

Invalid file identifier. Use fopen to generate a valid file identifier.

Error in sgolay (line 29)

D = fscanf(data, formatspec);

>> sgolay

Error using fscanf

Invalid file identifier. Use fopen to generate a valid

file identifier.

Error in sgolay (line 29)

D = fscanf(data, formatspec);

[Guillaume]

Ah, well, "The problem I have is that when I run the script, it creates this Savitsky-Golday folder, and performs everything as it should" is completely different from my code throws an error. Of course, if you get an error, the code won't do what it's meant to do. It just stops.

I didn't check the bit where you open the file since I assumed it didn't throw an error. As dpb correctly points out you build the filename wrong so you actually never open the file. So, use fullfile everywhere to make sure that you don't have problems with file separators. See dpb's answer.

[Nicola Fairbairn]

My fault, I should've pointed that out - it slipped my mind as I had moved the command window out of the way whilst I ran the code and didn't think to check it. Thanks for your help!

Answer 1

[dpb]

% location of data files
data_path = 'C:\....\Desktop\Fat-L100T10-785';
% create search string to pick up all txt files in folder
search_string = strcat(data_path, '\*.txt');
% find all files matching search string
files = dir(search_string);
...
for i = 1:numel(files)
    %% Raw Data Files
    data = fopen(strcat(data_path,files(i).name), 'r');
    ...

As Guillaume says, you're not using fullfile and so you missed the file separator in creating the filename to try to open by doing straight string catenation above...and since you didn't incorporate any error-checking, you didn't discover the problem at the time...

% location of data files
data_path = 'C:\....\Desktop\Fat-L100T10-785';
% create search string to pick up all txt files in folder
search_string = strcat(data_path, '\*.txt');
% find all files matching search string
files = dir(search_string);
% set the polynomial order for Savitzky-Golay
...
for i = 1:numel(files)
    filename = files(i).name;
    % open raw data
    [data,msg] = fopen(fullfile(data_path, filename), 'r');
    if data<0, error('msg'), end
    ...

Similarly, later on, use the fileparts function to retrieve file base name and build new filename...

    ...
    % define Filename
    [~,filename]=fileparts(files(i).name);                        % retrieve base filename less extensioni
    new_filename=fullfile(new_path,[filename '_SMOOTHED.txt']);   % and build the new qualified filename
    ...

[Guillaume]

And

search_string = strcat(data_path, '\*.txt');

should be

search_string = fullfile(data_path, '*.txt');

[dpb]

Good catch @G; missed noticing that.

The code becomes more generally usable as simply changing the wildcard string will let it work for .csv files, say, with few other changes. More generally, if were to specify the input and output file extensions as variables, then it would be almost totally generic.

ASIDE EDITORIAL COMMENT:

I've thought all along TMW ought to supply another function along with fullfile and fileparts which specifically adds an extension to the filename to avoid needing to write the specific catenation as above.

With the new string class one could use the "+" operator which is a little simpler, but since dir still (and probably always will owing to compatibility) returns a character string, the need to cast to string pretty much negates any gain.

[Nicola Fairbairn]

Thanks for your help, both!