Values in cell array keep getting overwritten

Hi there, I am new at MATLAB and currently trying to extract data from a particular subfolder and put it into a new cell array

basically i want to retrieve mean reaction times which is stored in a variable called a (location: parent_directory > containing many subject folders labelled img0001, img0002 etc > each subject folder contains a .mat file called stopchoicert.mat > variable a can be found here)

My code is as follows but instead of every iteration being placed into cell array, the values keep getting overwritten so i only have the mean reaction time of my last subject.

I have around 325 subjects but not all of them have mean reaction times, so I am unsure how many values in my cell array I should be getting at the end but the output rt that i get is a 326x1 cell array although only the 80th row contains the mean reaction time (of the last subject) and the rest of the cells are a 0X0 double

parent_directory = dir; %setting up the parent file to access files
rt = cell(size(parent_directory)); %an empty cell to insert all reaction times
for i = 1:length(parent_directory)
    if ~isempty(strfind(parent_directory(i).name,'img')) %find folders with img in it
        foldername = [parent_directory(i).folder '/' parent_directory(i).name]; %create a new path
        cd(foldername); %loop through the paths
        if isempty(dir('stopchoicert.mat'))
            continue; %if folder does not contain stopchoicert.mat then skip
        end
        load ('stopchoicert.mat')
        rt{i,1}= a %a contains the mean reaction time which i want to store in cell array
    end %loop through all folders to extract all mean reaction times and store into cell array
end 
rt

please help and lmk if you need any more info

Answer 1

[Stephen23]

Much simpler and much more robust:

D = 'path to the main directory';
S = dir(fullfile(D,'img*'));
for k = 1:numel(S)
    F = fullfile(D,S(k).name,'stopchoicert.mat');
    T = load(F);
    S(k).data = T.a;
end
C = {S.data};

If not all of the subfolders contain the file, then you could do something like this:

for k = 1:numel(S)
    F = fullfile(D,S(k).name,'stopchoicert.mat');
    if exist(F,'file')==2
        T = load(F);
        S(k).data = T.a;
    end
end

[zexi Leong]

Hi there! thank you for your help! Sorry but could you explain to me what line 3 in your code means? why would it need to be ==7? what does the second input file mean?

Lastly, I am not too sure how to create the field 'data'? when i run the code, there is an error of referencing a non-existent field 'data'

Thank you for your help!!

[zexi Leong]

This is my edited code based on your suggestion but i will get the error of Reference to non-existent field 'data'

D = '/Users/jessieleong/Documents/MATLAB/crtgeoff_analysis copy 2';
S = dir(fullfile(D,'img*'));
for k = 1:numel(S)
    F = fullfile(D,S(k).name,'stopchoicert.mat');
    if exist(F,'file')==7
        T = load(F);
        if ~exist(F,'file')
            continue
        S(k).data = T.a;
        end
    end
end
C = {S.data};
C

[Stephen23]

"could you explain to me what line 3 in your code means? why would it need to be ==7? what does the second input file mean? "

According to its documentation, the exist function will check for the "existence of variable, script, function, folder, or class". By default it searches for all of these, but we can help it by specifying the second optional argument: in this case 'file' tells exist to only check for files. The output from exist also depends on what it finds (i.e. variable, script, etc), and when you read the documentation you will find that an output value 7 corresponds to a file.

"This is my edited code based on your suggestion but i will get the error of Reference to non-existent field 'data'"

Your apparently random changes to my code are the reason why the data is not being imported.

1. for some unknown reason you decided to add another exist call after loading the file data. Consider the two possibilities a) if the file does not exist, then trying to import its data will throw an error, so your check does nothing, or b) if the file does exist, then load has already imported its data, in which case your exist is total superfluous. That is why my code already checked if the file exists before loading its data.
2. You included the part of the code that actually stores the imported data in an if statement that runs only when the file does not exist (how do you imagine storing data from non-existent files?)... but this is all nested inside my original if statement which already checks if the file exists: can you tell me how it is ever possible for both if statements to have true conditions? i.e. for your if statement to run, a file has to both exist and not exist. Perhaps you are checking for Schroedinger's file?

As far as I can tell, you made these random illogical changes because you were under the incorrect impression that the code needed a continue statement. If it needed a continue statement, I would have used one. I recommend actually looking at my code and thinking about its logic, especially how the if statement that I used simply runs if the file exists, and skips to the next loop iteration if the file does not exist (no continue is required).

" I am not too sure how to create the field 'data'?"

By using my answer, and not adding random changes that prevent it from working properly.

[Stephen23]

zexi Leong's "Answer" moved here:

Hi sir, I am really grateful for the quick reply.

D = '/Users/jessieleong/Documents/MATLAB/crtgeoff_analysis copy 2';
S = dir(fullfile(D,'img*'));
for k = 1:numel(S)
    F = fullfile(D,S(k).name,'stopchoicert.mat');
    if exist(F,'file')==7
        T = load(F);
        S(k).data = T.a;
    end
end
C = {S.data};
C

But even before my incorrect addition of the continue statement, i occured the same error.

[Stephen23]

Is this the correct path?

D = '/Users/jessieleong/Documents/MATLAB/crtgeoff_analysis copy 2';

You could just use D = '.'; to specify the current directory, but only if those subfolders are in the current directory (which appears to be the case, judging by your screenshot).

Note that the main advantage of using an absolute path is that your code and your data can be stored separately (which they should be).

[zexi Leong]

Hi sir, sorry to disturb you yet again ):

But it still does not work even after i change the directory to '.' The error still lies in line 10

C = {S.data};

I don't think the issue lies in the directory and i believe it is the full path from what I can tell because I am using a macbook so i do not have the c drive, unless you could show me a way to get the full path on a Macbook - I would be really grateful for that!

[Stephen23]

@zexi Leong: I am not using a mac, so I cannot help with the correct syntax. You will have to check the path syntax yourself.

I am happy to help you to debug the code. Please show the output of these commands:

size(S)
exist(F,'file')

[zexi Leong]

size(S)

ans =

0 1

exist(F, 'file')

Undefined function or variable 'F'.

THANK YOU!!!

[Stephen23]

Note that S is empty, this means that dir did not find anything on the requested path that matched the requested name. This is often caused by using a wrong path (e.g. a spelling mistake), or looking in the wrong folder (e.g. one level higher than required).

So we need to find out why dir cannot find those subfolders.

Please show the output of these commands:

D = '/Users/jessieleong/Documents/MATLAB/crtgeoff_analysis copy 2';
dir(fullfile(D,'*'))
dir(fullfile('.','*'))
dir('*')

[zexi Leong]

D = '/Users/jessieleong/Documents/MATLAB/crtgeoff_analysis copy 2';
dir(fullfile(D,'*'))

.

..

.DS_Store

_DS_Store

f.m

f1.m

geoffre

img0001

img0002

img0003

img0004

img0005

img0006

img0007

img0008

img0009

img0010

img0011

img0012

img0013

img0014

img0015

img0016

img0017

img0018

img0019

img0020

img0021

img0022

img0023

img0024

img0025

img0026

img0027

img0028

img0029

img0030

img0031

img0032

img0033

img0034

img0035

img0036

img0037

img0038

img0039

img0040

img0041

img0042

img0043

img0044

img0045

img0046

img0047

img0048

img0049

img0050

img0051

img0052

img0053

img0054

img0055

img0056

img0057

img0058

img0059

img0060

img0061

img0062

img0063

img0064

img0065

img0066

img0067

img0068

img0069

img0070

img0071

img0072

img0073

img0074

img0075

img0076

img0077

img0078

img0079

img0080

img0081

img0082

img0083

img0085

img0086

img0088

img0089

img0090

img0091

img0092

img0093

img0094

img0095

img0096

img0098

img0099

img0100

img0101

img0102

img0103

img0104

img0106

img0113

img0114

img0115

img0116

img0117

img0118

img0119

img0120

img0121

img0122

img0123

img0125

img0128

img0129

img0130

img0131

img0135

img0139

img0141

img0142

img0143

img0145

img0147

img0151

img0153

img0154

img0155

img0160

img0161

img0162

img0163

img0165

img0166

img0167

img0168

img0169

img0170

img0171

img0174

img0175

img0176

img0178

img0179

img0184

img0185

img0187

img0189

img0190

img0191

img0192

img0193

img0194

img0195

img0197

img0198

img0199

img0200

img0201

img0202

img0203

img0204

img0205

img0206

img0207

img0208

img0209

img0210

img0211

img0212

img0213

img0214

img0215

img0216

img0217

img0218

img0219

img0220

img0221

img0222

img0223

img0224

img0225

img0226

img0227

img0228

img0229

img0230

img0231

img0232

img0233

img0234

img0235

img0236

img0237

img0238

img0239

img0240

img0241

img0242

img0243

img0244

img0245

img0246

img0247

img0248

img0249

img0250

img0251

img0252

img0253

img0254

img0255

img0256

img0257

img0258

img0259

img0260

img0261

img0262

img0263

img0264

img0265

img0266

img0267

img0268

img0269

img0270

img0271

img0272

img0273

img0274

img0276

img0277

img0278

img0280

img0281

img0282

img0283

img0284

img0286

img0287

img0288

img0289

img0290

img0291

img0292

img0293

img0294

img0295

img0296

img0297

img0298

img0299

img0300

img0301

img0302

img0303

img0304

img0305

img0306

img0307

img0308

img0309

img0310

img0311

img0312

img0313

img0314

img0315

img0316

img0317

img0318

img0319

img0320

img0321

img0322

img0323

img0324

img0325

img0326

img0330

img0331

img0334

img0338

img0342

img0344

img0345

img0347

img0349

img0356

img0357

img0358

img0359

img0360

img0361

img0362

img0363

img0366

img0372

img0373

img0374

img0375

img0377

img0378

img0379

img0380

img0381

img0387

img0389

img0392

img0393

img0395

ssrt.m

ssrt_181019.m

ssrt_181019verstwo.m

td3008v1.m

tdanalysis3008.m

tdanalysis3008v2.m

tdanalysisaugtwentieth.m

dir(fullfile('.','*'))

.

..

Choicert8.log

Cogent-2008-12-22-20-10-21.res

Cogent-2008-12-22-20-11-04.res

Demo_random8.log

Direction_stroop.log

IQ.mat

Position_stroop.log

Random8.log

SSRT.log

SimpleRT

TwochoiceRT_start.log

certaintyeq.mat

certaintyequivalent

choicerteight.mat

crt22measures_20090809.txt

crt2dir.log

crt2flanker.log

crt2pos.log

crtStop.log

crta22.mat

crtb22.mat

crtdirectiontwo.mat

crtflankertwo.mat

crtpositiontwo.mat

crtstarttwo.mat

crtstoptwo.mat

demochoicert8.log

directions.mat

flanker.log

flankers.mat

positiont.mat

randomgens.mat

simprt.mat

stopchoicert.mat

subject.mat

tdiscount.mat

temporaldiscount.log

temporaldiscount.res

dir('*')

.

..

Choicert8.log

Cogent-2008-12-22-20-10-21.res

Cogent-2008-12-22-20-11-04.res

Demo_random8.log

Direction_stroop.log

IQ.mat

Position_stroop.log

Random8.log

SSRT.log

SimpleRT

TwochoiceRT_start.log

certaintyeq.mat

certaintyequivalent

choicerteight.mat

crt22measures_20090809.txt

crt2dir.log

crt2flanker.log

crt2pos.log

crtStop.log

crta22.mat

crtb22.mat

crtdirectiontwo.mat

crtflankertwo.mat

crtpositiontwo.mat

crtstarttwo.mat

crtstoptwo.mat

demochoicert8.log

directions.mat

flanker.log

flankers.mat

positiont.mat

randomgens.mat

simprt.mat

stopchoicert.mat

subject.mat

tdiscount.mat

temporaldiscount.log

temporaldiscount.res

>>

[zexi Leong]

Sorry just for my understanding, the '*' that you put in is used to look at any file in that directory right?

[Stephen23]

" the '*' that you put in is used to look at any file in that directory right?"

Any file or subfolder.

Please try this:

D = '/Users/jessieleong/Documents/MATLAB/crtgeoff_analysis copy 2';
S = dir(fullfile(D,'*'));
size(S)

[zexi Leong]

ans =

328 1

[Stephen23]

@zexi Leong: why do you get different results when you run exactly the same code in your script vs. in the command window? Please run these commands in two locations:

1. at the end of your script,
2. in the command window.

And show the outputs:

pwd()
numel(dir(fullfile(D,'*')))
numel(dir(fullfile(D,'img*')))

[zexi Leong]

I managed to figure out the issue! Thank you so much for your help! Would not have gotten this far without you.

D = 'C:\Users\zexi\Downloads\crtgeoff\crtgeoff_analysis subjects data';
S = dir(fullfile(D,'img*'));
for k = 1:numel(S)
    F = fullfile(D,S(k).name,'stopchoicert.mat');
    if exist(F,'file')==2
        T = load(F);
        S(k).data = T.a;
    end
end
C = {S.data}

I could get a cell with all the values that i want already!

I have a struct labelled S with both the subject ID and the respective data - Next, I hope to be able to do my anova analysis based on this struct.

[Stephen23]

@zexi Leong: what was the issue? I notice you seem to have changed the filepath and/or OS

Remember to accept my answer if it helped you: accepting answers is an easy way to show your appreciation of the volunteers who help you on this forum.

[zexi Leong]

Yes I switched computers hence the file path is different but the issue with the code was that for the exist function, I was supposed to be looking for a file and not a folder? So i changed it from 7 to 2. This resolved all of my problems.

[Stephen23]

"So i changed it from 7 to 2. This resolved all of my problems. "

Ah, well spotted! I will change my answer too, so that it does not confuse any future readers.

Answer 2

[Divya Yerraguntla]

Hi Zexi

Try using the following modified code to serve your purpose. Have a look at the comments for a better understanding regarding the changes made.

parent_directory = dir;
num_dir = numel(parent_directory([parent_directory(:).isdir]))-2;% To find the number of sub folders on the parent directory
rt = cell(num_dir,1);
for i = 3:length(parent_directory) % The parent_directory.name field contains names of folders from the 3rd index location, hence i starts from 3
    if ~isempty(strfind(parent_directory(i).name,'img')) 
        foldername = [parent_directory(i).folder '/' parent_directory(i).name]; 
        cd(foldername); 
        if isempty(dir('stopchoicert.mat'))
            continue; 
        end
        load ('stopchoicert.mat');
        rt{i-2,1}= a; % As i starts from 3, go back by 2 indices to fill rt
    end 
end 
rt

Hope it helps!

[Stephen23]

This is factually incorrect: "The parent_directory.name field contains names of folders from the 3rd index location, hence i starts from 3"

As anyone experienced with MATLAB (or anyone who reads this forum) will know, the order of the files returned by dir is determined by the OS, and not by MATLAB. None of the OS MATLAB runs on guarantee a particular order (although apparently this author assumed that there is one).

https://www.mathworks.com/matlabcentral/answers/484430-using-struct-dir-selpath-what-do-and-mean-as-struct-name#answer_395754

The author incorrectly assumed that the first two results correspond to the folder shortcuts '.' and '..', but in fact it is trivial to define some filenames where this is not the case:

>> fclose(fopen('+test.txt','wt'));
>> fclose(fopen('-test.txt','wt'));
>> fclose(fopen('@test.txt','wt'));
>> S = dir('*');
>> S.name
ans = +test.txt
ans = .test.txt
ans = .
ans = ..
ans = @test.txt

Bad code includes

• the use of cd (slow, difficult to debug), rather than absolute/relative filenames.
• string concatenation instead of fullfile.
• loading directly into the workspace.