Weird spaces undetected by strfind

I am trying to locate a phrase called "Faceresponse.acc" in a large array. I used this code to get all of the text out of my .txt file:

fid = fopen('HWFC_Car_P New-1-1.txt', 'r');

for i = 1:6600

    mystuff{i} = fgetl(fid);

end

fclose('all');

%The text in the text file looks like "Faceresponse.acc" but for some reason when I look at some of the strings in 'mystuff', it's spaced out: 'F a c e r e s p o n s e'. So I found this phrase in mystuff{977}, made sure it was class "char", and did this:

b = mystuff{977} strfind(b, 'F a c e')

It returned nothing. I tried just typing out 'F a c e r e s p o n s e' and assigning it to 'b', instead of using mystuff{977}, and strfind had no trouble locating the spaced out 'F a c e'. I also tried strfind for other things in the string, and it was fine. But it would NOT index the spaces between the letters.

So my question: what's going on here? What are the spaces between the letters if strfind is not identifying them as spaces?

[José-Luis]

What do double('F a c e r e s p o n s e'), or double(myStuff{977}), return?

[Ryan Egan]

double(mystuff{977})

Columns 1 through 15

     0     9     0     9     0    70     0    97     0    99     0   101     0    82     0

Columns 16 through 30

   101     0   115     0   112     0   111     0   110     0   115     0   101     0    46

Columns 31 through 43

     0    65     0    67     0    67     0    58     0    32     0    49     0

%It seems that the spaces are zeros.

[Chris]

Could it be a font issue. I had a similiar thing whe using fgetl to read a .doc. At a certain point there was a 'delta' symbol using wingdings, after that line each character had a space after it. I got rid of the odd font and that fixed it.

Answer 1

[Image Analyst]

It's probably written in some double byte unicode font, which are designed to handle every language in the whole world, so it needs two bytes instead of just one like Western ASCII style. I'm not really sure how to deal with unicodes since I don't encounter them. Maybe you can just do something like this:

for i = 1:6600
    thisLine = fgetl(fid);
    mystuff{i} = thisLine(2:2:end); % Take just every other byte.
end

[Ryan Egan]

This works nicely and avoids the whole problem. Thanks!

Answer 2

[José-Luis]

This sounds like an encoding problem. To find out what encoding your installation of Matlab uses:

   feature('DefaultCharacterSet')

To see the encoding of your file, you could do like this, but the best bet would probably be to ask the author of the program that generated the text file.

If the encodings are different, then you could try changing the encoding of Matlab to the one of the file, e.g.:

    feature('DefaultCharacterSet', 'UTF8')

But as Matt says, it is probably easier to pass addtional arguments to fopen()

Finally, here's an interesting read regarding Matlab and encoding.

[Walter Roberson]

The article is a fairly reasonable summary. It does make a little mistake right near the end where it says that char() takes unicode values; the mistake is that char() only takes unicode values up to 65535 and does not provide any mechanism for codepoints above that. There are two unicode related routines that can be used to deal with code points above that or to deal with "code pages" or the like.

[José-Luis]

Good to know. Thank you.

Answer 3

[Matt J]

FOPEN let's you specify different encoding schemes for reading from the file. I'm guessing that you might need a different one from what you're using. Was the file created on the same platform as you're now using to read it?

[Ryan Egan]

No, the text file was an output from E-prime, which is an stimuli presentation and response-logging program that psychologists use for experiments.

If you look at the comment I made above, however, it looks like when I do double(mystuff{977}), the spaces are converted to zeros. Does that help?

[Matt J]

It just confirms what we suspected already, I think. That the spaces are not really spaces:

>> double(' ')

ans =

      32

Have you tried opening the file in Notepad or something and resaving it (maybe using "Save As" instead of just "Save")?

[Ryan Egan]

Well, I could potentially need to run data analyses on hundreds of files like this over the next little while, and my lab members may need it for years to come, so I'm trying to write a script that will last the longest and doesn't require any reformatting.

I already have a script that can run the analyses when I export the file into a different format, but that can take a while depending on the sheer number of files I have to analyze.

[Matt J]

Yes, but does resaving the file as I described work? Let's first identify at least one solution to the problem, and then worry about optimizing it later. Also, did you follow Jose-Luis' suggestion about finding the encoding that your file uses?