How to read 21-bit binary data stored as text?

I have a binary sequence stored as a text file. (See text file attached.) I have tried reading the file using csvread and fscanf.

Every time it wants to fill a column of data with scientific notation, then something gets lost in the translation back to binary then to decimal.

I keep getting twos and sometimes nines in my binary string and the bin2dec is not happy with twos and nines. It only knows how to read zeroes and ones.

I tried reading as %d and %ld, but the numbers I get are not even close to matching the binary values. I also tried to read as a string, but that reads in a continuous stream of characters with no break. I think it's looking for a space to break on. I have no spaces in the data, but there are likely carriage returns or linefeeds or both at the end of each line. Can fscanf be made to recognize carriage returns or linefeeds?

csvread example

%read 21-bit data using csvread
filename = '21bitsequence.csv';
raw_data = csvread(filename);
temp1 = num2str(raw_data,'%.0f');
temp2 = bin2dec(temp1);

fscanf example:

%read 21-bit data using fscanf
filename = '21bitsequence.csv';
fileID = fopen(filename, 'r');
formatSpec = '%f';
raw_data = fscanf(fileID, formatSpec);
temp1 = num2str(raw_data, '%.0f');
temp2 = bin2dec(temp1);
fclose(fileID);

Here's what I see in temp1:

val =

111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110000000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110020000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110100000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111110120000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111000000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111020000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111100000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000
111111111111111110000

Follow up question... What if I need to treat the binary as signed binary?

[CAROL GARRITTY]

I'm sorry, but the signed part is not working properly. If I create a binary sequence from -128 to +128 (see attached) represented as signed binary (sign bit extended to all upper bits), the sequence does fine from -128 to -1 then it flips to +1048576 which is 2^20.

[CAROL GARRITTY]

Possible solution is to scale the data?

% read 21-bit signed binary data
temp = regexp(fileread('21bitsequence128.txt'), '\r?\n', 'split');
if isempty(temp{end}); temp(end) = []; end         %empty after final \n
nums = bin2dec(temp);

signed_nums = nums;
num = length(signed_nums);
for i = 1:num 
    if (signed_nums(i) > 2^20-1);
        signed_nums(i) = signed_nums(i) - 2^21;
    end
end

Why doesn't this thing paste properly?

[Walter Roberson]

Did you notice that I made a correction to the original code two days ago? "I just noticed and corrected a mistake in the positive case; please update any copy you have made."

Answer 1

[Walter Roberson]

    temp = regexp( fileread('21bitsequence.txt'), '\r?\n', 'split');
    if isempty(temp{end}); temp(end) = []; end         %empty after final \n
    nums = bin2dec(temp);

    nums_signed = typecast(uint32(bin2dec(regexprep(temp, {'^1', '^0'}, {'111111111111', '000000000000'}))),'int32');

[Marcus Lundmark]

Works great. Thank you. Though I'm still puzzled about how the signed version works.

[Walter Roberson]

For the signed conversion, you need to assume that the first bit should be extended "left" to fill 32 bits. Once you have 32 bits you can convert to decimal double, then convert the double to uint32. Then you can typecast() the uint32 to int32 to do the conversion.

To do the sign extension, I do a pattern search for leading 1 and replace it with twelve 1's, which is 11 new 1's together with the 1 that I am substituting for; the pattern search for leading 0 is replace with twelve 0's, which is 11 new 0's together with the 0 that I am substituting for.

I just noticed and corrected a mistake in the positive case; please update any copy you have made.