View a YUYV encoded image in MATLAB

Hi all,

I am kinda new to handling YUYV encoded image in matlab. I used a sensor to record data and generate a *.bag file. The bad file contains color, depth and infrared channel. Using basic command I am able to read depth and infrared channels but the colour images are encoded with YUYV format.

The data screenshot below is also a row vector whereas most of the resources requires to be fed a (image.height, image.width,3) variable, for conversion to RGB image.

How should I proceed to view this image in matlab?

Answer 1

[DGM]

Here's my guess, assuming that everything is as simple as it appears. There's a good chance this isn't exactly right.

If there are issues with the way the data is written (striped/tiled), it's easiest to try to suss out the luma data first. This is a convenience because luma alone is a viewable image. If you can find luma, you can use that info to figure out where Cb,Cr are.

% i'm assuming that the stream has been read as a uint8 vector
yuyv = zeros(614400,1,'uint8'); % fake data

% this can probably be read programmatically
sz = [480 640];

% separate channels from stream
yuyv = reshape(yuyv,[],2); % rearrange [luma chroma]
Y = yuyv(:,1); % every other byte is luma
Cb = yuyv(1:2:end,2); % chroma elements alternate
Cr = yuyv(2:2:end,2);

% reshape into 2D
Y = reshape(Y,[],sz(1)).';
Cb = reshape(Cb,[],sz(1)).'; % half horizontal resolution
Cr = reshape(Cr,[],sz(1)).'; % half horizontal resolution

% deal with subsampled chroma
% i'm just going to replicate
Cb = imresize(Cb,sz,'nearest');
Cr = imresize(Cr,sz,'nearest');

% concatenate and convert
RGB = ycbcr2rgb(cat(3,Y,Cb,Cr));

[Chhayank Srivastava]

Thanks you,

I got the hint from your previous response, your code with my dataset generates below image(maybe a peculiarity with my data) but your approach/hint certainly helped.

I read from the below sources and found them to be quite helpful (incase someone else reading it wants to read for themselves)

https://learn.microsoft.com/en-us/windows/win32/medfound/recommended-8-bit-yuv-formats-for-video-rendering

http://kctd-cctv.com/knowledgeencyclopedia/24-477.html

and made my own version of this.

https://github.com/chhayank-sri/Bag_data_extraction.git

Thanks once again for the hint.

Answer 2

[Aniketh]

Hi Chhayank,

To convert a yuv encoded image to rgb you can first

1. Reshape the YUYV data into a matrix with dimensions [height, width*2]. Each pixel in the YUYV format consists of two bytes, representing Y (luma) and U/V (chroma) values.
2. Extract the Y (luma) and U/V (chroma) values from the reshaped matrix.

then use this function:

function images=yuv2rgb(YUV)
Y = double(YUV(:,:,1));
U = double(YUV(:,:,2));
V = double(YUV(:,:,3));
% Conversion Formula
R =uint8( 1 * Y + 0  * U + 1.4022* V);
G = uint8(1 * Y  -0.3456  * U  -0.7145* V);
B= uint8(1 * Y + 1.7710  * U + 0 * V );
image=cat(3,uint8(R),uint8(G),uint8(B));
images=ycbcr2rgb(YUV);
end

[Walter Roberson]

The original data is not a 3d YUV array with at least 3 panes. The original data is a vector that represents YUYV data.

looking at the size of the vector 614400 and dividing by (640 * 480) leads us to the suspicion that each YUYV pixel occupies 2 bytes of data. That hints at 4 bits per field and a need to demosaic

[Walter Roberson]

I find a Mathworks page that says YUYV is also known as YCbCr 4:2:2

Simulink decoding: https://www.mathworks.com/help/supportpkg/xilinxsoc/ref/hdmitx.html

[Aniketh]

The last line in the function I provided, did use ycbcr2rgb for the conversion, forgive me if that's wrong, but doesn't that do the same thing or is it different?

[Chhayank Srivastava]

Hi @Aniketh ,

Thank you for your response.

How do I go about point 2. From [height,width*2] image reshaped matrix, how do I get Y, U and V channel?

[Chhayank Srivastava]

@Walter Roberson Yes you are correct. There are 16 bits per pixel or 2 bytes per pixel. There is also 1280 stride in bytes (as shown in the step size). How can I use this information to convert this 1d array to 3d array with YUV channel?

[DGM]

The given code attempts to do the conversion twice, neither of which will work directly.

• Directly applying the inverse transformation for YCbCr alone won't work without accounting for the chroma offsets.
• ycbcr2rgb() expects its inputs to be a MxNx3 image or a Mx3 color table. That's not the case until the data is split/replicated/reshaped/concatenated (which Aniketh did suggest).

In order to do all that reshaping, it's necessary to know for certain how the data is stored. At this point, we have three options:

1. find some sort of documentation which specifies how the file is encoded
2. work with a sample image to verify the encoding details
3. keep guessing

If 1 or 2 are desirable, either attach a sample image or include some information about whether the originating device is documented. I don't know about Walter or Aniketh, but I don't have the Robotics Toolbox, so if the raw data can be exported as a .mat file, that would make it available to people who can't open the .bag file.

As for #3, my guess is included below.