What do I do to make a function step that writes all frames, not just the last frame?

readObject = vision.VideoFileReader('video_001.avi', 'AudioOutputPort',true);
%% Video File Writer Objekt erstellen
writeObject = vision.VideoFileWriter('video17.avi', 'FileFormat','avi', 'AudioInputPort',true, 'FrameRate',readObject.info.VideoFrameRate, 'Quality',90);
writeObject.AudioInputPort=true;
%% Video Frame aus .avi-Objekt holen und in .mp4-Objekt schreiben

while ~isDone(readObject)
    [frame, audio] = step(readObject); %[frame, audio] = avi();
end 
  step(writeObject,frame, audio); %mp4(frame, audio);

release(readObject);
release(writeObject);

I want to read the audio and frames by using vision.VideoFileReader; then do some processing on the frames, then write frames and audio in a file, but I do not want to use the 'while' used during reading

Answer 1

[Walter Roberson]

It is not possible to achieve your goals. You have chosen to use vision.VideoFileReader() which does not have a "read all frames" function, only a "read the next frame" function. You must use a loop to read all of the frames.

Well, we could cheat on the spirit of the law but adhere to the letter of the law by recording the

while ~isDone(readObject)

as

for iteration = 1 : inf
    if isDone(readObject); break; end
    something
end

This uses a for loop instead of a while loop.

It would be quite legitimate to ask whether we could find out how many frames there were before doing the reading, and then use a for loop over that count. If we could then we would be able to pre-allocate storage for all the frames and audio together, which would be beneficial.

The answer to that question is that it turns out that the frame counts you get in video files are typically not reliable, especially for variable-frame-rate files. The frame count tends to be based upon multiplying the time the video is supposed to last, by the frame rate -- but you can't do that accurately if the frame rate is variable. It also turns out that it is common to use nominal frame rates instead of exact frame rates, such as using 30 fps when the actual rate is 29.95 fps. Frame rates are not integers either, so you tend to get rounding errors on the exact count.

So unfortunately the only reliable way to find out the exact number of rames ahead of time is to read through the file and count them.

That said, there is still quite a bit of efficiency gain to be had by estimating the number of frames. Even deliberately over-estimate, and allocate more than you need, and then use a while loop structure to loop for the actual end of frames, keeping count as you read them in. Afterwards, trim back from the (overestimated) memory to the actual count: this only involves resizing the data structure once, instead of having to re-size every time you add another frame.

[amenah mwuafaq]

Thank you for answering me Walter Roberson

This is what I want this function

 step(writeObject,frame, audio); %mp4(frame, audio);
 

work sepratly from this "while"

while ~isDone(readObject)
    [frame, audio] = step(readObject); %[frame, audio] = avi();
end 

[Walter Roberson]

filename = 'video_001.avi';
fileinfo = aviinfo(filename);
ENF = round(fileinfo.NumFrames + fileinfo.FramesPerSecond); %give another 1 second of buffer

%pre-allocate !!! Important for efficiency
frames = cell(ENF,1);
audios = cell(ENF,1);

readObject = vision.VideoFileReader('video_001.avi', 'AudioOutputPort',true);
FC = 0;
while ~isDone(readObject)
    FC = FC + 1;
    [frames{FC}, audios{FC}] = step(readObject);
end

if FC < ENF               %trim any over-estimate
    frames(FC+1:end) = [];
    audios(FC+1:end) = [];
end

Now you can process the information in cell array frames and audios however you like, and run the write operation afterwards.

This uses a while loop, which you specifically said you do not want to do. However, you need to use a while loop or equivalent to deal with the fact that you do not have an accurate frame count ahead of time.

[amenah mwuafaq]

Sorry, it seems my question was not clear to explain my problem.

My problem with the writing function is how can I write all the video and audio frames

Because the step function writes only the last frame.

What loop am I using that enables me to write all frames and audio?

[Walter Roberson]

step(writeObject,frame, audio);

The reason that only writes the last frame is that it is after your loop in which you consistently overwrote the frame and audio variable.

Your existing loop already reads all frames and audio. However it throws all except the last of them away.

The code I posted reads all the frames and audio and keeps them. You can then do whatever processing you need on the stored frames and audio, and then you can can loop writing out the modified versions. For example

NewOrder = randperm(FC) ;
cellfun(@(F, A) step(writeObject, F, A), frames(NewOrder), audios(NewOrder));

to write back in a random order.

[amenah mwuafaq]

I ran out of code you wrote, but also didn't write frames and audio in the file.

[amenah mwuafaq]

Thank you for your answer Walter Roberson

But why do I have the same problem?

Frames and audio not written to the file

I appreciate any help for me.

[Walter Roberson]

Can you make your avi file available? I am having difficulty finding an uncompressed .avi with audio, and reading audio from a compressed video is only supported on Windows.

[amenah mwuafaq]

This is the video

[Walter Roberson]

Unfortunately on my Mac, it is unable to determine the Codec for the file (which other programs tell me is a DiVX codec)

[amenah mwuafaq]

What do you advise me now?

[Walter Roberson]

The attached code will read the given video file if it can be read; if the video cannot be read then it gives up.

If audio can be read with the video, then the audio is read as well.

If audio cannot be read with the video, either because there is no audio in the file, or because the reader has trouble reading the audio (for example compressed audio can only be read on Windows systems), then fake audio (Handel) is added, to prove the point that audio can indeed be written to the file.

You had wanted to modify the frames before writing them out. This particular demonstration does the simple modification of re-ordering the frames randomly. You can do whatever modification of the frames you like: just do it in the section "%now modify the frames in any desired way".

The program emits messages about what it is doing along the way.

If someone else is reading this and wants to try the code but the video_001.avi is not readable, then you can substitute a different file name, such as 'rhinos.avi' : I have specifically tried it with rhinos.avi (for which it will generate the fake audio, since rhinos.avi has no audio.)

filename = 'video_001.avi';

filename_out = 'video17.avi';

fprintf('probing file: "%s"\n', filename);

%estimate number of frames
keep_going = true;
try
    fileinfo = VideoReader(filename);
catch ME
    keep_going = false;
    if strcmp(ME.identifier, 'MATLAB:audiovideo:VideoReader:unsupportedText')
        fprintf('VideoReader says that is a text file!\n');
    elseif strcmp(ME.identifier, 'MATLAB:audiovideo:VideoReader:UnknownCodec')
        fprintf('VideoReader says video codec not recognized\n');
    elseif strcmp(ME.identifier, 'MATLAB:audiovideo:VideoReader:Unexpected')
        fprintf('VideoReader says it cannot decode file\n');
    else
        fprintf('Videoreader reports exception message: "%s"\n', ME.identifier);
    end
end

if keep_going
    
    framerate = fileinfo.FrameRate;
    
    ENF = round(fileinfo.NumFrames + framerate); %give another 1 second of buffer
    delete(fileinfo)
    
    %pre-allocate !!! Important for efficiency
    frames = cell(ENF,1);
    audios = cell(ENF,1);
    
    fprintf('Starting to read video\n');
    
    %unfortunately if there is no audio, lastwarn will be the same as if
    %there was audio that could not be read in
    vision_no_audio_in_file = 'MATLAB:system:nonRelevantProperty';
    vision_no_compressed_audio = 'vision:VideoFileReader:compAudioNotSupported';
    vision_no_audio_compressed_video = 'vision:VideoFileReader:audioNotSupported';
    wstate = warning('off', vision_no_audio_in_file);
    warning('off', vision_no_compressed_audio);
    warning('off', vision_no_audio_compressed_video);
    
    readObject = vision.VideoFileReader(filename, 'AudioOutputPort',true);
    
    warning(wstate)
    
    video_has_audio = readObject.info.Audio;
    if ~video_has_audio
        fprintf('Audio could not be read in. Substituting fake audio\n');
        handel = load('handel.mat');
        fake_audio = handel.y;
        fake_ns = length(fake_audio);
        fake_ss = round(handel.Fs / framerate);
    end
    
    FC = 0;
    while ~isDone(readObject)
        FC = FC + 1;
        [frames{FC}, audios{FC}] = step(readObject);
        if ~video_has_audio
            st = randi(fake_ns - fake_ss + 1);
            audios{FC} = fake_audio(st : st + fake_ss - 1);
        end
    end
    
    if FC < ENF               %trim any over-estimate
        frames(FC+1:end) = [];
        audios(FC+1:end) = [];
    end
    
    release(readObject)
    
    fprintf('Finished reading video\n');
    
    %now modify the frames in any desired way
    NewOrder = randperm(FC) ;
    modified_frames = frames(NewOrder);
    modified_audio = audios(NewOrder);
    
    %now write out the video
    
    fprintf('starting writing video\n');
    
    writeObject = vision.VideoFileWriter(filename_out, 'FileFormat','avi', 'AudioInputPort',true, 'FrameRate',readObject.info.VideoFrameRate, 'Quality',90);
    cellfun(@(F, A) step(writeObject, F, double(A)), modified_frames, modified_audio);
    release(writeObject);
    
    fprintf('Finished writing video to file "%s"\n', filename_out);
end

[amenah mwuafaq]

Thank you for your answer. Walter Roberson

Unfortunately I did not reach the desired answer

But thank you again for interacting with me.

[Walter Roberson]

What message did you get when you ran my code?

If you substituted rhinos.avi for your file, was (fake) audio successfully written to the output file?

Did you still find that only the final frame was written? Did that hold true if you substituted rhinos.avi as well?

I put a couple of hours work into getting that code right, and I would like to figure out what problems you are having with it.

[amenah mwuafaq]

Mr. Walter Roberson , I thank you so much for your time and effort to help me. Your code is very good. But, it does not serve my project, because when modifying frames, it stores them in a cell array, and this will take a long time when performing mathematical operations such as (*, /,) on frames.

Can I store them in an ordinary array?

[Walter Roberson]

The * and / operations can only be used on 2D matrices. Storing all of the video for all frames in a numeric array requires a minimum of 3 dimensions (height, width, time) and 4 dimensions in your case (color).

Remember that it takes time and memory to extract a portion of a numeric array, whereas if you are doing frame-by-frame operations the cell array already has single frames ready for processing without extraction. Therefore storing in a numeric array will not necessarily be notably better.

Your sample video video_001.avi contains only 250 frames of 640 x 360 . The time difference between processing the cell array and the numeric array would probably be on the order of 1/100th of a second.

None the less, since you feel you need it, here is the revised code that stores as numeric arrays. The video is stored as a 4D array (height, width, color, time) and the audio is stored as a 3D array (samples, channels, time.)

filename = 'video_001.avi';

filename_out = 'video17.avi';

fprintf('probing file: "%s"\n', filename);

%estimate number of frames
keep_going = true;
try
    fileinfo = VideoReader(filename);
catch ME
    keep_going = false;
    if strcmp(ME.identifier, 'MATLAB:audiovideo:VideoReader:unsupportedText')
        fprintf('VideoReader says that is a text file!\n');
    elseif strcmp(ME.identifier, 'MATLAB:audiovideo:VideoReader:UnknownCodec')
        fprintf('VideoReader says video codec not recognized\n');
    elseif strcmp(ME.identifier, 'MATLAB:audiovideo:VideoReader:Unexpected')
        fprintf('VideoReader says it cannot decode file\n');
    else
        fprintf('Videoreader reports exception message: "%s"\n', ME.identifier);
    end
end

if keep_going
    
    framerate = fileinfo.FrameRate;
    
    ENF = round(fileinfo.NumFrames + framerate); %give another 1 second of buffer
    delete(fileinfo)
    
    
    fprintf('Starting to read video\n');
    
    %unfortunately if there is no audio, lastwarn will be the same as if
    %there was audio that could not be read in
    vision_no_audio_in_file = 'MATLAB:system:nonRelevantProperty';
    vision_no_compressed_audio = 'vision:VideoFileReader:compAudioNotSupported';
    vision_no_audio_compressed_video = 'vision:VideoFileReader:audioNotSupported';
    wstate = warning('off', vision_no_audio_in_file);
    warning('off', vision_no_compressed_audio);
    warning('off', vision_no_audio_compressed_video);
    
    readObject = vision.VideoFileReader(filename, 'AudioOutputPort',true);
    
    warning(wstate)
    
    video_has_audio = readObject.info.Audio;
    if ~video_has_audio
        fprintf('Audio could not be read in. Substituting fake audio\n');
        handel = load('handel.mat');
        fake_audio = handel.y;
        fake_ns = length(fake_audio);
        fake_ss = round(handel.Fs / framerate);
    end
    
    FC = 0;
    while ~isDone(readObject)
        FC = FC + 1;
        [thisframe, thisaudio] = step(readObject);
        if ~video_has_audio
            st = randi(fake_ns - fake_ss + 1);
            thisaudio = fake_audio(st : st + fake_ss - 1);
        end
        
        if FC == 1
          %pre-allocate !!! Important for efficiency
          frames = zeros(size(thisframe,1), size(thisframe,2), size(thisframe,3), ENF, class(thisframe));
          audios = zeros(size(thisaudio,1), size(thisaudio,2), ENF, class(thisaudio));
        end
        
        frames(:,:,:,FC) = thisframe;
        audios(:,:,FC) = thisaudio;

    end
    
    if FC < ENF               %trim any over-estimate
        frames(:,:,:,FC+1:end) = [];
        audios(:,:,FC+1:end) = [];
    end
    
    release(readObject)
    
    fprintf('Finished reading video\n');
    
    %now modify the frames in any desired way
    NewOrder = randperm(FC) ;
    modified_frames = frames(:,:,:,NewOrder);
    modified_audios = audios(:,:,NewOrder);
    
    %now write out the video
    
    fprintf('starting writing video\n');
    
    writeObject = vision.VideoFileWriter(filename_out, 'FileFormat','avi', 'AudioInputPort',true, 'FrameRate',readObject.info.VideoFrameRate, 'Quality',90);
    arrayfun(@(idx) step(writeObject, modified_frames(:,:,:,idx), modified_audios(:,:,idx)), 1:FC);
    release(writeObject);
    
    fprintf('Finished writing video to file "%s"\n', filename_out);
end

[amenah mwuafaq]

I have a question, please.

If I wanted to change the part about frames, how do I do that?

I want to generate a random matrix the size of a frames matrix and then multiply it in a frame matrix.

I have a question, please

 Since I wanted to change the part on FRAMES processing, how do I do that?

I want to generate a random array with the size of a frames array, and then multiply it by the frame array.

Like this

randomMatrix=randi(FC);
resulr=frames*randomMatrix

[Walter Roberson]

The frames matrix is a 4D matrix. You cannot use the * operator with a 4D matrix, only with scalars or vectors or 2D arrays.

The number of rows and columns of the frames array are not equal to each other. If you were to ask that each pane of frames were to be handled independently, then you could potentially define your question in terms of a number of operations on 2D arrays. However, when you have two 2D arrays of the same size, A and B, the only time that A*B is valid is if size(A,2) == size(B,1) and that condition can never be met with two arrays the same size when the number of rows is different from the number of columns.

If would suggest that you reconsider using the * operator. It would make more sense to do something like,

resulr = frames .* randi(FC, size(frames));

[amenah mwuafaq]

Mr. Walter Roberson , Thank you for your guidance and support.