Audio Video dynamic sycing manipulation

Hello,

I am doing autism research at York University. Before working with Matlab, I need to know Matlab has the features I need for my experiment.

Purpose: Generally I am testing the limits of peoples' sensitivity to noticing when audio and video are out of sync when watching a movie. (with audio either ahead and delayed).

Input Functionality: I want to present audio and video to participants, with the audio out of sync by a range of values from -1000 to 1000 ms. (think of a kung fu movie in the 70's) and allow people to manipulate the delay or advance of th audio dynamically (while movie is playing) by keypress, until they feel they are in sync. Each keypress should advance or delay audio by 50ms.

Output Functionality:

As this is a multi-trial experiment, I also hope to have control randomizing multiple trials automatically, and recording results of data to an excel or text file for each participant. (Similar to E-Prime)

Desired Results include: 1. the final offset where people percieved synchrony, 2. the time it took, 3. and the pattern of key presses.

I couldn't find information about this feature or audio video capabilities on the website, so any help in this would be very much appreciated.

Regards, Mag

Answer 1

[Walter Roberson]

This would be difficult to program in regular MATLAB, as MATLAB itself was not designed for real-time work. Coordinating audio and video is known to be difficult in MATLAB.

The most highly developed work in this area that I know of is the Psychophysics Toolbox, http://psychtoolbox.org/

Psychophysics Toolbox Version 3 (PTB-3) is a free set of Matlab and GNU/Octave functions for vision research. It makes it easy to synthesize and show accurately controlled visual and auditory stimuli and interact with the observer.

Answer 2

[Daniel Shub]

I disagree with Walter on this one. This type of experiment can be easily implmented with MATLAB, the problem lies in the hardware (which you do not specify). For example if you are willing to invest money (20k+) in a hardware system designed for psychophysics, you will be fine. If you want to do it with an off the shelf soundcard and videocard, it becomes much harder... Psychtoolbox can probably meet your requirements if you are willing to stick to 32bit and Mac (which is hard to find nowadays). Psyctoolbox and Windows for audio-visual sync, is more of a crap shoot. To use psychtoolbox you would need to learn MATLAB and the psychtoolbox.

Since it sounds like you are not wedded to MATLAB, I would consider psychopy (which uses python, but acts as a stand alone application). It has many of the same difficulties with audio-visual sync (it is a hard currently unsolved problem), but is more of a direct replacement to e-prime.

Answer 3

[Maggie]

Thanks guys,

I will look into both of these options. It would be convenient if I could tap into the functionality of VLC player (video lan) which has the ability to dynamically advance or delay sound (by keying 'g' or 'h') I wonder if E-prime or Psychopy would allow this kind of extension.

Mags

[Daniel Shub]

Advancing and delaying sound is easy to do with MATLAB, psychtoolbox, or PsychoPy. If you want to drop into the VBA underbellies of E-Prime, you it can probably advance/delay sound also. The hard part is getting to zero delay and having a consistent (ideally zero) delay between the audio/video everytime you start the stimulus.

[Walter Roberson]

Daniel, could you point us to the MATLAB mechanisms you would use? I do not recall seeing synchronization method that would be accurate enough.

[Daniel Shub]

Let me clarify. I never said sync. In fact I said getting a consistent delay is hard. Advancing/delaying is easy since the timing of audio and video cards is rock solid. The problem is it is difficult to start them at the same time. In other words, if you mange to get your audio and video synced, they will stay synced, provided you do not stop, or mess with, either stream.

For example using psychtoolbox you can present a single long (say 1 hour) stimulus that contains a single white video frame and an acoustic click every second. If you route your sound card and a photometer (to measure the screen flip) to an oscilloscope there will be a delay, but it will be constant.

[Huw S]

For other people coming across the problem of audio synchronisation with matlab, I can say that not all soundcards are made equal with regards to synchronisation and that onboard PC soundcards absolutely can drift in time.

I was testing recently and had matlab outputting voltage from an NI instruments box, as well as auditory output using PTB. Using an oscilliscope we saw that over the course of around 7minutes, the audio output had drifted about 250ms from it's original synchronisation point, it looks a certain amount of samples are occasionally dropped.

The fix to this was using a more expensive soundcard that has internal clock and digital-analogue converter. I don't know the specifics on WHY this solved the problem, but on tests of up to 15 minutes, synchronisation remained constant to sub ms precision.