(Conversion) synchronize audio-synchronize audio to video

FFmpeg Document 6

(18:44:22)

Reprinted

Tags: Miscellaneous

Category: translation documents

Tutorial 6: synchronize audio

 

Synchronize audio

 

Now we have a decent player. So let's take a look at what other parts are not handled. Last time, we concealed a bit of synchronization problem, that is, synchronizing audio to a video instead of other methods in the same step. We will adopt the same method as video: Make an internal video clock to record how long the video thread has played, and then synchronize the audio to the above. Later, we will also look at how to widely synchronize audio and video to an external clock.

 

Generate a video clock

 

Now we need to generate a video clock similar to our last sound clock: an internal value that gives the current video playback time. At the beginning, you may think this is as simple as updating the timer using the time stamp of the previous frame. However, do not forget that the interval between video frames is long, measured in milliseconds. The solution is to track another value: the time value when we set the previous timestamp. Therefore, the current video time value is pts_of_last_frame + (current_time-time_elapsed_since_pts_value_was_set ). This solution is similar to the get_audio_clock function.

In our large struct, we will put a double precision Floating Point variable video_current_pts and a 64-Bit Width integer variable video_current_pts_time. The clock update will be placed in the video_refresh_timer function.

Void video_refresh_timer (void * userdata ){       If (is-> video_st ){ If (is-> pictq_size = 0 ){ Schedule_refresh (is, 1 ); } Else { Vp = & is-> pictq [is-> pictq_rindex];   Is-> video_current_pts = VP-> PTS; Is-> video_current_pts_time = av_gettime ();

Do not forget to initialize it in the stream_component_open function:

Is-> video_current_pts_time = av_gettime ();

Now we need a way to get information:

Double get_video_clock (videostate * Is ){ Double Delta;   Delta = (av_gettime ()-is-> video_current_pts_time)/1000000.0; Return is-> video_current_pts + delta; }

 

Extract clock

 

But why do we need to force the video clock? We changed the video synchronization code so that the audio and video won't try to synchronize with each other. Imagine that it has a command line parameter like ffplay. So let's abstract the same thing: we will create a new encapsulation function get_master_clock to detect the av_sync_type variable and decide whether to call get_audio_clock, get_video_clock, or other functions to obtain the clock. We can even use the computer clock. This function is called get_external_clock:

Enum { Av_sync_audio_master, Av_sync_video_master, Av_sync_external_master, };   # Define default_av_sync_type av_sync_video_master   Double get_master_clock (videostate * Is ){ If (is-> av_sync_type = av_sync_video_master ){ Return get_video_clock (is ); } Else if (is-> av_sync_type = av_sync_audio_master ){ Return get_audio_clock (is ); } Else { Return get_external_clock (is ); } } Main (){ ... Is-> av_sync_type = default_av_sync_type; ... }

 

Synchronize audio

 

Now is the most difficult part: synchronizing audio to video clock. Our strategy is to measure the sound position, compare it with the video time, and then calculate the number of samples to be corrected, that is: do we need to accelerate playback by dropping samples or by interpolation samples?

We will run a synchronize_audio function each time we process a sound sample to correctly contract or expand the sound sample. However, we do not want to synchronize audio every time we find any deviation, because this will make the synchronized Audio more than the video package. Therefore, we set a minimum continuous value for the synchronize_audio function to limit the synchronization time, so that we will not always adjust it. Of course, just like the last time, "out of sync" means that the difference between the sound clock and the video clock is greater than our threshold.

So we will use a score coefficient called C, so now we can say that we have obtained N lost audio samples. The number of lost synchronization may change a lot, so we need to calculate the mean value of the loss of synchronization duration. For example, during the first call, it is displayed that the length of the lost synchronization is 40 ms, and the next call is 50 ms. However, we will not use a simple mean, because the nearest value is more important than the previous one. So we will use a score system called C, and then use this formula to calculate the difference: diff_sum = new_diff + diff_sum * C. When we are ready to find the average difference, we use a simple calculation method: avg_diff = diff_sum * (1-C ).

Note: Why is it here? This formula seems amazing! Well, it is basically a weighted average using an equi-ratio series. I don't know if there is a name (I even checked Wikipedia !), But if you want more information, here is an explanation.

Below are our functions:

Int synchronize_audio (videostate * is, short * samples, Int samples_size, double PTS ){ Int N; Double ref_clock;   N = 2 * is-> audio_st-> codec-> channels;   If (is-> av_sync_type! = Av_sync_audio_master ){ Double diff, avg_diff; Int wanted_size, min_size, max_size, nb_samples; Ref_clock = get_master_clock (is ); Diff = get_audio_clock (is)-ref_clock;   If (diff <av_nosync_threshold ){ // Accumulate the diffs Is-> audio_diff_cum = diff + is-> audio_diff_avg_coef * Is-> audio_diff_cum; If (is-> audio_diff_avg_count <audio_diff_avg_nb ){ Is-> audio_diff_avg_count ++; } Else { Avg_diff = is-> audio_diff_cum * (1.0-is-> audio_diff_avg_coef );     } } Else { Is-> audio_diff_avg_count = 0; Is-> audio_diff_cum = 0; } } Return samples_size; }

Now we have done a good job. We already know how to adjust the audio using videos or other clocks. So let's calculate the number of samples to be added and removed, and write the code in the "shrinking/expanding buffer code" section:

If (FABS (avg_diff) >=is-> audio_diff_threshold ){ Wanted_size = samples_size + (INT) (diff * is-> audio_st-> codec-> sample_rate) * n ); Min_size = samples_size * (100-sample_correction_percent_max) /100 ); Max_size = samples_size * (100 + sample_correction_percent_max) /100 ); If (wanted_size <min_size ){ Wanted_size = min_size; } Else if (wanted_size> max_size ){ Wanted_size = max_size; }

Remember that audio_length * (sample_rate * # of channels * 2) is the number of audio samples in audio_length seconds. Therefore, the number of samples we want is the number of audio samples we want to add or reduce based on the sound offset. We can also set a range to limit the length of a correction, because if we change too much, users will hear harsh voices.

 

Corrected sample count

 

now we need to correct the sound. You may notice that our synchronization function synchronize_audio returns a sample number, which tells us how many bytes are sent to the stream. So we only need to adjust the number of samples to wanted_size. This will make the sample smaller. But if we want to make it bigger, we cannot just make the sample size bigger, because there is no extra data in the buffer zone! So we must add it. But how can we add it? The most stupid way is to try to calculate the sound, so let's add the final sample at the end of the buffer with the existing data.

If (wanted_size <samples_size ){   Samples_size = wanted_size; } Else if (wanted_size> samples_size ){ Uint8_t * samples_end, * q; Int NB;     NB = (samples_size-wanted_size ); Samples_end = (uint8_t *) samples + samples_size-N; Q = samples_end + N; While (NB> 0 ){ Memcpy (Q, samples_end, N ); Q + = N; Nb-= N; } Samples_size = wanted_size; }

Now we use this function to return the number of samples. What we need to do now is to use it:

Void audio_callback (void * userdata, uint8 * stream, int Len ){   Videostate * Is = (videostate *) userdata; Int len1, audio_size; Double PTS;   While (LEN> 0 ){ If (is-> audio_buf_index >=is-> audio_buf_size ){ Audio_size = audio_decode_frame (is, is-> audio_buf, sizeof (is-> audio_buf), & PTS ); If (audio_size <0 ){ Is-& gt; audio_buf_size = 1024; Memset (is-> audio_buf, 0, is-> audio_buf_size ); } Else { Audio_size = synchronize_audio (is, (int16_t *) is-> audio_buf, Audio_size, PTS ); Is-> audio_buf_size = audio_size;

We need to insert the synchronize_audio function. (At the same time, make sure to check the code when initializing the above variables, which I did not go into details ).

Last thing before the end: we need to add an if statement to ensure that the video will not be synchronized during the video master clock.

If (is-> av_sync_type! = Av_sync_video_master ){ Ref_clock = get_master_clock (is ); Diff = VP-> PTS-ref_clock;     Sync_threshold = (delay> av_sync_threshold )? Delay: Av_sync_threshold; If (FABS (diff) <av_nosync_threshold ){ If (diff <=-sync_threshold ){ Delay = 0; } Else if (diff> = sync_threshold ){ Delay = 2 * delay; } } }

You can add it. Make sure that all the variables I have not described in detail in the entire program have been initialized. Then compile it:

Gcc-O tutorial06 tutorial06.c-lavutil-lavformat-lavcodec-LZ-LM 'sdl-config -- cflags -- libs'

Then you can run it.

Next time, we want to make the movie fast-forward and fast-forward.

(Conversion) synchronize audio-synchronize audio to video