Gstreamer multi-channel stream synchronization

Gstreamer (translation self-part-synchronisation.txt)

This document describes the overview of gstreamer multi-channel stream synchronization technology.
The following three components are used for the synchronization of gstpipeline:
1) gststclock, which is global for all elements in gstpipeline.
2) timestamps of the ststbuffer.
3) new_segment before the buffer (Data Stream) is sent.

A gstclock
~~~~~~~~~~
The unit of the ststclock is nanoseconds. A counter describes the current time. This value is also the absolute time.
The following sources provide time counters:
1) system time, accurate to a subtle level, obtained using the g_get_current_time function.
2) audio devices (based on sampling rate ).
3) A network source, such as a RTP source, with a timestamp in the RTP package.
......
In gstreamer, The stststclock object provided by any element can be used in pipeline. The gstpipeline object selects one from the clock provider and distributes it to all other elements.
The stststclock always increments upwards and does not have to start from 0.

Running time
~~~~~~~~~~~~
When pipeline selects a clock, it will maintain running_time Based on the selected clock, running_time indicates the total time in the playing state, calculated as follows:
1) if the pipeline status is null/ready, running_time is undefined.
2) In the paused status, running_time is kept at the pause time. If pipeline is paused for the first time, running_time is 0.
3) In the playing status, running_time is the difference between absolute_time and base_time. When playing is in the state, base_time = absolute_time-running_time.
4) When flushing seek is executed, running_time is set to 0, and base_time is reset, which is equivalent to restarting.
When pipeline is in the playing-> paused state, it stores running_time. When it returns to the playing state, it starts computing running_time again. When pipeline is in the paused state, the calculation of running_time is not affected no matter whether the selected clock continues to increase steadily or not.
The clock and pipeline use running_time to synchronize all elements. In the playing state, running_time can be observed in every element. The formula is as follows:
C. running_time = absolute_time-base_time
C. running_time, that is, running_time. The value increases monotonically at the clock rate.

Timestamps
~~~~~~~~~~
The ststbuffer timestamps and the new_segment event before the buffer (Data Stream) define a conversion from buffer timestamps to running_time. The calculation is as follows:
Definition:
B. running_time: running_time
B: gstbuffer
-B. timestamp: Buffer timestamp (maid)
NS: newsegment event
-Ns. Start: Start field in the newsegment event
-Ns. Stop: Stop field in the newsegment event
-Ns. Rate: Rate Field of newsegment event
-Ns. abs_rate: absolute value of Rate Field of newsegment event
-Ns. Time: Time field in the newsegment event
-Ns. accum: total accumulated time of all previous newsegment events. This field is kept in the ststsegment structure.
The ststbuffer instance that satisfies B. timestamp> = NS. Start & B. timestamp <= NS. Stop is valid and other buffers outside the specified range are discarded.
B. The conversion formula from timestamp to B. running_time is as follows:
If (NS. rate> 0.0)
B. running_time = (B. timestamp-ns. Start)/ns. abs_rate + NS. accum
Else
B. running_time = (NS. Stop-B. timestamp)/ns. abs_rate + NS. accum
Because B. running_time is running_time, running_time comes from the newsegment event and buffer.
Therefore, the running_time calculated based on the first effective buffer is 0, since B. timestamp = NS. Start and NS. accum = 0.

Synchronisation
~~~~~~~~~~~~~~~
We can use the following method to get running_time:
1) Use the selected clock and the base_time of the element:
C. running_time = absolute_time-base_time
2) use the buffer Timestamp and the previous newsegment event (assuming that the playback speed is positive ):
B. running_time = (B. timestamp-ns. Start)/ns. abs_rate + NS. accum
The prefix C and B of running_time indicate two different calculation methods.
Synchronous playback is to ensure that when playing a buffer with a timestamp of B. running_time, the clock must reach C. running_time, that is, when:
B. running_time = C. running_time,
The data in the buffer. Therefore, it can be extended to the following formula:
B. running_time = absolute_time-base_time
Or
Absolute_time = B. running_time + base_time
Note that B. sync_time is the absolute time when the buffer should be played. Therefore, the following result is obtained:
B. sync_time = B. running_time + base_time
For multiple streams, buffers with the same running_time will be played simultaneously. The demuxer element sends a newsegment message on all its SRC pads to ensure that the synchronized buffer has the same timestamp.

Stream time
~~~~~~~~~~~
Stream time is the stream position. The value is between 0 and the length of the media file (unit: Time ). It has the following functions:
1) Position Query
2) Set position in the seek event
3) The position used to synchronize controller values
Stream time can be calculated as follows. The derivation process is as follows:

C. running_time = absolute_time-base_time
B. running_time = (B. timestamp-ns. Start)/ns. abs_rate + NS. accum
=>
(B. timestamp-ns. Start)/ns. abs_rate + NS. accum = absolute_time-base_time;
=>
(B. timestamp-ns. Start)/ns. abs_rate = absolute_time-base_time-ns. accum;
=>
(B. timestamp-ns. Start) = (absolute_time-base_time-ns. accum) * NS. abs_rate
Because
Stream_time = (B. timestamp-ns. Start) * NS. abs_applied_rate + NS. Time
=>
Stream_time = (absolute_time-base_time-ns. accum) * NS. abs_rate * NS. abs_applied_rate + NS. Time
The final formula is often used to query the current playback position in the sink element.

It should be noted that stream time has never been involved in clock synchronization of multiple streams.