[summary] Video and audio codec technology 0 Basic Learning methods

Did not find the reprint function where ... So I can only copy it over.

Always want to video audio codec technology to do a simple summary, but suffer from the time is not sufficient, has not been able to complete. Today has a lot of free, and finally can summarize a video and audio technology to start a tutorial, can facilitate more people learn from the zero to learn audio and video technology. It should be noted that this article refers to the visual and audio technology, referring to the theoretical level of video and audio technology, does not involve programming-related things.

0. Video and audio technology in life

Usually we turn on the computer to save their own list of movies, usually as shown in the following picture, a lot of all sorts of movies. (In fact, professional film and television enthusiasts will be the film and television files categorized, but I am lazy, a brain to put the film together)

Because of the different sources of downloads, these movie files have different formats, expressed in different suffixes: avi,rmvb,mp4,flv,mkv, and so on (with different icons, of course). Note here that these formats represent the encapsulation format. What is the encapsulation format. is the specification of packaging video data and audio data into a single file. Only by looking at the suffix of the file, it is difficult to see the specific use of the visual and audio coding standards. Generally speaking, the difference between the different packaging formats is small, each has its pros and cons.

Note: Some package formats support a wide range of video and audio coding standards, should be considered a relatively good package format, such as MKV, and some packaging format to support the visual audio coding standards are very few, should fall behind the package format, such as RMVB.

Here's how to use the video player to view the audio and video technology used in a file:

Note: Using media Player Classic as a demo here, in fact, the Storm audio and video, KMPlayer these operations are not too much.

1. Open a video file

2. Right-click on the video screen to select Properties.

3. In the Pop-up Properties dialog box, on the MediaInfo tab, you can view the video audio technology used in this file. As shown in the following illustration, the "Transformers Trailer _h1080p.mov" uses the MPEG4 QuickTime Encapsulation Format (MOV) and uses the H.264 (AVC) compression coding standard.

Here is a tool for viewing media information: MediaInfo. MediaInfo is a dedicated view audio format tool, the detailed use of software can refer to:

Introduction to MediaInfo Use (new version supports HEVC)

Source code analysis can refer to:

MediaInfo Source Code Analysis 1: Overall structure [series of articles]

Video and audio technology in life so far, the following describes the specific video and audio technology.

1. Video Player principle

Video and audio technology mainly includes the following: packaging technology, Compression coding technology, as well as audio compression coding technology. Streaming media protocol technology is also included if the network transmission is considered.

The source code detailed resolution of the video player (Media player CLASSIC-HC,MPLAYER,FFPLAY,XBMC) can refer to a series of articles:

Media player Classic:media player CLASSIC-HC source Code Analysis 1: Overall structure [series of articles]

Mplayer:mplayer Source Code Analysis

Ffplay:ffplay Source Code Analysis: Overall flow chart

XBMC:XBMC Source Code Analysis 1: Overall structure and compilation method [series articles]

Not to elaborate here, just briefly explain the principle of video player.

Video player to play a video file on the internet, need to go through the following steps: Solution protocol, solution package, decoding video and audio, video and audio synchronization. If you play local files, you do not need to solve the protocol, for the following steps: Solution encapsulation, decoding video and audio, video and audio synchronization. Their process is shown in the picture.

The function of the protocol is to parse the data of the streaming media protocol into the standard corresponding encapsulation format data. When audio and video are spread on the web, various streaming media protocols such as HTTP,RTMP, MMS, etc. are often used. These protocols also transmit some signaling data while transmitting video and audio data. These signaling data include control of playback (playback, pause, stop), or a description of the status of the network. In the process of solving the protocol, the signaling data is removed and only the audio and video data are preserved. For example, data transferred using the RTMP protocol, after the protocol operation, outputs the FLV format data.

The function of the solution package is to separate the data of the encapsulated format into the audio stream compression coded data and the video stream compression coding data. A variety of packaging formats, such as Mp4,mkv,rmvb,ts,flv,avi, and so on, its role is to compress the encoded video data and audio data in a certain format together. For example, data in FLV format, after the solution encapsulation operation, output H.264 encoded video stream and AAC encoded audio stream.

The function of decoding is to encode the video/audio compression data and decode it into uncompressed video/audio raw data. Audio compression coding standards include aac,mp3,ac-3, and so on, video coding standards include h.264,mpeg2,vc-1 and so on. Decoding is the most important and complex part of the whole system. By decoding, compressed-encoded video data output becomes uncompressed color data such as YUV420P,RGB and so on, and compressed-encoded audio data output becomes uncompressed audio sampling data, such as PCM data.

The role of video and audio synchronization is based on the solution of the packaging module to obtain the parameters of the information, synchronized decoding of video and audio data, and video audio data to the system's video card and sound card playback out.

In the next few sections we will list the main streaming media protocols, the encapsulation format, and the video and audio coding standards. More detailed comparisons can refer to:

Video parameters (Streaming media system, package format, Video coding, audio coding, player) contrast

The analysis of protocol data, encapsulated format data, Video coding data, audio coded data, video pixel data and audio sampling data in this article can be referred to the following series:

Introduction to video and audio data processing: RGB, YUV pixel data processing

Introduction to video and audio data processing: PCM Audio sampling data processing

Introduction to visual and audio data processing: H.264 video Code flow analysis

Introduction to video and audio data processing: AAC Audio Code Flow Analysis

Introduction to visual and audio data processing: FLV Encapsulation Format parsing

Introduction to video and audio data processing: UDP-RTP Protocol resolution
2. Streaming Media Protocol

The streaming media protocol is the stipulation that the communication between the server and the client follows. The main streaming media protocols on the current network are shown in the table.

Overview of major streaming media protocols

Internet on-demand

Name	Launch mechanism	Transport Layer protocol	Client	Currently uses the domain
RTSP+RTP	IETF	tcp+udp	VLC, WMP	IPTV
RTMP	Adobe Inc	TCP	Flash	Internet live
RTMFP	Adobe Inc.	UDP	Flash	Internet live
MMS	Microsoft Inc.	Tcp/udp	WMP	Internet Live + on-demand
HTTP	www+ietf	TCP	Flash

RTSP+RTP is often used in the area of IPTV. Because it uses UDP transmission video audio, support multicast, high efficiency. But its disadvantage is that the network is bad circumstances may lose packets, affect the quality of video viewing. Therefore, the research about IPTV video quality is quite many.

RTSP specification can refer to: RTSP Agreement Learning Notes

Measure the quality of service in the RTSP+RTP system: Quality of Service (QoS) of network video transmission

Shanghai IPTV Code Flow Analysis results can be referred to: IPTV video code Flow analysis

Because of the instability of Internet network environment, RTSP+RTP is used for Internet video and audio transmission less. The Internet video service usually uses TCP as the Transport layer protocol of its streaming media, so such protocols as rtmp,mms,http are widely used in the Internet Audio-visual service. Such protocols do not lose packets, thus guaranteeing the quality of the video, but the transmission efficiency will be relatively low.

In addition RTMFP is a relatively new streaming media protocol, which is characterized by peer-to-peer support.

Rtmp I do a lot of research: such as RTMP specification simple analysis, or rtmp streaming media playback process

Source code Analysis of related tools: Rtmpdump source Analysis 1:main () function [series articles]

RTMP Protocol Learning: rtmp Streaming Media Technology 0 Basic Learning Methods 3. Encapsulation Format

The main function of the package format is to store the video stream and audio stream in a file in a certain format. Today's popular packaging format is shown in the following table:

Main package Format list

Name	Launch Agency	Streaming media	Supported video encoding	Supported audio encoding	Current areas of Use
Avi	Microsoft INC.	does not support	Almost all formats	Almost all formats	BT download Film and television
MP4	MPEG	Support	MPEG-2, MPEG-4, H.264, H.263, etc.	AAC, MPEG-1 Layers I, II, III, AC-3, etc.	Internet video site
Ts	MPEG	Support	MPEG-1, MPEG-2, MPEG-4, H.264	MPEG-1 Layers I, II, III, AAC,	IPTV, Digital TV
FLV	Adobe INC.	Support	Sorenson, VP6, H.264	MP3, ADPCM, Linear PCM, AAC, etc.	Internet video site
MKV	Corecodec INC.	Support