(Conversion) camera video acquisition compression and transmission

Introduction:
What is the basic function of the camera or video transmission? Video Transmission:
It is to upload images one by one to the screen. Because of the fast transmission speed, you can see continuous and dynamic images, just like playing movies. Generally, when the number of frames transmitted reaches 24 frames per second, the image continuity occurs.
Next we will introduce the whole process of Video Acquisition, compression, and transmission.
1. Working Principle of the camera (obtaining video data)
The working principle of the camera is roughly as follows: the scene is projected to the image sensor surface through an optical image generated by the lens (lens), and then converted to an electrical signal, after A/D (modulus conversion) after conversion, it becomes a digital image signal, which is then sent to the digital signal processing chip (DSP) for processing, then transmitted to the computer through the USB interface for processing, and the image can be seen through the display. Is the workflow of the camera:


Note 1: An image sensor is a semiconductor chip with hundreds of thousands to millions of PHOTODIODES on its surface. When the diode is exposed to light, it will produce a charge.
NOTE 2: digital signal processing chip DSP (Digital Signal Processing) function: optimizes digital image signal parameters through a series of complex mathematical algorithm operations, and transmit the processed signal to devices such as PCs through interfaces such as USB.
DSP architecture framework:
1. ISP (image signal processor)
2. JPEG encoder (JPEG image decoder)
3. USB device controller (USB device controller)
However, videos require that the obtained video images be transmitted over the Internet to remote computers for display, which involves the transmission of the obtained video images.
During transmission of such images, the images must be compressed. Generally, the compression methods include H.261, JPEG, and MPEG. Otherwise, the bandwidth required for transmission will increase. I do not know whether to pay attention to RealPlayer. When playing a movie, there will be a transmission speed of 250 kbps, 400 kbps, 1000 kbps under the player... The higher the image quality, the higher the speed. This is also the principle of video transmission by the camera. If the resolution of the camera is adjusted to 640 × 480, the size of each captured image is about 50 kb, with 30 frames per second, the video transmission speed required by the camera is 50*30/S = 1500 kbps = 1.5 Mbps. In real life, the resolution for online video chats is 320 × 240 or even lower, and the number of frames transmitted is 24 frames per second. In other words, when the video transmission rate is less than 300 kbps, people can enjoy smooth video transmission and chat. If a higher compression video, such as MPEG-1, can reduce the transmission rate to less than kbps. This is the network transmission speed required by the camera during normal video chats.
Ii. Video Compression
Video compression is the core of video processing. It can be divided into non-real-time compression and Real-Time Compression Based on the real-time performance. Video Transmission (such as QQ video instant chat) requires that videos be compressed to real-time compression.
The following describes why videos can be compressed.
Video compression is lossy compression. Generally, the compression ratio of video compression is very high.
The high compression ratio is because the video image has a large degree of time and space redundancy. The so-called time Redundancy refers to the fact that two adjacent images have similar pixel values at the same position and have great correlation, Especially static images and even two identical images, motion images are also highly correlated through some calculation (motion estimation), while spatial correlation refers to the same frame of images, and adjacent two pixels are also correlated. These correlations are the initial assumptions of the video compression algorithm. In other words, if the two conditions are not met (full-white noisy images, scene frequently switched images, etc.), the video compression effect will be poor.
Motion estimation is the key algorithm used to remove time correlation. It finds the most matched position of the current macro block in the previous image. In many cases, we only need to record the relative coordinates, that's enough. This saves a lot of code words and increases the compression ratio. In video compression algorithms, motion estimation is always the most critical and core part. Spatial correlation is removed through DCT transformation. Data in the time domain is mapped to the frequency domain, and then the DCT coefficients are quantified. Basically, all lossy compression is performed, there will be quantification, which increases the compression ratio most significantly.
The original image file is relatively large and must be compressed before it can be quickly transmitted and played smoothly. The compression ratio is used to measure the image compression size. Generally, the compression ratio of the camera is greater. That is to say, if the size of the image for 30 seconds is 30 mb before the image is not compressed, the size of the image will be 6 MB after the image is compressed according to the ratio of 5:1 of the camera.
Video compression algorithms include M-JPEG, MPEG, H.264, wavelet, JPEG 264, and AVS.
Basically, this is the core of video compression.
Iii. Video Transmission
In order to ensure real-time digital video network transmission and image quality, the choice of transport layer protocol is the key to the entire design and implementation. The Internet uses two transmission protocols on the IP layer: TCP (Transmission Control Protocol), connection-oriented network protocol, and UDP (User Datagram Protocol ), it is a connectionless network protocol.
TCP transmission: TCP (Transmission Control Protocol) is a connection-oriented network transmission protocol. Supports multi-data stream operations, provides throttling and error control, and even sorts messages that arrive in disordered order. Therefore, TCP transmission provides reliable data transmission services.
The general process of using TCP transmission:
After the client sends a connection request to the server, the server receives the request and sends a connection confirmation to the client. After the connection is implemented, both parties transmit data.
UDP transmission: UDP (User Datagram Protocol) is a connectionless network transmission protocol. It provides a basic low-latency data transmission service. You do not need to establish a connection in advance like TCP transmission. UDP has no timing, traffic control, or congestion management mechanism. The lost data is not retransmitted. Therefore, it provides an unreliable application data transmission service. However, in a good network environment such as a LAN, using UDP to transmit data is relatively reliable and efficient.
IP multicast technology: Multicast technology is a network technology that allows one or more senders to send data packets from one or more senders to multiple receivers. The Multicast Source sends the data packets to a specific multicast group. Only the host that joins the multicast group can receive the data packets. Multicast can greatly save network bandwidth, because no matter how many target addresses there are, only ships a single packet on any link of the network.
1. TCP/IP protocol and real-time transmission
The TCP/IP protocol was initially designed to provide non-real-time data services. The IP protocol is responsible for data transmission between hosts without error detection or error correction. Therefore, data is often lost or out of order. To ensure reliable data transmission, people use the TCP protocol for IP data transmission to improve the error detection and Error Correction capabilities at the receiving end. When packet loss or errors are detected, the sender is required to resend the packet, which inevitably leads to transmission latency and network bandwidth consumption. Therefore, traditional TCP/IP protocols provide poor ability to transmit real-time audio and video data. Of course, TCP is also an option when transmitting the video and audio data used for playback. If there is enough buffer and enough network bandwidth, it is also possible to transmit close to real-time video and audio over TCP. However, if the packet loss rate is high and the network condition is poor, it is almost impossible to use the TCP protocol for video or audio communication.
TCP and other reliable transport layer protocols such as xtp are not suitable for real-time video/audio transmission, mainly due to the following reasons:
1. TCP retransmission mechanism
We know that in TCP/IP, when the sender finds data loss, it will require re-transmission of the lost data packet. However, this requires one or more cycles (according to the fast retransmission mechanism of TCP/IP, this requires three additional frame latencies ), this retransmission is almost disastrous for video and audio data communication with high real-time requirements, because the receiver has to wait for the arrival of retransmission data, this results in latency and breakpoint (audio discontinuous or video solidification, etc ).
2. TCP congestion control mechanism
TCP's congestion control mechanism reduces its congestion window when detecting packet loss. On the other hand, the number (Bit Rate) produced by audio and video in a specific encoding method cannot be changed suddenly. Correct congestion control should be to change the encoding method of audio and video information, and adjust the frame rate or image width of video information.
3. TCP packet header size
Another drawback that TCP is not suitable for real-time video/audio transmission is that its packet header is larger than that of UDP. The TCP packet header is 40 bytes, while the UDP packet header is only 12 bytes. In addition, these reliable transport layer protocols cannot provide time stamp and encoding and decoding information, and these information is exactly the receiver (client). Therefore, TCP is not suitable for real-time transmission of audio and video information.
4. Slow startup
Even if the network is in good running status and there is no packet loss, a connection needs to be established because of TCP startup. Therefore, it takes a long time to initialize the network, in a real-time video/audio transmission application, the minimum latency is what we expect.
Therefore, TCP is not suitable for transmitting real-time audio and video data. To achieve real-time transmission of audio and video data, we need to seek other channels.
2. RTP protocol is suitable for real-time video and audio transmission.
RTP (Real-time transport protocol)/RTCP (Real-Time Transport Control Protocol) is an application-oriented transport layer protocol, it does not provide any guarantee of Transmission reliability and traffic congestion control mechanism. It is a transmission protocol designed by IETF (Internet Engineering Task Force) for real-time video and audio transmission. The RTP protocol is located on top of the UDP protocol and is functionally independent of the following transport layer (UDP) and network layer, but cannot exist as a layer separately, generally, the lower layer UDP protocol is used to multicast (Multicast) or Unicast (unicast) real-time audio and video data, so as to realize the transmission of multi-point or single-point audio and video data.
UDP is a connectionless Datagram Delivery Service. Although it is not as reliable as TCP, it cannot guarantee the quality of service (QoS) of the real-time video/audio transmission service ), RTCP is required to monitor data transmission and service quality in real time. However, because UDP transmission latency is lower than TCP, it can be well matched with audio and video streams. Therefore, in practical applications, RTP, RTCP, and UDP are used for audio and video media while TCP is used for data and control signaling transmission.
Conclusion: if the receiving end and the sending end are in the same LAN, due to the full bandwidth guarantee, TCP can also provide better performance in meeting the real-time performance of video transmission, the video transmission performance of TCP and UDP-based RTP is not much different. Because the bandwidth in the LAN is not the main contradiction, the latency of video data transmission is mainly reflected in the processing latency, which is determined by the processing capability of the processor and the processing mechanism used. However, when video data is transmitted in the Wide Area Network, the transmission performance is greatly dependent on the available bandwidth. Because TCP is a connection-oriented transport layer protocol, its retransmission mechanism and congestion control mechanism, the network condition will be deteriorated further, resulting in disastrous latency. At the same time, in this network environment, when the video data transmitted through TCP is re-built and played back at the receiving end, the breakpoint is very obvious, which is obviously intermittent, real-time transmission and transmission quality cannot be guaranteed. Relatively speaking, the real-time and quality of video data transmitted using RTP is much better.
4. Remote display of video images
When a compressed video is transmitted over the Internet to a remote location, the video information transmitted over the Internet must be decoded first and then displayed. The decoder chip has certain performance requirements, which is lower than the encoder. However, after all, it is video data processing. General chips (which do not support MMX and other multimedia commands) may be difficult. Display devices mainly include televisions, monitors, and monitors. Their signal interfaces are different. TV monitors are analog electrical signals, and the input of displays is digital signals.
The above describes how the camera obtains image data and where the acquired data is stored, how to compress and transmit it, and how to release and play it out in different places.

(Conversion) camera video acquisition compression and transmission