Video coding and decoding 4: Video Processing and coding standards

Chapter 4 Video Processing

1. Video Processing

• Operations on video image quality enhancement before and after video compression
• The subjective image quality output by the video codec system is not only related to the compression algorithm performance, but also influenced by video processing.
• Processing of videos before compression is called preprocessing (pre-processing)
• Processing a compressed video is called post-processing)

2. Preprocessing

• Purpose of preprocessing
  • In order to reduce the damage to the original image, maintain the important features of the original image, so that the original image can be efficiently compressed
    • Noise Pollution
    • Illumination difference
    • Jitter
  • To convert the video format
    • Split rows
    • Space Scaling
    • Frame Rate Conversion
• Noise Removal

• Deinterlace)
  • The video image scanned by line has a line-by-line effect on the row-by-line scanning playback device, reducing the subjective quality.

3. method for removing the line

• Separate rows in a single Field
  • Interpolation of even (ODD) rows using odd (even) rows

• De-interlacing between two
  • Use the pixel values of two fields (including the Motion Vector Information) to calculate the pixel value of one frame.

• Motion Compensation Method
  • Use the Motion Vector information of the forward and backward reference fields to produce high-quality frame by row

4. MpEG deduplication

• MPEG-2 post-processing

5. Space Scaling

• Up Sample Method
  • Nearest-neighbor interpolation)
    • The value of each pixel point in the original image is used as the value corresponding to four adjacent pixel points in the sampled image.
    • Will produce a step shape effect
  • Linear and bilinear interpolation

  P (x, y) = P () xy + P () Y + P () x (1-y) + P) (1-y)

  • Bicubic interpolation)
    • Interpolation of the current pixel with neighboring 16 pixel values
    • Smooth interpolation

• Down Sample Method
  • Shrink Original Image

5. Super Resolution

• Optical Flow: Motion Vector of pixels

6. Frame Rate Conversion

• Drop-down
  • Convert a 24-frame/second movie to a 60-frame/second TV video.
  • 24/30 = 4/5

• Frame Rate downsampling (frame skip)

• Frame Rate sampling

7. Post-processing

• The purpose of post-processing is to remove various effects of decoded images, restore lost image data to the maximum extent, and improve the subjective quality of decoded images.
  • The quantization method causes the loss of the conversion coefficient information, resulting in distortion of the decoded image and the continuity of the block boundary.
    • Block artifact and ringing artifact)
    • Distortion)
  • Decoding image quality problems caused by video stream transmission and Decoding
    • Frame jump (frame skip)
    • Data Loss)

8. Block Effect

• Cause of Block Effect
  • Because the block is processed independently

9. Block effect filtering Removal

• Block effect Filtering
  • Out Loop Filter ):
    • The filter filters the image in the display buffer outside the decoding loop.
    • Non-standard part. Only the decoder needs out-of-the-loop filtering.
    • MPEG-2, MPEG-4 sp
  • Loop filter ):
    • The filter is in the decoding loop. The filtered image can be used as a reference frame for motion estimation and compensation.
    • As part of the standard, all codecs require loop filtering.
    • AVS, H.264/AVC

10. Ringing Effect

• Causes of the ringing effect
  • Due to the serious quantification of high-frequency coefficients

11. Data Loss

• Data Loss

12. H.264 loop Filtering

• Filter Boundary

• Adaptive Filtering

• Filter strength

• Filter decision threshold

• Filter strength
  • The filtering intensity is related to the quantization parameter QP.
    • The higher the QP, the larger the quantization degree will produce more block effects, and more filters are needed.
    • The lower the QP, the smaller the quantization degree, the less block effect will be generated, and the weaker the filter will be used to prevent the boundary from becoming blurred.
  • Two filters are available based on the filter strength (BS ).
    • 1 ≤ BS ≤ 3, basic filtering operation
    • BS = 4, strong filtering operation
  • Boundary filter computing

• Truncation parameter C

• BS = 4
• More Obvious Block Effect
• More severe filtering is required to remove the block effect.
• Perform strong filter on the boundary when the following conditions are met:

• Modify when FilteringP0,P1,P2Pixel value
• Modify only when weak FilteringP0Pixel value

 

Chapter 2 video encoding standards

1. International Standards Organization

• ITU (International Telecommunications Union): International Telecommunication Union
• ISO (International Standardization Organization): International Organization for Standardization
• IEC (International Electrotechnical Commission): International Electrotechnical Commission

2. ITU

• ITU-T (telecommunication): Telecommunications Standardization Department
• ITU-R (Radiocommunication): Radio Communication Division
• ITU-D (Development): Telecommunications Development Department
• VCEG (Video Coding Experts Group): Video Coding Expert Group, belonging to study group 16/question 6 of ITU-T (ITU-T sg16/Q.6)

3. VCEG

• VCEG standards:
  • H.261: the first video encoding standard used for video conferences transmitted by ISDN (px64kbps). It mainly uses full-pixel motion compensation.
  • H.262 (MPE-2): standard developed in partnership with MPEG
  • H.263 (H.263 +, H.263 + +): used for low-bit video encoding. It is mainly used for video conferencing and video calls with psdn less than 54kbps and uses 1/2 Pixel Motion Compensation, data loss and error robustness considerations
  • H.26L: predecessor standard of H.264
  • H.264 (MPEG-4 AVC): currently the highest coding efficiency standard

4. ISO/IEC

• JPEG: ISO/IEC JTC1/sc29/WG1
• MPEG: ISO/IEC JTC1/sc29/wg11

5. MPEG

• MPEG standards:
  • MPEG-1: H.261 + semi-Pixel Motion Compensation + bidirectional prediction + band structure encoding for VCD, LAN Video (max. 1.5 Mbps)
  • MPE-2: MPEG-1 + barrier video encoding + scaling, for HDTV (18 Mbps), sdtv (2-5 Mbps) digital signal transmission and DVD (6-8 Mbps) Storage, compatible with MPEG-1 standards
  • MPEG-4 ASP (P-2): Developed from H.263 standard, based on the video object plane encoding, can control the video object in the decoding end, synthesis scene, and audio and video interaction function
  • MPEG-4 AVC (P-10) (H.264/AVC): standards developed in collaboration with vecg

6. video encoding standards

• International Standards Organizations and Their coding standards

7. Standard Application

8. Other standard organizations

• AVS (China audio and video standard)
  • Avs1.0: for HD Digital TV broadcasting
  • AVS-M for mobile video
  • AVS-S for surveillance video
• SMPTE (the society of motion picture and television engineers)
  • VC-1

9. Development of video encoding standards

MPEG-1

• Standard documentation: ISO/IEC 11172-2
• Coding Technology: Based on H.261
  • Frame B encoding: frame B cannot be referenced by other frames.
  • 16x16 Accuracy Motion Compensation
  • Bit Rate Control: Macro Block-level Bit Rate Control, and adjustment of Macro Block quantization parameters to control the bit rate
  • Supports random video access, playback, code stream editing, and audio/video synchronization.
• Reference software: Simulation Model 3 (Sm3)

11. MPEG-2

• Standard documentation: ISO/IEC 13818-2
• Coding Technology: Based on MPEG-1
  • Frame and field Motion Compensation
  • Three types of Scaling
    • Space Scaling
    • Time Scaling
    • SNR Scaling
• Reference software: Test Model 5 (TM5)
• Profile: tool set
• Level: parameter restrictions
• Use different profile and level combinations for different applications

12. H.263 Series

• H.263 encoding technology: Based on H.261
  • Motion Compensation with half pixel precision
  • Optional data stream hierarchy
  • Four efficient coding modes
    • D option-unrestricted motion vector mode
    • E Option-syntax-based Arithmetic Coding
    • F option-advanced prediction mode, obmc
    • G option-Pb Frame Mode
  • Supports more video and image resolutions
  • Reference software tmn2.0 (Test Model near-term 2.0)
• H.263 + coding technology: Based on H.263
  • Allows custom video image formats
  • Added 12 options to improve compression efficiency
    • I option-advanced in-frame prediction Mode
    • J option-Block Removal Filter
    • K option-strip structure mode
    • L option-supplemental enhancement information, used for decoder display related information
    • M option-enhanced Pb Frame Mode
    • N option-select a Reference Frame
    • O option-grading, time grading, spatial grading, SNR grading
    • P option-reference frame re-sampling
    • Q option-reduced resolution update
    • R option-independent Fragment Decoding
    • S option-alternative inter-VLC
    • T option-improved quantification
• H.263 ++ coding technology: Based on H.263 +
  • Three options are added to enhance the anti-code capability.
    • U option-enhanced reference frame selection
    • V option-Data Partition strip, transform quantization coefficient and Motion Vector Data Separation
    • W option-added supplementary enhancement information, including fixed-point IDCT options, image information, information types, repeated image headers, alternate field indications, sparse reference frame Recognition

13. MPEG-4

• Standard documentation: ISO/IEC 14496-2
• Coding Technology: Based on H.263
  • Added object-based encoding and operations
    • Shape segmentation Encoding
    • Scaling static texture Wavelet Transformation
    • Mesh Animation Encoding
    • Face Animation Encoding
    • Content synthesis, semi-synthetic Encoding
    • 10-bit, 12-bit deep sampling
    • See software vm7 (Verification Model 7
• Object-based encoding
  • The scenario is divided into multiple objects.
  • Each object is described by shape, motion, and texture features.

• Video Object Plane (VOP)

• Sprite Encoding
  • Encode the entire video background and the affined ing from the sequence, and place some background content on the decoded image through the affined ing.

• Mesh Animation Encoding
  • Objects can be described by the motion vectors of the initial mesh and nodes.

• Profile

14. H.264/AVC

• Standard documentation: ISO/IEC 14496-10 (ITU-T H.264)
• Predecessor: H.26L (Long Range), based on MPEG-2/4
• Reference software JM (joint model)

15. H.264 encoding structure

16. H.264-intra Encoding

• Four 16x16 brightness Modes
• 9 models with a brightness of 4x4
• Four 8x8 color Modes
• High encoding complexity
• Not conducive to error recovery

17. H.264-Motion Compensation

• Variable Block Motion Compensation
• Pixel Motion Compensation
  • Pixel filter (1,-5, 20, 20,-5, 1)
• 1/8 Pixel Precision color Motion Compensation
  • Use bilinear Filtering for color
• Image outer Motion Vector
• Multi-Reference Frame
  • Different reference frames can be used for four 8x8 sub-blocks of each macro block.

18. H.264 multi-Reference Frame

• P Frame
  • Only reference the reference frame of a list, list0
• Frame B
  • Refer to the reference frames of the two lists, list0 and list1
• Each list can contain a maximum of 16 reference frames.
  • List0: Buffer of previous reference frames
  • List1: Future reference frame buffering

19. h.274-weighted prediction

20. H.264-switching Images

• SP/Si frame implementation
  • Code stream Switching
  • Random cut-in
  • Error Recovery
  • You can also implement fast forward and return
• Features
  • Identical reconstructed images can be obtained for different reference images
  • SP frame encoding efficiency is lower than P frame but higher than I frame
  • The encoding efficiency of Si frames is lower than that of I frames.

21. H.264-FMO

• Flexible Macro Block sequence (flexible MB order, FMO)
  • The Macro Block has different combinations of components.
  • Seven different macro block combinations
• FMo enhances the decoding error recovery capability

22. H.264-Data Partition

• The loss of upper-layer syntax elements seriously affects the decoding of compressed code streams.
• Divide the code stream into three parts based on Importance
  • A: header information, including macro block type and Motion Vector
  • B: intra coefficient Information
  • C: inter coefficient Information
• Use different priorities to transmit partitions with different importance levels for error recovery

23. H.264-ASO, RP

• Arbitrary slice order, Aso)
  • Independent decoding between strip
  • Strip can be transmitted in any order
  • The lost strip can be decoded and restored.
• Redundant picture)
  • Add redundant images to the code stream. If the image data is lost, you can use redundant images instead.

24. H.264-video encoding Layer

• Independent Network encoding Layer
• The VCL layer includes:
  • Sequence parameter set: sequence parameter set (SPS), storing sequence header information
  • Image parameter set: picture parameter set (PPS), storing image header information
  • Strip the following data
  • Additional enhancement information (SEI)

25. H.264-network extraction layer

• Extracted VCL data
• Suitable for data transmission and media storage at the transport layer
• The Nal Unit defines a common transmission format for data packets and code stream systems.

• The NAL units are connected together to form the nal stream.
• NAL unit for stream transmission

• NAL unit for data packet transmission

26. H.264-profiles

• Profile: subset of code stream syntax

• Fidelity range extensions

27. H.264-Levels

• Level: parameter restrictions

28. AVS vs. H. 264

29. Performance Comparison

30. AVS Member