H. 11 format, iOS hard codec, and support 1,h.264 format for HEVC hard codec in iOS
The network representation layer nal,h.264 flow consists of a frame-by-frame Nalu;
SPS: Sequence parameter set, used for a series of sequential coded images;
PPS: An image parameter set that is used to encode one or more independent images in a video sequence;
These two frames are also independent of the Nalu.
I-frame: keyframes, frames within the frame after encoding , showing a more complete frame;
P-frame: Referring to the previous frame, it is possible to compare the motion estimation of the previous frame with the change part;
B-frame: Will refer to the frame before and after, other similar p-frame. Both the B and P frame contain inter-frame encoding and intra- frame encoding .
Each nalu is delimited by the start Code: (3 bytes) or the XX (4 bytes).
2,ios on the hard coding and decoding of H.
IOS 8 supports hard coding of H. A by Videotoolbox;
The CMSAMPLEBUFFERREF structure is interesting and needs to be understood,
Cmvideoformatdesc contains Sps,pps and other useful information;
- When encoding, Cvpixelbuffer->cmsamplebufferref (which contains the compressed cmblockbuffer);
- Decoding, through the vtdecompressionsession to decode, Cmsamplebufferref (contains Cmblockbuffer)->cvimagebufferref (decoded frame data);
2.1 Encoding
Refer to the WWDC 2014 PDF, from the camera or read the cvpixelbuffer->encoder->cmsamplebufferref->nalus of the video file output.
Classic encoder recommended Live push stream Lflivekit, code structure is good, data flow is very clear, is suitable for reading;
Https://github.com/LaiFengiOS/LFLiveKit/blob/master/LFLiveKit/coder/LFHardwareVideoEncoder.m
The coding steps are roughly:
- 1,vtcompressionsessioncreate via wide, high, Codectype (kcmvideocodectype_h264), coded callback Vtcompressionoutputcallback such as initializing Vtcompressionsessionref instance compressionsession;
- 2,vtsessionsetproperty set more important encoding properties, such as
Kvtcompressionpropertykey_maxkeyframeinterval, key frame interval
Kvtcompressionpropertykey_realtime
Kvtcompressionpropertykey_averagebitrate
Kvtcompressionpropertykey_profilelevel
Kvtcompressionpropertykey_allowframereordering and so on; the last vtcompressionsessionpreparetoencodeframes;
- 3, add original Cvpixelbuffer and pts to Vtcompressionsessionencodeframe
根据frameCount 计算出的pts
Cmtime Presentationtimestamp = Cmtimemake (Framecount, (int32_t) _configuration.videoframerate);
and whether I-frame:
if 0 = @{(__bridge nsstring *) Kvtencodeframeoptionkey_forcekeyframe: @YES};}
- 4,vtcompressionoutputcallback callback by parsing cmsamplebufferref processing sps,pps,i-frame and non-i-frame respectively, adding startcode:00 00 00 01 then go through rtmp rollout.
2.2 Decoding
Encoding inverse process, usually in the player and other scenes applications, Nalu + Sps,pps->cmblockbuffer->cmsamplebufferref, and then the cmsamplebufferref wrapped frame data input into Vtdecompressionsessiondecodeframe, Cvimagebufferref directly upload OpenGL ES display via callback).
I wrote a simple iOS H. Decoder stream:
Https://github.com/edisongz/VideoHardwareStreamDecoder
3,ios 11 support for HEVC
IOS One + A9 chip starts support for HEVC, System control Avplayer, and Safari browser support, and HEVC has a 40% higher compression efficiency compared to H.
Hard coding requires IOS11 + A10 chip
Hard decode requires iOS one + A9 chip
IOS 11 New API
Decoderlet hardwaredecodesupported = vtishardwaredecodesupported (KCMVIDEOCODECTYPE_HEVC)//encoderlet error = Vtcompressionsessioncreate (Kcfallocatordefault, 3840, 2160, KCMVIDEOCODECTYPE_HEVC, encoderSpecification, nil, nil, Nil, nil,//using Vtcompressionsessionencodeframewithoutputhandler &session); if error = = Kvtcouldnotfindvideoencodererr { //no HEVC encoder}
The specific HEVC Videotoolbox hardware codec still needs to be perfected; the subsequent completion is added here:
Https://github.com/edisongz/VideoHardwareStreamDecoder
Summary
The hard codec of the iOS environment is older WWDC's session, this year's WWDC 2017, the addition of hard codec HEVC code, but also need to find a machine under test.
Reference
Hard-coded part of Lflivekit
WWDC Direct Access to Video Encoding and decoding
WWDC working with Heif and HEVC
Github writes its own streaming decoding
H # Format, iOS hard codec, and iOS 11 support for HEVC hard codecs