Live555 Study Notes 17-hsf-videostreamparser

17: Detailed description of hsf-videostreamparser

How can we make full use of live555 for real-time h264 RTP transmission?
As you can see, in the existing hsf-videofileservermediasubsession, sink uses hsf-videortpsink and source uses hsf-videostreamframer. However, this connection is very complicated and many other nodes need to be inserted between the two nodes, the actual situation is as follows: bytestreamfilesource --> hsf-videostreamparser --> hsf-videostreamframer --> hsf-fuafragmenter --> hsf-videortpsink. wow! Is it really so complicated? That's right!
Of course, you don't need to care about their ins and outs. You only need to implement a source that can collect images and perform h264 encoding (of course, you can use the CPU or DSP for encoding ), then replace bytestreamfilesource with it. For example, your source can be called hsf-bytestreamsource. of course, to improve efficiency, the collection and encoding should be executed in another thread.

However, I still want to know what the hsf-videostreamparser is and what is the use of parser? What does it do? How does it work with hsf-videostreamframer? Is there memory copy between them?

Imagine a problem:
What role is hsf-videostreamframer? According to the Code of hsf-videofileservermediasubsession, hsf-videostreamframer actually represents the source, and the source facing the sink is it. However, it connects a bytestreamfilesource. Look code:

FramedSource* H264VideoFileServerMediaSubsession::createNewStreamSource(unsigned /*clientSessionId*/,unsigned& estBitrate){estBitrate = 500; // kbps, estimate// Create the video source:ByteStreamFileSource* fileSource = ByteStreamFileSource::createNew(envir(),fFileName);if (fileSource == NULL)return NULL;fFileSize = fileSource->fileSize();// Create a framer for the Video Elementary Stream:return H264VideoStreamFramer::createNew(envir(), fileSource);}

Right? I'm not fooling around, right?
Bytestreamfilesource acquires data from a file. No matter what media format it is, it only reads the file. therefore, it is obvious that hsf-videostreamframer uses bytestreamfilesource to obtain data from the file, and then hsf-videostreamframer then analyzes the data. for example, find every NALU and send it to sink. however, hsf-videostreamframer does not analyze it by itself, but uses parser. Therefore, an hsf-videostreamparser is added to the string.
Hsf-videostreamparser has two source pointers: framedsource * finputsource and fusingsource. We can see that hsf-videostreamparser concatenates finputsource and fusingsource, and finputsource is bytestreamfilesource.

Let's imagine what hsf-videostreamparser does: hsf-videostreamframer transmits its buffer (actually sink) to hsf-videostreamparser. Whenever hsf-videostreamframer wants to obtain a nalu, it needs to be done with hsf-videostreamparser, hsf-videostreamparser reads a batch of data from bytestreamfilesource and analyzes it. If a NALU is obtained, it is transmitted to hsf-videostreamframer. alas, hsf-videostreamframer is really a bad guy who gets nothing!

Let's take a look at the actual process:

// Sink calls getnextframe () of source (bytes) to obtain data, // hsf-videostreamframer derives from mpegvideostreamframer, so the following function will be called: void mpegvideostreamframer: dogetnextframe () {fparser-> registerreadinterest (FTO, fmaxsize); continuereadprocessing ();} void mpegvideostreamframer: callback (void * clientdata, unsigned char */* PTR */, unsigned/* size */, struct timeval/* presentationtime */) {mpegvideostreamframer * framer = (mpegvideostreamframer *) clientdata; framer-> continuereadprocessing ();}

The first two are transitions and are executed here:

Void mpegvideostreamframer: continuereadprocessing () {// call Parser's Parser () to analyze a NALU. if a NALU is obtained, // aftergetting (this) is returned to the sink. unsigned acquiredframesize = fparser-> parse (); If (acquiredframesize> 0) {// we were able to acquire a frame from the input. // It has already been copied to the reader's space. fframesize = acquiredframesize; fnumtruncatedbytes = fparser-> numtruncatedbytes (); // "fpresentationtime "Shocould have already been computed. // compute" fdurationinmicroseconds "now: fdurationinmicroseconds = (fframerate = 0.0 | (INT) fpicturecount) <0 )? 0: (unsigned) (fpicturecount * 1000000)/fframerate); fpicturecount = 0; // call our own 'after getting' function. because we're not a 'leaf' // source, we can call this directly, without risking infinite recursion. aftergetting (this);} else {// it does not mean that data is not obtained in Parser !! // We were unable to parse a complete frame from the input, because: //-we had to read more data from the source stream, or //-the source stream has ended .}}

Have you noticed the comments in the else {} of the above function? This is a difficult phenomenon to understand and will be explained later. Here we will first look at how the Parser () function obtains data and analyzes it.
In Parser (), reading new data is caused by functions such as test4bytes () and skipbytes (). They all call ensurevalidbytes1 ():

void StreamParser::ensureValidBytes1(unsigned numBytesNeeded){// We need to read some more bytes from the input source.// First, clarify how much data to ask for:unsigned maxInputFrameSize = fInputSource->maxFrameSize();if (maxInputFrameSize > numBytesNeeded)numBytesNeeded = maxInputFrameSize;// First, check whether these new bytes would overflow the current// bank.  If so, start using a new bank now.if (fCurParserIndex + numBytesNeeded > BANK_SIZE){// Swap banks, but save any still-needed bytes from the old bank:unsigned numBytesToSave = fTotNumValidBytes - fSavedParserIndex;unsigned char const* from = &curBank()[fSavedParserIndex];fCurBankNum = (fCurBankNum + 1) % 2;fCurBank = fBank[fCurBankNum];memmove(curBank(), from, numBytesToSave);fCurParserIndex = fCurParserIndex - fSavedParserIndex;fSavedParserIndex = 0;fTotNumValidBytes = numBytesToSave;}// ASSERT: fCurParserIndex + numBytesNeeded > fTotNumValidBytes//      && fCurParserIndex + numBytesNeeded <= BANK_SIZEif (fCurParserIndex + numBytesNeeded > BANK_SIZE){// If this happens, it means that we have too much saved parser state.// To fix this, increase BANK_SIZE as appropriate.fInputSource->envir() << "StreamParser internal error ("<< fCurParserIndex << "+ " << numBytesNeeded << " > "<< BANK_SIZE << ")\n";fInputSource->envir().internalError();}// Try to read as many new bytes as will fit in the current bank:unsigned maxNumBytesToRead = BANK_SIZE - fTotNumValidBytes;fInputSource->getNextFrame(&curBank()[fTotNumValidBytes], maxNumBytesToRead,afterGettingBytes, this, onInputClosure, this);throw NO_MORE_BUFFERED_INPUT;}

We can see a strange phenomenon: This function does not return a value, but an exception is thrown, and this exception is thrown as long as the function is executed.
Let's analyze what this function has done first:
First, determine whether your buffer can accommodate the required data volume. If you cannot, you can only prompt that you finally obtain a lump of data from bytestreamfilesource. curback () returns the buffer of parser. the callback function aftergettingbytes is passed in by callback. Therefore, after obtaining the data, the hsf-videostreamframer function is executed. After several intermediate steps, the above void mpegvideostreamframer: continuereadprocessing () is executed (). wow, I saw a problem: Parser () is nested in Parser ()! After the second execution of Parser () is complete, ensurevalidbytes1 () is returned, and then exits due to an exception thrown. Where is the exit? Returned to the last called Parser (), because the code in try {} catch {}. Catch {} is written in Parser () as follows:

catch (int /*e*/){#ifdef DEBUGfprintf(stderr, "H264VideoStreamParser::parse() EXCEPTION (This is normal behavior - *not* an error)\n");#endifreturn 0; // the parsing got interrupted}

It can be seen that Parser () returns 0 at this time, and Parser () returns 0 and runs it to the else {} section in mpegvideostreamframer: continuereadprocessing (). Go back and check it. In fact, nothing is done. that is to say, when Parser () is called for the first time, it only obtains data from bytestreamfilesource, so this Parser () does not do anything after obtaining data, but actually analyzes the NALU and () is completed, not in itself, but in its nested call that causes Parser. confused:
To obtain data, the sink executes mpegvideostreamframer: continuereadprocessing (), mpegvideostreamframer: continuereadprocessing to call Parser (), and Parser () to use the data () get data from bytestreamfilesource. After obtaining data, mpegvideostreamframer: aftergettingbytes () is called and transitioned to mpegvideostreamframer: extract (), mpegvideostreamframer: extract () nested call !, In mpegvideostreamframer: continuereadprocessing (), Parser () is called again. At this time, Parser () needs to use the data to find the data, so after analyzing and analyzing a nalu, returned to mpegvideostreamframer: continuereadprocessing (), mpegvideostreamframer: continuereadprocessing () will call aftergetting (this) to return the data to the sink. after the sink processes the data, it returns to mpegvideostreamframer: continuereadprocessing (), mpegvideostreamframer: continuereadprocessing () in catch {}, Parser () returns to mpegvideostreamframer: continuereadprocessing () in the first call, mpegvideostreamframer: continuereadprocessing () finds that Parser () has not obtained NALU, therefore, the sink will continue to use source-> getnextframe ()-> mpegvideostreamframer: continuereadprocessing ()... in this way, obtain NALU again.

A story of twists and turns! But the lecture is finished!

As you can see, Parser has its own buffer and its size is fixed:
# Define bank_size 150000
When you write the source, each time a frame of data is output, it contains multiple nalu, so you only need to make sure that your frame does not exceed 150000 bytes, you can copy to FTO with confidence. If your frame is too large, change the macro.

I will announce it hereLive555 QQ group number:185130569Welcome to the study of streaming media, where we can discuss anything related to streaming media, live555, FFMPEG, VLC, rtmp, etc ....