Live555 source code analysis-MPG file processing

Live555 supports most single-Stream files. It only supports *. mpg, *. MKV, and *. webm audio/video hybrid files. In fact, my goal is to expand the supported formats, such as Avi, so I will analyze the processing of MPG files.

MPG file processing involves many classes, and the relationships between classes are complex. For the RTP packaging process (completed in rtpsink), some media types are similar (there are differences in details, these are all done in the *** rtpsink related to a specific media), so the main analysis is the process of getting data from the source. A subsession corresponds to a stream. When a session has multiple subsessions, You need to independently control each subsession. We can see that when processing the "play" command, each subsession in the Session calls a startstream operation, as shown below:

Void rtspserver: rtspclientsession: handlepai_play (servermediasubsession * subsession, char const * CSeq, char const * fullrequeststr) {// now the media data has been transmitted // now, start streaming: for (I = 0; I <fnumstreamstates; ++ I) {If (subsession = NULL/* means: aggregated operation */| subsession = fstreamstates [I]. subsession) {unsigned short rt1_qnum = 0; unsigned rtptimestamp = 0; // start data transmission on each subsession, that is, start playing fstreamstates [I]. subsession-> startstream (foursessionid, fstreamstates [I]. streamtoken, (taskfunc *) noteclientliveness, this, rt1_qnum, rtptimestamp, handlealternativerequestbyte, this );...}}...}

The previous article has analyzed the handling process of the play command. The subsession-> startstream call is used to start playing on each stream. The source data obtained from the source will be in multiframedrtpsink :: packframe.

Void multiframedrtpsink: packframe () {If (foutbuf-> haveoverflowdata ()){...} else {... /// obtain the next frame from source // fsource-> getnextframe (foutbuf-> curptr (), foutbuf-> totalbytesavailable (), aftergettingframe, this, ourhandleclosure, this );}}

For MPEG, fsource is an mpeg1or2videostreamframer or mpeg1or2audiostreamframer instance (according to the source code in live555, there is also an AAC audio format, which is not analyzed for ease ). Their Inheritance relationships are as follows:
Mpeg1or2videostreamframer-> mpegvideostreamframer-> framedfilter-> framedsource-> mediasource
Mpeg1or2audiostreamframer-> framedsource-> mediasource

First, analyze the processing of MPEG Video.
The implementation of the mpeg1or2videostreamframer class is simple. It is mainly to create an mpeg1or2videostreamparser instance for the corresponding syntax analyzer. Apparently, mpeg1or2videostreamframer (a direct subclass of framesource) is a filter. It can be found that its input source is an mpeg1or2demuxedelementarystream instance. For a single stream file, the bytestreamfilesource class instance is generally packaged. From the following we can find that the object is directly read by the bytestreamfilesource instance. For the syntax analysis part, we only care about how to parse audio and video data from the file, so we track the mpeg1or2demuxedelementarystream class encapsulated by the filter directly. In the syntax analyzer, The getnextframe function of mpeg1or2demuxedelementarystream is called.

Getnextframe is a non-virtual function defined in framedsource. The implementation is as follows:

Void framedsource: getnextframe (unsigned char * To, unsigned maxsize, aftergettingfunc * aftergettingfunc, void * handle, onclosefunc * onclosefunc, void * oncloseclientdata) {// make sure we're re not already being read: If (fiscurrentlyawaitingdata) {envir () <"framedsource [" <This <"]: getnextframe (): attempting to read more than once at the same time! \ N "; envir (). internalerror () ;}fto = to; fmaxsize = maxsize; fnumtruncatedbytes = 0; // by default; cocould be changed by dogetnextframe () fdurationinmicroseconds = 0; // by default; cocould be changed by dogetnextframe () functions = aftergettingfunc; functions = functions; fonclosefunc = onclosefunc; foncloseclientdata = oncloseclientdata; functions = true; dogetnextframe (); // get the next frame}

Framedsource: getnextframe initializes some member variables. Other work is handed over to the dogetnextframe function. It is a pure virtual function on framesource and is re-implemented in the subclass mpeg1or2demuxedelementarystream.

void MPEG1or2DemuxedElementaryStream::doGetNextFrame() {  fOurSourceDemux.getNextFrame(fOurStreamIdTag, fTo, fMaxSize,       afterGettingFrame, this,       handleClosure, this);}

Foursourcedemux is defined as a reference to the image of mpeg1or2demux (directly inherited from medium. Why is it a reference instead of a pointer? Let's think about it. We need to create an instance for each stream in the file (of course, only the video-related classes are listed), including callback, mpeg1or2videortpsink, mpeg1or2videostreamframer, and mpeg1or2videostreamparser, however, there is only one file corresponding to all streams, and the process of finally reading the file must be one. Each stream in the file corresponds to an mpeg1or2demuxedelementarystream instance, but corresponds to the same mpeg1or2demux instance. Here we use references instead of pointers to demonstrate that foursourcedemux belongs to an mpeg1or2demuxedelementarystream instance.

Before creating an mpeg1or2demux instance, A bytestreamfilesource will be created. Let's take a look at the definition of the mpeg1or2demux: getnextframe function.

Void metadata: getnextframe (writable streamidtag, unsigned char * To, unsigned maxsize, framedsource: writable * handle, void * handle, framedsource: onclosefunc * onclosefunc, void * handle) {// first, check whether we have saved data for this stream ID: // check whether streamidtag stream data already exists in the cache // If (usesaveddata (streamidtag,, maxsize, aftergettingfunc, aftergettingclientdata) {return;} // note that the callback function is set here // then save the parameters of the specified stream ID: registerreadinterest (streamidtag,, maxsize, aftergettingfunc, functions, onclosefunc, oncloseclientdata); // next, if we're all the only currently pending read, continue looking for data: If (fnumpendingreads = 1 | response) {fhaveundelivereddata = 0; continuereadprocessing (); // continue reading data} // otherwise the continued read processing has already been taken care}

In the above Code, the usesaveddata function is called first to check whether the data in the cache contains the corresponding stream. If the data does not exist, it will only be read from the file. Now we can make a guess that the cache is required, and the data of different streams in the file is staggered, and the data is generally read in order during reading. Now we want to read a video frame, but the current file pointer reads an audio/video frame, so we need to save the audio frame to the cache until we can read the video data. Is the actual situation true?

Let's take a look at the implementation of the mpeg1or2demux: usesaveddata function.

Boolean identifier: usesaveddata (u_int8_t streamidtag, unsigned char * To, unsigned maxsize, framedsource: Random * handle, void * handle) {struct outputdescriptor & out = foutput [streamidtag]; // foutput is a cache array. // normally, if (Out. saveddatahead = NULL) return false; // common case unsigned totnumbytescopied = 0; // read all data from the outputdescriptor cache // whi Le (maxsize> 0 & out. saveddatahead! = NULL) {outputdescriptor: saveddata & saveddata = * (Out. saveddatahead); unsigned char * From = & saveddata. data [saveddata. numbytesused]; unsigned numbytestocopy = saveddata. datasize-saveddata. numbytesused; If (numbytestocopy> maxsize) numbytestocopy = maxsize; memmove (to, from, numbytestocopy); to + = numbytestocopy; maxsize-= numbytestocopy; out. saveddatatotalsize-= numbytestocopy; totnumbytesc Opied + = numbytestocopy; saveddata. numbytesused + = numbytestocopy; If (saveddata. numbytesused = saveddata. datasize) {out. saveddatahead = saveddata. next; If (Out. saveddatahead = NULL) out. saveddatatail = NULL; saveddata. next = NULL; Delete & saveddata;} Out. iscurrentlyactive = true; If (aftergettingfunc! = NULL) {struct timeval presentationtime; presentationtime. TV _sec = 0; presentationtime. TV _usec = 0; // shocould fix ##### (* aftergettingfunc) (latency, latency, 0/* numtruncatedbytes */, presentationtime, 0/* durationinmicroseconds ????? ##### */);} Return true ;}

To analyze the above Code. The cache is defined as an array of outputdescriptor types.
Outputdescriptor_t foutput [2, 256];
A 256 instance is defined, but only this sample can use streamindex streamidtag to directly access array elements. The streamidtag should be less than 256. Each stream corresponds to an outputdescriptor instance. The actual data is stored in the outputdescriptor: saveddata linked list.

Now let's continue to look at the mpeg1or2demux: getnextframe function. We need to know how to separate different media streams. Let's look at the continuereadprocessing function.

Void mpeg1or2demux: Random () {While (fnumpendingreads> 0) {unsigned char acquiredstreamidtag = fparser-> parse (); // File Syntax analysis if (acquiredstreamidtag! = 0) {// if the data is not read, this value is 0 // we get a frame from the input source // we were able to acquire a frame from the input. struct outputdescriptor & newout = foutput [acquiredstreamidtag]; newout. iscurrentlyawaitingdata = false; // indicates that we can read the next frame, in the parse, the stream that is being processed will be determined based on this value // indicates that we can be read again // (this needs to be set before the 'after getting' call below, // In case it tries to read another frame) // call "afte R getting "function // call our own 'after getting' function. because we're not a 'leaf' // source, we can call this directly, without risking infinite recursion. if (newout. faftergettingfunc! = NULL) {(* newout. Timeout) (newout. aftergettingclientdata, newout. framesize, 0/* numtruncatedbytes */, newout. presentationtime, 0/* durationinmicroseconds ????? ##### */); -- Fnumpendingreads;} else {// we were unable to parse a complete frame from the input, because: //-we had to read more data from the source stream, or //-we found a frame for a stream that was being read, but whose // reader is not ready to get the frame right now, or //-the source stream has ended. break ;}}}

Mpeg1or2demux: The implementation of continuereadprocessing seems very familiar! Yes, it is similar to mpegvideostreamframer: continuereadprocessing. Both functions call the syntax analyzer for syntax analysis. However, you should note that the former syntax analyzer is an mpegprogramstreamparser class instance, which completes parsing the composite file, the main purpose is to extract audio and video streaming data (in fact, the Demux process) from it, while the latter is an mpeg1or2videostreamparser, which further analyzes the specified stream data. The data source of mpegprogramstreamparser is a bytestreamfilesource instance, which is a parameter passed in the mpeg1or2demux constructor. If the data is not read, fparser-> parse () returns 0.

Let's look at mpegprogramstreamparser: parse function implementation

Unsigned char mpegprogramstreamparser: parse () {unsigned char acquiredstreamtagid = 0; try {do {Switch (fcurrentparsestate) {Case parsing_pack_header: {parsepackheader (); // analyze the Baotou break ;} case parsing_system_header: {parsesystemheader (); // analyze the system header break;} case parsing_pes_packet: {acquiredstreamtagid = parsepespacket (); // analyze the stream data break ;}}} while (acquiredstreamtagid = 0); Return acquiredstreamtagid;} catch (INT/* E */) {# ifdef debug fprintf (stderr, "mpegprogramstreamparser: parse () exception (this is normal behavior-* not * an error) \ n "); fflush (stderr); # endif return 0; // The parsing got interrupted }}

If you are not familiar with the mpeg file format, you just need to read the mpegprogramstreamparser: parsepespacket function for getting stream data packets.

Unsigned char mpegprogramstreamparser: parsepespacket (){... //// check whether the source is waiting for this stream type. If yes, submit the data to it. // check whether our using source is interested in this stream type. // If so, deliver the frame to him: mpeg1or2demux: outputdescriptor_t & out = fusingdemux-> foutput [stream_id]; If (Out. iscurrentlyawaitingdata) {unsigned numbytestocopy; If (pes_packet_length> out. maxsize) {numbytestocopy = out. maxsize;} else {numbytestocopy = pes_packet_length;} getbytes (Out. to, numbytestocopy); // copy data out. framesize = numbytestocopy; // set out. presentationtime later ##### acquiredstreamidtag = stream_id; pes_packet_length-= numbytestocopy; 0} else if (Out. iscurrentlyactive) {// this stream is required, but it is not now. When necessary, we can deliver the data /// someone has been reading this stream, but isn' t right now. // We can't deliver this frame until he asks for it, so punt for now. // The next time he asks for a frame, he'll get it. restoresavedparserstate (); // so we read from the beginning next time fusingdemux-> fhaveundelivereddata = true; throw reader_not_ready; // throw an exception} else if (Out. ispotentiallyreadable & out. saveddatatotalsize + pes_packet_length <1000000/* limit */) {// this stream is also required, but it is not read currently. Therefore, it is saved to the cache (outputdescriptor) /// someone is interested in this stream, but hasn' t begun reading it yet. // save this data, so that the reader will get it when he later asks for it. unsigned char * Buf = new unsigned char [pes_packet_length]; getbytes (BUF, bytes); character: outputdescriptor: saveddata * saveddata = new character: outputdescriptor: saveddata (BUF, pes_packet_length); // create a saveddata instance if (Out. saveddatahead = NULL) {out. saveddatahead = out. saveddatatail = saveddata;} else {out. saveddatatail-> next = saveddata; out. saveddatatail = saveddata;} Out. saveddatatotalsize + = pes_packet_length; pes_packet_length = 0;} skipbytes (pes_packet_length);} // check for another PES packet next: setparsestate (expiration); Return acquiredstreamidtag ;}

The mpegprogramstreamparser: parsepespacket function reads the data in the stream. If the read data is not required, it is saved to the cache, that is, its corresponding outputdescriptor instance. Note that the value of the local variable acquiredstreamidtag is initialized to 0. It is assigned only when the stream data is read. Therefore, if the required data is not read, this function returns 0.