After reading stagefright and OMX il for more than a month, I feel a little bit about the framework. I am still feeling it. I will record it and share it with you, I am also reading the multimedia framework at the beginning. please correct me if you have any improper understanding.
Because stagefright and openmax run on two different processes, the communication between them must be processed by openbinder, but they do not know about openbinder, therefore, the communication between stagefright and openmax cannot be analyzed. In addition, this section disregards the audio. Assume that the video is an AVC encoded file in MP4 format. Opencore was the first to look at it, but it was too complicated because it took into account the portability of multiple platforms. Later, it changed to stagefright, the OMX il used by stagefright is the code in opencore ........ so I expect Google to rewrite an OMX il framework to fully replace opencore. First, let's take a brief look at how stagefright works. stagefright uses event to drive, and the event scheduler and event run in the same thread, the player inserts an event into the queue to drive the entire decoding process. the workflow of the event scheduler is as follows:
1. Check whether the queue is empty. If it is empty, wait for the event to be inserted.
2. Get the first event in the queue
3. Perform latency operations after calculating the delay time required by the event
4. Remove the event from the queue and execute the event
The event scheduler schedules tasks by repeating such a process continuously. In the specific code, the scheduling process is changed based on special circumstances. Currently, the event scheduler has the following types of events:
1. onvideoevent
2. onstreamdone
3. onbufferingupdate
4. oncheckaudiostatus
5. onprepareasyncevent The player class of stagefright is awesomeplayer. This class mainly includes the following members (excluding the audio part ):
1. mvideosource (video decoding)
2. mvideotrack (reads video data from multimedia files)
3. mvideorenderer (convert the format of the decoded video. The format used by Android is rgb565)
4. misurface (redrawing layer)
5. mqueue (event queue) The abstract process of stagefright is as follows: The following uses an MP4 file (AVC encoding) to analyze the abstract workflow of awesomeplayer (excluding audio)
1) set the absolute path of muri to xxxx. MP4.
2) Start mqueue. This will create a thread to run threadentry and name it timedeventqueue. This thread is the event scheduler.
3) open the file specified by muri. If the header of the file XXXX. MP4 is (... ftypisom...), mpeg4extractor will be selected as the separator.
4) use mpeg4extractor to separate the audio and video tracks of MP4 and return the mpeg4source video tracks to mvideotrack.
5) Select the decoder based on the encoding type in mvideotrack. avcdecoder is selected for the AVC encoding type (assuming OMX is not used), return to mvideosource, and set msource in mvideosource to mvideotrack.
6) insert the onvideoevent to the queue and start decoding and playback.
7) read the parsed video buffer through the mvideosource object
8) If the parsed buffer has not reached the AV timestamp synchronization time, the operation will be postponed to the next round.
9) If mvideorenderer is empty, initialize it. If OMX is not used, mvideorenderer is set to awesomelocalrenderer.
10) use the mvideorenderer object to convert the parsed video buffer to the rgb565 format and send it to the display module for image rendering.
11) re-insert onvideoevent into the event scheduler to loop As the collection layer of the underlying decoding component, OMX il provides a unified interface for the upper-layer Multimedia Framework. In stagefright of android2.2, stagefright uses OMX il implementation in opencore, to use the OMX il framework, you need to set mvideosource to the omxcodec class. The external interfaces of the OMX il layer mainly include the following:
1) stagefright uses omx_masterinit to initialize the OMX framework and load component
2) stagefright uses omx_mastergethandle to match component in OMX. If the matching succeeds, omx_handletype is returned for communication between omxcodec and component.
3) omxcodec uses omx_sendcommand to set the component Status and operate the component Port
4) omxcodec uses eventhandler to notify omxcodec of command execution results
5) omxcodec uses omx_getparameter and omx_setparameter to obtain and set the attribute parameters of component.
6) omxcodec uses omx_usebuffer to set the buffer used by compoment to the buffer allocated by omxcodec.
7) omxcodec uses omx_emptythisbuffer to pass undecoded buffer to component for decoding.
8) omxcodec uses omx_fillthisbuffer to pass an empty bffer to component to store decoded frames.
9) compoment uses emptybufferdone to notify omxcodec that the inputport buffer has been read.
10) compoment uses fillbufferdone to notify omxcodec that the outputport buffer has been filled.
The initialization process is as follows: The data of OMX component interacts mainly through the port. The port is divided into input and output, and the port is decoded by sharing the buffer with omxcodec, for example: buffer processing is mainly driven by the following four commands:
Omxcodec uses omx_emptythisbuffer to pass undecoded buffer to component. After receiving this command, component reads the data in the input port buffer and assembles it into frames for decoding, after the data in the buffer is read, emptybufferdone is called to notify omxcodec.
Compoment uses emptybufferdone to notify omxcodec that the inputport buffer has been read. After omxcodec receives the command, it reads the new video buffer to the buffer of the input port through mvideotrack and calls omx_emptythisbuffer to notify component
Omxcodec uses omx_fillthisbuffer to pass an empty bffer to component to store decoded frames. After receiving this command, component copies decoded frame data to the buffer, and then calls fillbufferdone to notify omxcodec
Compoment uses fillbufferdone to notify omxcodec that the outputport buffer has been filled. After omxcodec receives the command, it passes the decoded frame to misurface for Image Rendering. After drawing, omx_fillthisbuffer is used to notify component that the buffer is empty
The abstract diagram is as follows:The decoding in OMX Il is divided into two parts. The following uses AVC decoding as an example:
1) Use assemblepartialframes to assemble the buffer of the input port into a frame.
2) Transmit the frame to avcdecoder_omx for decoding and output it to the buffer of the output port.
For example: Assume that the input port buffer has two buffers: buffer_1 and buffer_2, and the data carried by these two buffers can constitute one frame. assemblepartialframes first applies for a memory area tmp_buffer_1, copy the valid data of buffer_1 to tmp_buffer_1, and then apply for a memory area tmp_buffer_2. After the application is completed, copy the data of tmp_buffer_1 to the first half of the application, in step 2, copy valid data of buffer_2 to the lower half and combine them into one frame.
After the combination is completed, the buffer of tmp_buffer_2 and output port is handed over to avcdecoder_omx for decoding. avcdecoder_omx copies the decoded frame data to the output port buffer. |
