Reprint please the head source link and the tail two-dimensional code together reprint, this article from countercurrent fish yuiop:http://blog.csdn.net/hejjunlin/article/details/52629598
Foreword: The OMX event callback is introduced in the previous article, starting from today, go into the OpenMAX frame of codec part. Look at today's agenda as follows:
- A picture review audio and video synchronization
- A picture of the location of OpenMAX in the Android system
- What is OpenMAX?
- OpenMAX IL Introduction
- OpenMAX IL structure
- Usage of OpenMAX in Android
- The interface and implementation of OpenMAX
- OpenMAX adaptation Layer in Android
- MP3,AAC decoding sequence diagram
A picture review audio and video synchronization
A picture of the location of the OpenMAX in the Android system:
What is OpenMAX?
The following is the official website translation:
- OpenMAX? is an unlicensed, cross-platform application interface API that provides a full range of streaming media codecs and application portability by enabling media acceleration components to be implemented across multiple operating systems and processor hardware platforms in the development, integration, and programming phases. The
- OpenMAX API will be provided with the processor to enable libraries and codec developers to take advantage of the full acceleration potential of the new device at high speed-without worrying about its underlying hardware structure. The
OpenMAX is divided into 3 layers:
- First layer: OpenMAX DL (development layer, development layer)
- OpenMAX DL defines an API that is audio, video A collection of frequency and image functions. The vendor is able to implement and optimize on a new processor, and then the codec vendor uses it to write a wider range of codec functions. It includes processing functions for audio signals such as FFT and filter, image raw processing, such as color space conversion, video raw processing, to achieve optimizations such as MPEG-4, H, MP3, AAC, and JPEG codecs.
- second tier: OpenMAX il (integration layer, integration layer)
- OpenMAX IL as audio, video, and image codec for multimedia codecs Interact and support components (for example, resources and skins) in a unified behavior. These codecs may be a mixture of hardware and software, and the user is transparent to the underlying interface applied to embedded, mobile devices. It provides an application and a media framework that is transparent. The S codec vendor must write a private or closed interface that integrates into the mobile device. The main purpose of IL is to use feature sets to provide a system abstraction for codecs to solve the problem of portability between multiple different media systems.
- Third layer: OpenMAX al (appliction layer, Application layer)
- OpenMAX AL API provides a standard between application and multimedia middleware Quasi-interface, multimedia middleware to provide services to achieve the desired API capabilities.
This article is derived from the countercurrent fish yuiop:http://blog.csdn.net/hejjunlin/article/details/52629598
Three levels of OpenMAX (from OpenMAX website):
Hint: In the actual application, the OpenMAX three level uses more is the OpenMAX IL integration layer, because the operating system to the hardware difference and the multimedia application difference, OpenMAX's DL and the AL layer use relatively few.
OpenMAX IL Introduction
OpenMAX IL is positioned in the middle tier, OpenMAX il as audio, video and image codecs can interact with multimedia codecs and support components (such as resources and skins) in a unified behavior. These codecs may be a mixture of hardware and software, and the underlying interface for users is applied to embedded or/and mobile devices. It provides an application and a media framework that is transparent. This standardized interface does not exist in nature, and the codec vendor must write a private or closed interface that integrates into the mobile device. The main purpose of IL is to use feature sets to provide a system abstraction for codecs to solve the problem of portability between multiple different media systems.
The purpose of OpenMAX IL is to provide an abstraction layer for the graphics and audio and video of the hardware platform, which can provide a cross-platform support for the application of the upper layer. This is important for cross-platform media applications. The audio and video interfaces of different manufacturers are similar in function, but the interface design and usage are different, and there are many differences. If you want to make your own media application perfect running on different hardware vendors platform, you have to adapt to the underlying decoding interface of different chips. This is very cumbersome for application development. So you need an interface specification similar to OpenMAX IL. When it comes to audio and video related functions, the application simply calls these standard interfaces without needing to be concerned about hardware-related implementations under the interface. If you change the hardware platform, you just need to adapt the interface layer and hardware to fit the line. Upper-level applications do not require frequent changes. You can think of OpenMAX il as a porting layer interface in middleware, but now most of the middleware is home-defined.
This article is derived from the countercurrent fish yuiop:http://blog.csdn.net/hejjunlin/article/details/52629598
OpenMAX IL structure
OpenMAX IL hierarchy:
- The content in the dashed line is the content of the OpenMAX IL layer, which mainly implements the various components (Component) in OpenMAX il. On the lower level, OpenMAX il can call the interface of the OpenMAX DL layer, or it can invoke various codec implementations directly. On the upper level, OpenMAX il can be called to the OpenMAX AL layer (middleware) or to the application directly.
The main contents of OpenMAX Il are as follows.
- Client: Caller of OpenMAX IL
- Component (Component): OpenMAX il Unit, each component implements a function
- PORT: Input/output interface of the component
- Tunneling (tunneled): A way to connect two components directly
components, ports, tunneling ideas and GStreamer (a multimedia framework) are very similar in pipeline.
The component implements a single function, or source, Host, Accelerator, and sink.
Port is the component external input output.
A single component is concatenated together to form a complete function through tunneled.
OpenMAX Core is the part that assists individual components to run
The basic operation process of OpenMAX IL
, the client of OpenMAX IL implements a function by invoking four OpenMAX IL components. The four components are the source component, the host component, the accelerator component, and the sink component. The source component has only one output port, while the host component has an input port and an output port, and the accelerator component has an input port that calls the hardware codec, which is mainly reflected in the acceleration. Accelerator components and sink components are internally connected via private communication, without a clear component port.
When Openmal il is used, its data flow also has different processing methods:
- It can be either through the client or not through the client.
- Figure, the source component to the host component of the data flow is through the client;
- and the host component to the accelerator component of the data flow is not through the client, the use of a tunneling method;
- Accelerator components and sink components can even use private communication methods.
OpenMAX Core is the part that assists the individual components, it usually requires the initialization of each component, and in the real run, the focus is on the components of each OpenMAX IL, openmax core is not the focus, nor is it a standard.
- The Openmal IL component is the core of the OpenMAX IL implementation, where one component interfaces with the input and output ports and the port can be connected to another component. External to the component can send commands, also set/get parameters, configuration and so on. The port of a component can contain a queue of buffers (buffer).
- The core of the processing of the component is that buffer is consumed through the input port, and buffer is filled through the output port, so that the multi-component join can form the flow-type processing.
This article is derived from the countercurrent fish yuiop:http://blog.csdn.net/hejjunlin/article/details/52629598
Openmal the structure of a component in IL:
The functionality of a component is closely related to the type of port it defines, typically:
- Only one output port, for the source component;
- Only one input port, for the sink component;
- There are multiple input ports, one output port for the MUX component;
- There is an input port, multiple output ports for the Demux component;
- Input and output ports each component is an intermediate processing link, which is the most common component.
- There are different types of data supported by the port. For example, for an input, output port each component, its input port uses MP3 format data, the output port uses the PCM format data, then this component is a MP3 decoding component.
- Tunneling (tunneled) is a concept of how components are connected. By tunneling, one input port and one output port of different components can be connected together, in which case the process of two components is merged and processed together. Especially for single-input and single-output components, two components will be used as a similar one.
Usage of OpenMAX in Android
- Some parts of the Android system use the OpenMAX IL layer, basically using the standard OpenMAX IL layer interface, just a simple encapsulation. The standard OpenMAX IL implementation is easily added to the Android system as a plugin.
- Android Multimedia engine Opencore and stagefright can use OpenMAX as a multimedia codec plug-in, but not directly using the OpenMAX IL layer provided by the pure C interface, but it is a certain encapsulation (C + + package).
- After the android2.x version, the Android framework also encapsulates the interface of the OpenMAX IL layer, even using the binder IPC mechanism in Android. Stagefright uses this level of interface, Opencore is not used.
- Opencore uses the OpenMAX IL layer as the codec plug-in before the Android Framework layer encapsulation OpenMAX interface is introduced in later versions.
What Android OpenMAX implements
- The main use of Android is the OpenMAX codec feature. Although OpenMAX can also generate components such as input, output, file parsing-build, and so on, it is the most or codec component used in each system (not just Android). Media input, output links and systems are very large, the introduction of OpenMAX standard is more troublesome; file parsing-The construction process generally does not require hardware acceleration. Codec components are also the most capable of hardware acceleration, and therefore most commonly used.
- The implementation of the OpenMAX IL Layer and the standard OpenMAX IL layer in Android are basic, generally need to implement the following two links.
- Codec driver: In Linux kernel space, drivers need to be called through the Linux kernel, typically using non-standard drivers.
- OpenMAX IL layer: implements the components of different functions according to the standard header file of the OpenMAX IL layer.
Android also offers the OpenMAX adapter layer interface (packaged to fit the standard components of OpenMAX IL), which acts as an interface to the Android native layer and can be called by the Android multimedia engine.
interface of OpenMAX with implementation of OpenMAX IL layer Interface (1)
The interface definition for the OpenMAX IL layer consists of a number of header files, which is what is required to implement it, and their basic descriptions are as follows.
Tip: The OpenMAX standard has only header files, no standard libraries, and no function interfaces defined. For the implementation, the main implementation is the structure containing the function pointer.
Among them, the omx_componenttype struct defined in Omx_component.h is the core of the OpenMAX IL layer, which represents a component that reads as follows:
OpenMAX IL layer Interface (2)
- After the Omx_componenttype struct is implemented, each function pointer is what the caller can use. Each function pointer corresponds to the content defined in the omx_core.h.
Emptythisbuffer and Fillthisbuffer are the basic mechanisms that drive components to run, which means that the component consumes buffers to represent the contents of the corresponding component inputs, while the latter indicates that the component fills the buffer, representing the contents of the corresponding component output.
- Usebuffer,allocatebuffer,freebuffer is a port-related buffer management function that, for a component's port, can allocate its own buffers, some can use an external buffer, and therefore have different interfaces to manipulate it.
- SendCommand represents a command that sends a control class to a component. Getparameter,setparameter,getconfig,setconfig several interfaces for auxiliary parameters and configuration settings and fetches.
The componenttunnelrequest is used for tunneling between components, where two components and their connected ports need to be developed.
- The componentdeinit is used for the inverse initialization of the component.
Tip: The parameters of the OpenMAX function often contain macros such as omx_in and Omx_out, whose actual contents are empty, just to mark the direction of the parameter as input or output.
The definition of the port type in Omx_component.h is the Omx_portdomaintype enumeration type, as shown in the following:
Audio type, video type, image type, other type is the type of four ports defined by this OpenMAX il layer.
The definition of a port's content is represented by the Omx_param_portdefinitiontype class (also defined in omx_component.h), and its contents are as follows:
For a port, the focus of the content is as follows:
- Port direction (Omx_dirtype): contains both omx_dirinput (input) and--omx_diroutput (output)
- Number of buffers allocated by port and minimum number of buffers
- Type of port (Omx_portdomaintype): Can be four types
- Data structures in port format: represented by the format union, represented by four different types, corresponding to the type of the port
Omx_audio_portdefinitiontype,omx_video_portdefinitiontype,omx_image_portdefinitiontype and Omx_other_ Several specific format types, such as Portdefinitiontype, are defined in the four header files of omx_audio.h,omx_video.h,omx_image.h and Omx_other.h respectively.
Omx_bufferheadertype is defined in Omx_core.h, which represents the head structure of a buffer.
The enumeration type Omx_statetype command defined in Omx_core.h represents the state machine for OpenMAX, as follows:
The state machine of a OpenMAX component can be changed by an external command or by an internal occurrence. The migration relationship of the state machine of the OpenMAX IL component:
The enumeration type defined in Omx_core.h omx_commandtype represents the command type for the component, as shown in the following:
The Omx_commandtype type is used as a parameter in the SendCommand call, where Omx_commandstateset is the command that alters the state machine.
Contents of OpenMAX IL implementation
For the implementation of the OpenMAX IL layer, the general approach does not call the OpenMAX DL layer. The specific implementation of the content is the various components. The implementation of the OpenMAX IL component consists of the following two steps.
- Initialization functions for components: initialization of hardware and OPENMAX data structures, general partitioning of function pointers, initialization of private data structures, initialization of ports, and the use of Pcomponentprivate members to preserve the private data of this component as the context, Finally, the structure that fills the completed Omx_componenttype type is obtained.
- Omx_componenttype the various pointer implementations of the type struct: Implement each function pointer, need to use private data, from which the pcomponentprivate to get pointers, converted into actual data structure use. A port is defined as an external interface to the OpenMAX IL component. Most of the components commonly used in OpenMAX il are one of the input and output ports. For the most commonly used codec (CODEC) components, it is often necessary to invoke the hardware's codec interface for each component implementation. In internal processing of components, threads can be established for processing. The port of the OpenMAX component has default parameters, but it can also be set at run time, so a port can also support different encoding formats. The output of the audio encoding component and the input of the audio encoding component are usually the original data format (PCM format), the output of the video encoding component, and the input of the video encoding component are usually the original data format (YUV format).
Tip: In a particular hardware implementation, the codec section is similar, so you can often build a "base class" or public function of a OpenMAX component to perform public operations.
OpenMAX adaptation Layer in Android
The interface of the OpenMAX adaptation layer in Android is defined in the IOMX.h file in the frameworks/base/include/media/directory, and its contents are as follows:
- IOMX represents a component of the OpenMAX, according to the Android binder IPC mechanism, BNOMX inherit IOMX, the implementation needs to inherit the implementation bnomx. The Iomx class, in addition to the standard OpenMAX Getparameter,setparameter,getconfig,setconfig,sendcommand,usebuffer,allocatebuffer, In addition to interfaces such as Freebuffer,fillthisbuffer and Emptythisbuffer, the interface for creating renderers Createrenderer () is included, and the interface created is the Iomxrenderer type.
- Only the first createrenderer () function in IOMX is a pure virtual function, the second createrenderer () function and Createrendererfromjavasurface () are called by the first createrenderer ( ) function implementation.
The Iomxrenderer class represents a OpenMAX renderer, which is defined as follows:
- The iomxrenderer contains only a render interface whose parameter type, iomx::buffer_id, is actually void*, using different types depending on the renderer.
- In the IOMX.h file, there is also a iomxobserver that represents the Observer class, which represents the observer of OpenMAX, which contains only one onmessage () function, with the parameters omx_message the interface body, which contains the event type, Fillthisbuffer Complete and Emptythisbuffer complete several types.
Tip: The adaptation layer of OpenMAX in Android is OpenMAX il layer, which is called by Stagefright in Android system, and can be called by other parts.
MP3 AAC format decoding sequence diagram
The first time to get blog update reminders, as well as more Android dry, source code Analysis , Welcome to follow my public number, sweep the bottom QR code or long press to identify two-dimensional code, you can pay attention to.
If you feel good, easy to praise, but also to the author's affirmation, can also share this public number to you more people, original not easy
Android Multimedia Framework Summary (13) codec part of OpenMAX framework and its interface and adaptation layer realization
