Introduction to HDMI on Android (based on Qualcomm platform)

This article focuses on the application of HDMI on Android, And the overlay mechanism is more relevant. Overlay is a simple introduction here, and will be detailed in later articles.

If I remember correctly, the platform can support HDMI (1.3) output since. However, on 7x30, an HDMI transmitter is added through the RGB interface. In 8 series (8x60), Qualcomm also integrates this IC, the HDMI output is provided directly. (In this case, the peripheral devices will be gradually integrated, and the underlying estimation will be unemployment. It seems that there is not much work to do with hardware ).

Let's take a look at HW's capabilities, which is the structure of mdp4.0:

As you can see, mdp4 has four overlay pipe, two for UI (RGB), two VG for video and graphics, and two mixers, mixer1 is for primary display (it can be mddi interface, LCD or OLED of RGB interface); mixer2 is for external display, for example, an external HDMI transmitter can be used to connect to a TV through RGB interface2, or a analog TV signal such as NTSC/pal.

Note: vg1 and rgb1 are mixed into primary LCD by mixer1, and vg2 and rgb2 are mixed into external LCD by mixer2 (such as hdmi TV)

If mdp4.1 is used, the mddi interface is removed. In addition, there is only one RGB interface, and the other is integrated into the HDMI interface.

The above mentioned is related to the hardware platform. That is to say, the hardware has the ability to support HDMI output, but what about the software? Let's take a look at the situation of Android and Qualcomm.

I will not introduce the HDMI itself. You can find it on the Internet.

All Android users know that surfacefinger is responsible for managing the display windows of applications. Each window can be composed of multiple surfaces. surfaceflinger can use OpenGL (HW or SW) after all the surfaces are merged, the whole screen data is sent to the display device by calling the gralloc module or the overlay module (supported by mdp4.x. However, currently, only one display device is supported on Android (as of 2.2, 3.0 is unknown), that is, only one display device can be fixed on surfaceflinger, how can I use the HDMI application on Android phones?

Here we will introduce two practices. One is done by Qualcomm, called UI embedding ing and video embedding ing. The other is to add interfaces and implement the desired functions by the AP.

First, let's take a look at Qualcomm's practices in Android. It is literally not difficult to understand that UI tracking ing and Video Tracking ing are actually the data mirror originally displayed on the primary LCD to the HDMI interface. For the software framework:

First, let's take a look at the control of HDMI. On the right side, there is an HDMI service in the user space, which contains a listener (all in Java). After the HDMI cable is inserted, after the underlying HDMI driver detects (HPD), it sends the event to the HDMI daemon through kobject_uevent, And the daemon sends the event to the HDMI service. In addition to judging this event (Cable status), The HDMI Service ), in addition, you need to determine the on/off option of HDMI in Qualcomm setting, and then assign the result broadcast to each AP. Each AP also knows whether to enable HDMI output.

Next, let's take a look at the implementation of UI tracking ing (without video). It targets interface operations and the data is in RGB format. We know that each display device in the kernel corresponds to an FB, and framebuffer is allocated during initialization. Here, primary LCD corresponds to fb0 and HDMI corresponds to Fb1. Under normal circumstances, after surfaceflinger synthesize a main surface, it puts the data into fb0 through POST buffer (gralloc module), and then uses overlay (kernel, the upper layer is still seen through the IOCRL-FBIOPUT_VSCREENINFO command) output to primary
LCD; when the platform supports hdmi and UI compaction ing is enabled. CPP) an additional task (hdmi_ui_loop) will be created during initialization, and an overlay (mainly control and data, refer to overlaylib. h), the channel corresponding to this overlay is fixed to Fb1, Src FD is fb0, that is, the source data of this overlay is fb0, that is, the data on the primary LCD, rotate through rotator (the TV is a horizontal screen), scale up in overlay, and then deliver the TV through HDMI. In this case, the data sent to HDMI is actually a copy of the data copybit in fb0, which is somewhat distorted, but it is acceptable for the UI interface. Surfaceflinger is not involved in the above process.

How does video processing work?

First, let's take a look at what video playing ing is, in fact, playing a video on a mobile phone and outputting the video to the TV through HDMI. The content on the mobile phone is divided into two parts: one is the video itself and the other is the UI, this occupies two overlay pipe instances (one VG pipe and one RGB pipe). The video part on TV must have a VG pipe. In addition, due to the video size problem, the video cannot be in full screen mode, so an RGB pipe must be required to implement a background (full black ). Four pipelines are occupied, and there is no excessive pipe to upload the UI part to the TV. Therefore, when using the Qualcomm platform, video
During video embedding, only video images can be played on TV, and the UI (such as menus) cannot be displayed on TV.

Next, let's take a look at how video is handled? First, the processing mode of the UI on the mobile phone is unchanged, but the hdmi_ui_loop task mentioned above will be stopped (the UI does not need to be sent to HDMI, as explained above ); after the frame in the video part is decoded using opencore, the overlay will be created through surfaceflinger (refer to layerbuffer. CPP). When the system supports HDMI, create overlay will create two channels (two VG channels), including two control channels and two data channels, their Hal interfaces are overlaylib. in CPP, channel0
For fb0; channel1 for Fb1. If rotation is required, the corresponding memory is allocated from the system pmem. The overlay process in the AP is basically like this (you can refer to overlays. cpp, which is incomplete and I have added some ):

Sp <surfacecomposerclient> client = new surfacecomposerclient (); // create a surface Client

// Create pushbuffer Surface
Sp <surface> surface = client-> createsurface (getpid (), 0,320,240,
Pixel_format_unknown, isurfacecomposer: epushbuffers); // create a surface

// Get To The isurface
Sp <isurface> isurface = test: getisurface (surface); // obtain the surface interface.
Printf ("isurface = % P/N", isurface. Get ());

// Now request an overlay
Sp <overlayref> ref = isurface-> createoverlay (320,240, pixel_format_rgb_565); // create an overlay and obtain the Control CHannel
Sp <overlay> overlay = new overlay (REF); // initialize overlay and obtain the data channel

Overlay-> setfd (MFD); // sets the FD of SRC data

Overlay-> setcrop (X, Y, W, H); // you can specify the cropping information as needed)

Overlay-> queuebuffer (offset); // you can specify the offset of the displayed data.

In this way, if video player does not decode a frame, it will call the quene function to send the data to two data channels. The overlay engine will send the data to two display devices.

The background is not found yet in the code. It may be because the current version is not the final version and will be updated later.

The above practice is Qualcomm's, but it is restricted. For example, different content cannot be displayed on two screens. If we want to do this, we can do the same. It mainly depends on how the AP defines the rules. In addition, interfaces need to be added to the framework, which mainly provides a control interface for Fb1 devices, and also bypasses surfaceflinger. For example, when a video is played on a mobile phone, the video is transmitted to the TV through HDMI, and the mobile phone can browse the webpage. This function can be used to play a video on the AP background. The frame is output to the Fb1 device through the newly added interface instead of to the primary display device, and the browser UI is displayed normally. If it is implemented on the Qualcomm platform, you need to set Qualcomm
Disable the HDMI option in setting. Otherwise, Qualcomm and your AP will be confused. However, at present, the approach provided by Qualcomm seems to be able to meet the application, but the application is endless. As long as the user has such a requirement, the developer will do it!