Android Wi-Fi Display (Miracast) Introduction

Address: http://blog.csdn.net/innost/article/details/8474683Android Wi-Fi Display (Miracast) Introduction

In mid-November 2012, Google released Android 4.2. Although it is a Jelly Bean series with Android 4.1, it adds a lot of new features. In the display section, Android 4.2 builds on project butter and adds support for the Wi-Fi display feature. As a result, the entire display architecture has changed significantly.

This article first introduces the background of Wi-Fi display, and then combines the code to introduce the implementation of Wi-Fi display in Android 4.2.

A Background knowledge introduction

Wi-Fi display is often associated with miracast. In fact, Miracast is the authentication name of the Wi-Fi federation (Wi-Fi Alliance) for devices that support Wi-Fi display features. MIRACAST certified devices will maintain support and compatibility with the Wi-Fi display feature to the maximum extent possible. As a miracast, it is the Wi-Fi display (which will no longer differentiate between miracast and Wi-Fi display). The core feature of Wi-Fi display is the ability to share AV data between devices over a Wi-Fi network. Take a simple scenario: with Wi-fi display, the phone and the TV can be directly connected to Wi-Fi, without the need for hard-wired (such as HDMI) to post the video on the phone to the TV to display [①]. With the current trend of smart devices, Wi-Fi display is likely to help us achieve a truly multi-screen interaction in a relatively short period of time.

Technically speaking, Wi-Fi display is not a start-up, but rather leverages the existing Wi-Fi technology. Figure 1 shows the other Wi-Fi technology items used in the Wi-Fi display.

Figure 1 Support architecture for Miracast

As shown in Figure 1, the miracast-dependent Wi-Fi technology Item [②] has:

• Wi-Fi Direct, which is Wi-Fi peer. It supports two Wi-Fi devices to connect and communicate directly without an AP (Access point).
• Wi-Fi Protected Setup: Used to help users automatically configure Wi-Fi networks, add Wi-Fi devices, and more.
• 11N/WMM/WPA2: The 11n is the 802.11N protocol, which increases the Wi-Fi transmission rate from 56Mbps to 300 or even 600Mbps for 11a and 11g. WMM is the abbreviation for Wi-Fi multimedia and is a QoS service for real-time AV data. WPA2, the second edition of Wi-Fi Protected acess, is mainly used to encrypt the transmitted data.

Most of the above Wi-Fi technologies are implemented by hardware vendors. In Android, the most important thing for Miracast is the two basic technologies:

• Wi-Fi Direct: This feature is managed and controlled by Wifip2pservice in Android.
• Wi-Fi Multimedia: The multimedia system has also been modified to support miracast,android 4.2.

Below we introduce a few important points of knowledge of Miracast, first of all, the topological structure and audio-visual format content.

One important feature of Miracast is the ability to support Wi-Fi Direct. However, it also considers the interconnection problem between devices in the wireless network environment where AP devices exist. Readers can refer to the four topologies shown in 2.

Figure 2 The four topologies of Miracast

The content shown in Figure 2 is relatively simple and is not detailed here. In addition, in the Wi-Fi display specification, there is a case where source transmits video and audio content to different render device separately. Interested readers can refer to the Wi-Fi display technical specification.

In addition, Miracast provides the supported AV formats, as shown in table 1.

Table 1 Miracast Visual audio format support

Resolution	17 CEA formats with resolution from 640*480 to 1920*1080 with frame rates from 24 to 60 29 VESA formats with resolutions from 800*600 to 1920*1200 with frame rates from 30 to 60 12 handheld formats with resolutions from 640*360 to 960*540 with frame rates from 30 to 60
Video	H + HD
Audio	Required: LPCM 16bits,48khz sample rate, dual channel Optional: LPCM 16bits,44.1khz sample rate, dual channel Advanced Audio Coding Dolby Advanced Codec 3

Finally, let's briefly introduce the general work flow of miracast. Miracast the task of managing the interaction between two devices in session, the main steps are (in order):

• Device Discovery: Use Wi-Fi peer to find nearby Wi-Fi-enabled devices.
• Device Selection: When device a discovers device B, a devices need to prompt the user. Users can choose whether to pair with device B as needed.
• Connection Setup:source and display devices connect via Wi-Fi peer. Depending on the Wi-Fi Direct Technical specification, this step involves establishing a group owner and a client. Thereafter, the two devices will establish a TCP connection, and a port for the RTSP protocol will be created for subsequent session management and control work.
• Capability negotiation: Before the formal transmission of AV data, the source and display devices need to exchange some miracast parameter information, such as the audio and video formats supported by both parties. After the negotiation is successful, the subsequent process can continue.
• Session establishment and streaming: After the previous work is completed, the source and display devices will establish a miracast Session. You can then start transmitting AV data. The source side of the AV data is encoded via MPEG2TS and passed through the RTP protocol to the display device. The display device decodes the received data and eventually displays it.
• User Input back Channel setup: This is an optional step. It is mainly used to handle some control actions initiated by the user during the transfer process. These control data will pass through TCP between the source and display devices.
• Payload Control: During transmission, the device can dynamically adjust the transmission data and format according to the strength of the wireless signal and even the power status of the device. Adjustable content includes compression rate, audio format, resolution, and more.
• Session teardown: Stop the whole session.

Through the introduction of the above background knowledge, the reader can find:

• Miracast is essentially a Wi-Fi-based network application. This application includes the server side and the client.
• The server and client must support network protocols such as RTP/RTSP and corresponding codec technologies.

Two Android 4.2 miracast feature Implementation Introduction

Miracast's Android implementation involves multiple modules of the system, including:

• Mediaplayerservice and related modules: The reason is obvious, because the miracast itself is involved in RTP/RTSP and corresponding coding and decoding technology.
• Surfaceflinger and related modules: The role of Surfaceflinger is to mix each layer of UI data and post it to the display device to display. Now, Surfaceflinger will support multiple display devices. Remote devices that support Miracast are also present in the system as a standalone display device.
• Windowmanagerservice and related modules: Windowmanagerservice is used to manage the location and properties of each UI layer in the system. Not all UI layers will be posted to the remote device via Miracast. For example, the video in the phone can be posted to the remote device to display, but if during the playback process, suddenly pop up a password input box (perhaps a background application initiated), the password input box can not be delivered to the remote device to display. Therefore, Windowmanagerservice also needs to be modified to adapt to the needs of miracast.
• Displaymanagerservice and related modules: The Displaymanagerservice service is new for Android 4.2 to manage all display devices in the system.

This article will focus on the dynamic workflow of Surfaceflinger and Displaymanagerservice as well as miracast for space reasons.

2.1 Surfaceflinger's support for Miracast

The biggest change in Surfaceflinger in Android 4.2 compared to the previous version is the addition of an abstraction layer called Displaydevice. The related structure 3 shows:

Figure 3 Surfaceflinger Family class diagram

The figure 3 shows that:

• The surface system defines an enumeration of DisplayType that represents the display_primary of the phone screen and the display_external that represents the HDMI external device. Interestingly, as Wi-Fi Display, its device type is display_virtual.
• Then look at the Surfaceflinger class, which has a variable named Mdisplays that holds all of the current display devices (Displaydevice) in the system. In addition, Surfaceflinger controls the status of the display layer through mcurrentstate and mdrawingstate. Where mdrawingstate is used to control the state of the display layer currently being drawn, Mcurrentstate represents the state of all current display layers. The reason for these two state display layers is that the time that is present in the system is indeterminate, whether it is a miracast or an HDMI device. For example, users can choose to connect a miracast display device at any time. In order not to destroy what is currently being displayed, some information about this new display device will be saved to CurrentState. Wait until Surfaceflinger next time to mix screen before concentrating on processing.
• The types of mcurrentstate and mdrawingstate are the inner class state of Surfaceflinger. As shown in Figure 3, state first holds an array of display layers sorted by the z-axis by the Layersortedbyz variable (in Android, the base class for the display layer is layerbase). In addition, the corresponding displaydevicestate of each display layer is saved by displays variable.
• The function of displaydevicestate is to preserve the properties of the displaydevice corresponding to the display layer and a isurfacetexure interface. This interface will eventually be passed to Displaydevice.
• Displaydevice represents the display device, which has two important variables, one of which is mframebuffersurface and Mnativewindow. Mframebuffersurace is a framebuffersurface type, and the variable is not empty when the display device is not of type virtual. For Miracast, the display data is passed through the network to the real display device, and all surfaceflinger on the source side, there is no framebuffer. Therefore, when the device is virtual, its corresponding mframebuffersurface is empty. And Anativewindow is an older employee of the Android display system. The structure in the multimedia video I/O, OpenGL es and other places used more. In the normal UI drawing, the Isurfacetexture interface is used more. However, as early as the Android 2.3,google developers through the function pointer to the Anativewindow of the various operations and Isurfacetexture interface unified together.

How does the Miracast device as virtual add to the surface system by displaydevice this layer of abstraction? The following code is very important to understand the abstract role of Displaydevice. As shown in 4.

Figure 4 Surfaceflinger Code Snippet

The code in Figure 4 shows:

• For non-virtual devices, the Displaydevice framebuffersurface is not empty. Moreover, the construction parameters of Surfacetextureclient are derived from the Getbufferqueue function of Framebuffersurface.
• In the case of virtual devices, surfacetextureclient directly uses the surface variables that are carried in the state information.

With these two points above, we can infer the effect of the displaydevice shown in 5.

Figure 5 Isolation of the Displaydevice

Finally, take a look at the implementation of the mixed-screen operation in Surfaceflinger, as shown in code 6:

Figure 6 mixed-screen operation of Surfacefilnger

As shown in Figure 5, Surfaceflinger will traverse all the displaydevice in the system to complete the respective mixing screen.

2.2 Framework support for Miracast

To completely solve the problem of multi-display devices, Android 4.2 simply added a new service called Displaymanagerservice in the framework to unify the display devices in the system. Displaymanagerservice interacts with several other services in the system. As shown in the overall structure 7.

Figure 7 Displaymanagerservice and related class diagram

The Figure 7 shows that:

• Displaymanagerservice mainly implements the Idisplaymanager interface. Most of the functions of this interface are related to the Wi-Fi display operation.
• In addition, Displaymanagerservice and Windowmanagerservice interact closely. Because Windowmanagerservice manages all UI displays, including attributes, z-axis locations, and so on. Furthermore, Windowmanagerservice is an important channel for internal and surfaceflinger interactions within the system.
• Displaymanagerservice interacts with Displaydevice through Mdisplayadapters. Every displaydevice should have a displayadapter.
• The system defines four kinds of displayadapter. Headlessdisplayadapter and Overlaydisplayadapter are for fake devices. The overlaydisplay is used to help developers simulate multiple screens. Localdisplayadapter represents the Home screen, while Wifidisplayadapter represents the Wi-Fi Display.

2.3 Android Miracast Dynamic Workflow Introduction

When the user chooses to open Miracast from the settings program and finds a matching device [③], the system will initiate a connection to the matching devices through the Requestconnect function of Wifidisplaycontroller. Code 8 shows the following:

Figure 8 Requestconnect function implementation

In Figure 8, the Connect function is eventually called to connect to the specified device. In the Connect function comparison, the most important of these is the Updateconnection function, which we extract from some of the code, as shown in 9:

Figure 9 Updateconnection function fragment

In the code shown in Figure 8, the system creates a remotedisplay and listens on the display (listen). From the note, the Remotedisplay is the RTP/RTSP channel that interacts with the remote device. Also, once a remote device connection is available, a surface object is returned via ondisplayconnected.

According to Surfaceflinger's introduction, the reader can guess Miracast's first show is on the surface of Remotedisplay and its return.

Indeed, Remotedisplay will call Mediaplayerservice's Listenforremotedisplay function and eventually get a native Remotedisplay object. The related class is shown in Figure 10.

Figure Ten Remotedisplay class diagram

As shown in Figure 10, Remotedisplay has three important member variables:

• Mlooper, pointing to a Alooper object. This indicates that Remotedisplay is a system based on the dispatch and processing of messages.
• Mnetsession points to a Anetworksession object. From its API view, Anetworksession provides most of the network operations.
• Msource points to a Wifidisplaysource object. It derives from Ahandler, so it is the processor of the message in Mlooper. Note that the M1, M3, M5, and so on in the figure are the message names specified in the Wi-Fi display specification.

In the Remotedisplay constructor, the Wifidisplaysource start function is called. Thus, a message of type Kwhatstart is added to the message queue. The message is eventually processed by Wifidisplaysource, and the result is that a rtspserver is created. Code 11 shows the following:

Figure One-Kwhatstart message processing results

Later, the data sent by the client is added to the system through a message of type kwhatrtspnotify. The processing core of this message is in the Onreceiveclientdata function, which encompasses all the details of the network interaction between devices. Its core code 12 is as follows:

Figure Onreceiveclientdata Core Code schematic

The content of Figure 12 is more, it is suggested that readers should study according to their needs.

Based on the previous background knowledge, the interaction between the devices will be managed by the session. In code, the concept of a session is represented by Wifisource's inner class playbacksession. Let's look at the class diagram structure associated with it, as shown in 13:

Figure Playbacksession and related class diagram

The figure 13 shows that:

• Playbacksession and its inner class track are derived from Ahandler. Therefore, their work also relies on message loops and processing. Track represents a video stream or audio stream.
• The track internally points to a Mediapull object through the Mmediapull variable. The Mediapull object, however, holds a MediaSource object. In Playbacksession, the true type of this mediasource is Surfacemediasource. It shows that the source of the media comes from surface.
• Bufferqueue is derived from Isurfacetexure, which is the surface variable that represents the state of the virtual appliance in the Surfaceflinger code example, as described in the previous face Surfaceflinger.

When both devices are ready, Mediapull calls MediaSource's read function through Kwhatpull message processing. In the Read function implemented by Surfacemediasource, the mixed-screen data from Surfaceflinger is passed through Bufferqueue to Mediapull. Code 14 shows the following:

Figure Mediapull and Surfacemediasource's code schematic

From Figure 13:

• On the left, Mediapull calls the Read function of MediaSource continuously by kwhatpull messages.
• In the image on the right, Surfacemediasource's read function reads data through Mbufferqueue.

So where does the mbufferqueue data come from? Yes, it is from the surfaceflinger of Figure 4.

Of course, playbacksession will need to encode the data before it can be sent to the remote device. Because of the length of the relationship, this article will not discuss these issues.

Three summary

In this paper, the background knowledge of miracast and the implementation of Miracast in Android system are briefly introduced. From the author's personal point of view, there are several interesting readers to note:

• Be sure to combine Wi-Fi related protocols to understand Miracast. Key concerns include Wi-Fi peer and WMM.
• The implementation of Android Miracast requires a focus on understanding the Surfaceflinger and Remotedisplay modules. This part of the implementation is not only a large amount of code, but also between classes, and the relationship between threads is complex.
• Other points to be aware of are displaymanagerservice and related modules. This section contains APIs in the SDK. App developers should focus on whether these new APIs can help them develop more innovative applications.

In addition, Android has evolved very fast, especially on several important functional points. The authors also hope that domestic handset makers or those interested in mobile internet companies will be able to really put in the power to do some more depth and value of research and development work.

[①] Apple's air play technology and DLNA technology also implement similar scenarios. Readers interested in them can refer to the author's blog post http://blog.csdn.net/innost/article/details/7078539.

[②] Other optional technology items are Tdls and WMM Power Save. This article does not discuss these two things

[③] This part of the content and the Wifip2pservice combination of close, interested readers can self-study.

Android Wi-Fi Display (Miracast) Introduction