Android text rendering

Android text rendering

A common method for rendering text using OpenGL is to calculate a texture image that contains the displayed text, which usually uses a complicated packaging algorithm to minimize the redundant part of the texture, before creating such an image, you must be clear about the font used when the application is running, including the font shape, size, and other attributes.

On Android, generating text texture images in advance is not practical, because there is no way to know in advance which fonts and fonts are used by the application, and the application can even load custom fonts at runtime, this is one of the most restrictive factors. Android font rendering must implement the following:

You must be able to create a font cache at runtime. You must be able to process a large number of fonts. You must be able to process a large number of fonts, minimize the waste of textures, and quickly run on low-end and high-end devices. /GPU combination for normal operation for text rendering

Before we can see how low-level OpenGL font rendering works, we should start with the high-level APIs directly used by applications. These APIs are very important for understanding libhwui.
Text APIs
The application mainly uses the following four APIs to layout and draw text:

Android. widget. textView: A view android used to process layout and rendering. text. * create a set of formatted text and layout classes for android. graphics. paint is used to measure text (to measure text) android. graphics. canvas is used to render text (to render text)

TextView and android. text are implemented at a high level on the painting and Canvas. Until Android 3.0, Paint and Canvas were implemented directly on Skia. Skia is a software rendering library that provides a very good Freetype (very popular text raster open source code) abstraction.

VcHLNC40uf2zzLHktcO4/kernel + kernel/kernel +/R1LXEudi8/kernel +/kernel/LV38/Cw + kernel "" alt = "Android" hardware = "" rendering = """ src = "http://www.bkjia.com/uploads/allimg/150521/04151MK0-1.png" text = "" title = "\"/>

This means that when you call Canvas. when drawText () is used, the openGL rendering engine does not directly or indirectly receive the parameters you have sent, but receives the array of font identifiers and x/y positions.

Raster and Cache

Each draw call of font rendering is associated with a single font. font is used to cache various fonts, And the fonts are stored in a cache texture ), this cache structure can be used to contain font fonts. Cache texture is a very important object. It has multiple caches: idle Block List, pixel cache, OpenGL texture handle, and vertex cache (the mesh, mesh, which constitutes the basic unit of graphics ).

The data structure used to store these objects is quite simple:

Fonts is stored in the LRU Cache of the font Renderer. The Glyphs are stored in the map of each font, and the key is the glyph identifier Cache textures (Cache structure) the pixel cache of the idle position of the trace linked list block is the uint8_t or uint_32t array (cache of alpha and RGBA). The mesh has two attributes: the x/y position and the u/v coordinate texture are a GLuint type handle.

When the font Renderer is initialized, it creates two types of cache textures: alpha and RGBA. Alpha textures is used to store normal fonts. Because the Font does not contain color information, we only need to store anti-aliasing information. The RGBA cache is used to store emojis.
For each type of cache texture, the font Renderer creates some CacheTexture instances of different sizes. The cache size varies depending on the device. Below are some default sizes:

1024x512 alpha cache 2048x256 alpha cache 2048x256 alpha cache 2048x512 alpha cache 1024x512 RGBA cache 2048x256 RGBA cache

When a CacheTexture is created, the underlying cache is not automatically allocated, and the font Renderer allocates it only when needed. In addition to the 1024x512alpha cache, it is always allocated.

The glyphs are packaged in the form of columns in textures. When the Renderer encounters a non-cached glyph, it searches for the appropriate CacheTexture cache glyph in the order listed above.

This is where the block list is used. It contains the current allocation column and available free location of the cache texture. If the glyph matches the existing column, it is added to the end of the space occupied by the column.

If all columns are occupied, a new column is divided on the left of the remaining space. Since few fonts are of equal width, the Renderer assigns a width multiple of 4 pixels for each font. This is a good compromise between column reuse and texture packaging. Packaging is not optimal, but it provides fast implementation.

All the glyphs are surrounded by an empty border of pixels, which are stored in textures. For font textures with dual linear sampling, doing so can avoid manual processing.

It is very important to know whether to perform dimensional transformation during text rendering. The transformation is submitted to Skia/Freetype, which means the fonts in cache textures store the transformation, this improves the rendering quality in terms of performance consumption. Fortunately, text is rarely scaled dynamically, and even a few fonts are affected. There are other paint attributes that can affect the raster and storage of the font: Imitation bold, text skew, X scaling, style and line width.

Pre-Cache

Because libhwui is delayed rendering, It is opposite to the direct mode of Skia. At the beginning of a frame, it is known that it needs to be drawn to the font of the screen, the font Renderer displays as many pre-cached fonts as possible when the list is sorted. The advantage of doing so is to completely avoid or minimize the number of texture uploads in the middle of the frame. Texture upload is a resource-consuming operation that causes the CPU or GPU to pause. Worse, modifying texture in the middle of the frame will cause serious memory pressure on some GPUs.

When the ImaginationTech PowerVR sgx gpu is modified in the middle of the frame, it will force the driver to copy the texture of each modification, because some font texture is very large, if you accidentally upload texture, it is more likely to cause oom. Google paly has a calculator application on it. It uses mathematical symbols and numbers to draw buttons. oom may occur during font rendering, because the button is drawn one by one, each draw will trigger texture upload, which leads to the entire font cache copy, the system does not have so much memory to store so many cached copies.

Cache refresh

The textures used to cache the glyphs is quite large, so when other applications require more memory, they may be recycled by the system. When the user hides the current application, the system sends a message to notify the application to release as much memory as possible. In most cases, the application will destroy the largest textures cache. On the Andorid device, all the textures caches except the first created (1024x512 by default) are considered as large ones.

When no cache has space, textures will be refreshed, And the font Renderer will keep a LRU (least recently used) list, but will not use it for anything, if necessary, cache refresh is more intelligent by refreshing memory that is rarely used. Currently, this is not required, but remember that it is a potential Optimization Direction.

Batch Processing and merge

Android 4.3 introduces batch processing and merge of painting operations. An important optimization is to greatly reduce the number of commands passed to the OpenGL driver.

To merge, the font Renderer caches the text geometric data of multiple drawing commands. Each cached texture has a client array containing 2048 Quadrilateral (1 Quadrilateral = 1), all of which share an independent index buffer (VBO exists in the GPU ).

When libhwui executes a text drawing command, the text Renderer obtains the appropriate mesh (mesh, which forms the basic unit of the image) for each glyph and writes the position and u/v coordinates into it. Mesh will be transferred to the GPU after a batch ends or the quad cache is full. Rendering a single string may require multiple mesh. Each cached texture has a mesh.

This optimization is easy to implement and will greatly improve the performance. Since the text Renderer uses multiple cached texture, it is possible that most of the characters in the string are part of one texture, and some fonts are part of another texture. If no batch processing and merge operations are performed, a drawing command is submitted to the GPU every time the text Renderer needs to switch to a different cached texture.

I have encountered this problem in a test app. This app only renders "hello world" with different styles and sizes ", when the "o" letter and other fonts are different in texture. This will cause the text Renderer to render "hell" first, then "o", then "w", then "o", and then "rld ", A total of five drawing commands were executed and five texture bindings were executed. In fact, both of them were required twice. The current Renderer first draws "hell w rld" and then draws two "o" in one piece ".

Optimize texture upload

As mentioned above, the text Renderer tries to upload as little data as possible when updating the cached texture by tracking the changed part of each texture (dirty rect ), unfortunately, this method has two restrictions.

First, OpenGL ES2.0 does not allow uploading a random subrectangle. GlTexSubImage2D allows you to specify the x/y and width/height of the rectangle to update part of the texture, but it is assumed that the stride (stride) that exists in the main memory data is the width of the rectangle. This can be achieved by creating a CPU cache of the appropriate size, provided that the size of the changed rectangle is known. A better compromise is to upload a smallest pixel band containing dirty rect. Because the width of this pixel band is the same as that of texture, we may waste some bandwidth, but it is better than uploading the entire texture.

The second problem is that texture upload is a synchronous operation, which may lead to a long cpu wait, but it has little impact on the cache. However, for applications that use a lot of text or a language environment with a lot of fonts, such as Chinese, this problem may be perceived by users.

Fortunately, OpenGL ES3.0 provides a solution to these two problems. Now we can use the new Pixel storage attribute GL_UNPACK_ROW_LENGTH to upload the word rectangle. This attribute specifies the source data in stride or memory, but be careful that this attribute affects the global status of the current OpenGL context.

CPU wait during texture upload can be avoided by using pixel cache objects (PBO). In this case, all buffer objects in OpengGL, PBO and GPU, can be mapped to the memory. PBO has many interesting attributes, but we are concerned that after the ing is canceled from the memory, it can upload texture asynchronously, so that the operation order becomes:

GlMapBufferRange → write glyphs to buffer → glUnmapBuffer → glPixelStorei (GL_UNPACK_ROW_LENGTH) → glTexSubImage2D

Currently, calling glTexSubImage2D will return directly, rather than blocking the Renderer. The text Renderer now maps the entire buffer into the memory. Although it does not seem to cause performance problems, it is better to map the scope of the cache texture to be updated instead of all.

Shadow

Text is usually rendered together with the shadow, a resource-consuming operation. Because adjacent glyphs are blurred into each other, the text Renderer cannot separate pre-Blurred glyphs. There are many methods to blur the image, but to minimize the mixing operation and texture sampling in one frame, the shadow is stored in textures and will exist in multiple frames.