Flipboard Web Mobile-Create a fluid experience of 60 frames per second

At the dawn of smartphones and tablets, Flipboard launched a "mobile first" experience that allows us to rethink the principles of content layout in the page, as well as the factors related to touch screen and how to get a better user experience.

In order to build a complete experience, we bring Flipboard to the web side. What we do in Flipboard has an independent value on every user's device: organize the best stories from the themes, sources and people you care about the most. So bringing our services to the Web is also a logical extension.

When we started this project, we realized that we needed to move our thinking from the mobile experience to the web side, trying to improve the content layout and interaction of the web side. We want to achieve native application-like sophistication and performance, and still be aware of the real browser.

Earlier, after testing a large number of prototypes, we decided to use a scrolling approach as a web-side experience. Our mobile apps are well-known for a book-like experience, which is intuitive on the touchscreen. But a series of reasons indicate that scrolling on the web side of the experience is more natural.

To optimize scrolling performance, we know we need to ensure that page rendering is less than 16ms in frequency, while limiting reflow (reflow) and Redraw (repaints). This is especially important in animations. To avoid re-rendering in animations, there are two properties that you can safely animate: CSS transform and opacity. But that's too small a choice.

What do you do when you want to animate the width of an element?

How does a frame's scrolling animation work?

Notice that in the top of the icon from white to black. 　　 Here, two separate elements are used to cover each other and cut each other according to the following content. These types of animations have been criticized online, especially on mobile devices, for one simple reason:DOM is too slow .

Start using <canvas>

Most modern mobile devices have hardware-accelerated canvases, why don't we use them? The HTML5 game has been done. Can we really develop the application interface on canvas?

Immediate mode and retention mode

Canvas is an immediate-mode drawing API, which means that the information of the drawing object is not preserved when it is drawn. The opposite is the retention pattern, which is a declarative API that maintains the hierarchy of the objects being drawn.

The advantage of the retention mode API is that, for your application, they are often more likely to build complex scenes, such as the DOM. This usually leads to a performance cost that requires additional memory to save the scene and update the scene, which can be slow.

Canvas benefits from Immediate mode, allowing direct sending of drawing commands to the GPU. But if you use it to build a user interface, you need a higher level of abstraction. For example, some simple processing, such as drawing a text on a picture, can occur when you draw an asynchronously loaded resource onto an element. In HTML, it is easy to implement because of the order in which elements exist, as well as the presence of CSS z-index .

In <canvas>element to build the UI

Compared to Html+css,canvas, there are some congenital deficiencies, the lack of a lot of HTML + CSS for granted features.

Text

Canvas has a very simple API for drawing text: fillText(text, x, y [, maxWidth]) This function accepts three parameters: the text itself, and the coordinates that draw the starting point x y . But canvas can only draw one line at a time. If you need to wrap the text, you need to write your own function.

Image

You can use drawImage() functions to draw pictures on the canvas. This is a variadic function, and you can specify more parameters to control positioning and trimming. However, canvas does not care whether the image is loaded or not, and the function is called only after an image load event.

Element Cascade

It should be easy to specify whether an element should be on another in HTML and CSS, either in the order of DOM elements or using the Z-index property of CSS. Keep in mind, however, that canvas is the drawing API for immediate mode. When an element overlaps, or if one of them needs to be redrawn, it must be redrawn (or at least partially redrawn) in the same order.

The translator notes on the local redrawing to improve the performance of the article you can refer to: "Improving the performance of HTML5 canvas several methods (turn)"

Custom Fonts

Do you need to use a custom Web font? The canvas's text API does not care if the font is loaded. You need a way to know if a font is loaded and draw any area that relies on this font. Fortunately, modern browsers have a promise-based API. Unfortunately, iOS WebKit (iOS 8 o'clock) does not support it.

<canvas>The advantages

Given all these shortcomings, people began to question the choice of canvas instead of DOM. In the end, our discussion is decided by a very simple truth: You cannot create a scrolling list view of FPS based on the DOM .

Many people (including us) have tried, but have failed. Scrollable elements can be implemented in plain HTML and CSS by overflow:scroll : (combined with IOS -webkit-overflow-scrolling:touch ), but these do not give you frame-by-box control in scrolling animations, while mobile browsers are difficult to handle long and complex content.

To build an infinite scrolling list of fairly complex content, we need to implement a uitableview on the web side

Most devices today have a hardware-accelerated canvas implementation compared to the DOM, and can send drawing commands directly to the GPU. This means that we can render the elements very quickly, and in many cases we are talking about the millisecond scale.

Canvas is also a very "slim" API compared to HTML + CSS, which reduces bugs on the interface or inconsistencies between browsers. There is a reason to be more direct, canvas does not have Can I use?.

Translator Note UITableView is an IOS control

Faster DOM Abstraction

As mentioned earlier, for a bit of effect, we need a higher level of abstraction rather than simply drawing rectangles, text, and images. We've built a very small abstraction that allows developers to work on a node tree rather than dealing with a strict sequence of drawing commands.

Render Layer

The render layer (Renderlayer) is the base node on which the other nodes are built. Common attributes such as,,, top left width height , backgroundColor and zIndex present in this layer. Renderlayer is just an ordinary JavaScript object that contains these attributes and an array of child elements.

Image

We use the attached properties of the image layer to specify the image URL and information. You don't have to worry about the image Load event listening, the image layer will process and send a signal to the drawing engine to indicate that the picture needs to be updated.

Text

The text layer can display multiple lines of text truncation, which is very expensive to handle in the DOM. The text layer also supports custom fonts and also handles actions that are updated when the font is loaded.

Synthesis

These layers can be synthesized to build a complex interface. Here is a render layer tree

{
Frame: [0, 0, 320, 480],
BackgroundColor: ' #fff ',
Children: [
{
Type: ' Image ',
Frame: [0, 0, 320, 200],
IMAGEURL: ' http://lorempixel.com/360/420/cats/1/'
},
{
Type: ' Text ',
FRAME: [10, 210, 300, 260],
Text: ' Lorem ipsum ... ',
Fontsize:18,
Lineheight:24
}
]
}

Invalid layer

When a layer needs to be redrawn, such as when an image is loaded, it sends a signal to the drawing engine to indicate that its frame is dirty (dirty). These modifications are processed in batches using Requestanimationframe, avoiding layout jitter, and then redrawing the next frame canvas.

Up to 60fps Scrolling

For the web side, perhaps the most natural concern for us is how the browser scrolls the page. Browser vendors have done their best to improve scrolling performance.

This is actually a compromise. In order to achieve the scrolling index of FPS, the mobile browser stops the execution of JavaScript during scrolling, which is afraid of DOM modification causing reflux. Recently, IOS and Android have exposed the Onscroll event, they work more like a desktop browser, but if you try to keep DOM elements in sync while scrolling, there may be a difference in the implementation.

Fortunately, browser vendors are already aware of the problem. In particular, the Chrome team has opened up the work to improve the situation on the phone.

Back to canvas, the short answer is that you have to scroll with JavaScript.

The first thing you need is an algorithm for calculating the degree of scrolling. If you don't want to study math, Zynga's pure rolling implementation of open source is suitable for any situation similar to this layout.

We use a canvas element to complete the scrolling. In each touch event, the rendering tree is updated according to the current scrolling level. The entire render tree is then re-rendered using the new coordinates.

This sounds incredibly slow and can be used on canvas with an important optimization technique-the result of drawing operations on the canvas can be cached at the off-screen layer (off-screen) canvas. The Off-screen layer (off-screen) can be used to redraw the layer.

This technique can be used not only for image layers, but also for text and graphics. The two most expensive drawing operations are filled with text and images. But once these layers are drawn once, it's very fast to redraw them using the off-screen layer.

In the above demo, the content of each page is divided into two layers: the image layer and the text layer. A text layer consists of multiple elements grouped together. In each frame scrolling animation, both layers are redrawn using the bitmap cache.

Object Pool

During the scrolling of an infinite list, a large number of renderlayers will be created and destroyed. This creates a lot of garbage in memory and stops the main thread when it is garbage collected.

To avoid generating large amounts of garbage, renderlayers and related objects are pooled into a pool. This means that only a relatively small number of layer objects are created. When it is no longer needed, it is released back into the pool and can then be reused.

Fast Snapshot

Caching the properties of a composite layer can provide another benefit: the ability to use part of the rendering structure as a bitmap. Do you have a need to create a partial DOM structure snapshot? When you render these structures on canvas, it is incredibly fast. This puts a project into a magazine interface that uses this feature to perform a smooth transition of the timeline dimension. The snapshot contains the entire project minus the top and bottom.

A declarative API

Now we have the bricks to build an application. However, it may be tedious to build renderlayers by command. Wouldn't it be nice if we had a declarative API similar to how DOM works?

React

We are a loyal fan of the React framework. Its single-directed data flow and declarative APIs have changed the way people build applications. The most striking feature of react is the virtual DOM. Rendering as an HTML container is just a simple implementation of it in the browser. The recent introduction of the React Native proves this.

What if we combined our canvas layout engine with the react component?

React Canvas Introduction

React Canvas gives the React component the ability to render to Canvas.

The first version of the canvas layout engine looks like a command-style code. If you've done JavaScript DOM operations, you might have run code like this:

Create the parent layer
var root = renderlayer.getpooled ();
Root.frame = [0, 0, 320, 480];

Add an image
var image = renderlayer.getpooled (' image ');
Image.frame = [0, 0, 320, 200];
Image.imageurl = ' http://lorempixel.com/360/420/cats/1/';
Root.addchild (image);

Add some text
var label = renderlayer.getpooled (' text ');
Label.frame = [10, 210, 300, 260];
Label.text = ' Lorem ipsum ... ';
Label.fontsize = 18;
Label.lineheight = 24;
Root.addchild (label);

Of course, this will do the work, but who wants to write code like this? It's hard to imagine the result of rendering, except for error-prone

Using React canvas becomes the following:

var mycomponent = React.createclass ({
Render:function () {
Return (
<group style={styles.group}>
<image style={styles.image} src= ' http://... '/>
<text style={styles.text}>
Lorem ipsum ...
</Text>
</Group>
);
}
});

var styles = {
Group: {
left:0,
top:0,
width:320,
height:480
},

Image: {
left:0,
top:0,
width:320,
height:200
},

Text: {
Left:10,
top:210,
WIDTH:300,
height:260,
Fontsize:18,
Lineheight:24
}
};

You may notice that everything seems to be an absolute positioning implementation. That's true. The birth of our canvas rendering engine, accompanied with the goal of driving pixel-level layout, to realize the long ellipsis of multiple lines of text. Traditional CSS does not do this, so the absolute positioning of the way for us is very suitable. However, this approach is not suitable for all applications.

CSS Layout

Facebook has recently open source CSS for JavaScript implementations. He supports some subsets of CSS, including margin , padding , position and most importantly flexbox .

Consolidating CSS layouts into React Canvas is only a matter of time. Take a look at this example and see how we changed the component style.

Infinite scrolling of declarative expressions

How do you create a scrolling list of up to FPS, unlimited, paging in React Canvas? It turns out to be very easy to implement because react will do the virtual DOM diff. render()only the currently visible elements in the function are returned, and react is responsible for updating the virtual DOM tree required during scrolling.

var ListView = React.createclass ({
Getinitialstate:function () {
return {
scrolltop:0
};
},

Render:function () {
var items = this.getvisibleitemindexes (). Map (This.renderitem);
Return (
<group
Ontouchstart={this.handletouchstart}
Ontouchmove={this.handletouchmove}
Ontouchend={this.handletouchend}
Ontouchcancel={this.handletouchend}>
{Items}
</Group>
);
},

Renderitem:function (ItemIndex) {
Wrap each item in a <Group> which are translated Up/down based on
The current scroll offset.
var Translatey = (ItemIndex * itemheight)-this.state.scrollTop;
var style = {Translatey:translatey};
Return (
<group Style={style} key={itemindex}>
<item/>
</Group>
);
},

Getvisibleitemindexes:function () {
Compute the visible item indexes based on ' This.state.scrollTop '.
}
});

In order to tick (capture) the scroll, we call the scroller (scrolling component) method in the List view component setState() .

 ... 
 
// create the scroller instance on mount. 
 componentdidmount: function  ()  {
   this.scroller = new scroller ( This.handlescroll); 
}, 
 
// this is called by the scroller at each scroll  event. 
 handlescroll: function  (left, top)  {
   this.setstate ({ scrollTop:  top }); 
}, 
 
 handletouchstart: function  (e)  {
   this.scroller.dotouchstart ( E.touches, e.timestamp); 
}, 
 
 handletouchmove: function  (e)  {
   e.preventdefault (); 
   this.scroller.dotouchmove (E.touches, e.timestamp, e.scale); 
}, 
 
 handletouchend: function  (e)  {
   this.scroller.dotouchend ( E.timestamp); 
} 
 
 ... 

Although this is a simplified version, it shows some of the best features of React. Touch events are declared bound in the render () function. Each Touchmove event is forwarded to Scroller to calculate the current scrolling offset. A scrolling event emitted by each Scroller is used to update the Status List view component to render only the elements that are visible on the current screen. All this happens below 16ms because the React diff algorithm is very fast.

You can view the full implementation of this scrolling list of source code.

Practical application

React Canvas does not completely replace the DOM. We use the most critical areas of performance requirements in our mobile web app, mainly scrolling the Timeline view section.

Dom can be a better approach when rendering performance is not a problem. In fact, this is the only way to enter fields and audio/video for some elements

In a sense, Flipboard's mobile web is a hybrid (hybird) application. Compared with traditional native application and network technology, the content of Flipboard is all Web content. Its UI is based on a DOM implementation and uses canvas rendering where appropriate.

Accessibility

This area needs to be explored further. Using degraded content (the DOM subtree of the canvas) should allow screen readers such as VoiceOver to interact with the content. We saw different results on the device we were testing. In addition, there are standards for focus management, but currently it is not supported by browsers.

One approach proposed by Bespin in 2009 is that when elements are rendered to canvas, a parallel DOM is maintained for element synchronization. We are continuing to look at the right way to achieve accessibility.

Conclusion

In the pursuit of FPS, we sometimes take drastic measures. Flipboard provides a case study of the limitations of mobile networks. While this approach may not be suitable for all applications, we have increased the application's interaction and performance levels to compete with local applications. We want to allow other compelling examples to emerge by opening up the work we do in React Canvas.

Visit flipboard.com with your mobile phone and experience it. Or if you don't have a Flipboard account, experience a series of magazines on Flipboard. Please let us get your ideas.

Special thanks to Charles, Eugene and Anh for their editorial and advice.

Http://jcodecraeer.com/a/qianduankaifa/css3/2015/0305/2549.html

Flipboard Web Mobile-Create a fluid experience of 60 frames per second