21.app backend How to efficiently update content

In the app's home page or notification bar, it's often necessary to get the latest data through the API. So how do you optimize in this section to make it more efficient to get content? In this paper, a strategy of efficient data acquisition is realized by push-pull and incremental updating.

1. Efficient update of data policies in app scenarios

, in the app home page, there will often be this waterfall stream form of content, specific reference to Sina Weibo app.

This content has the following characteristics:

1. The frequency of user visits is good (the homepage is usually accessed frequently)

2. It looks like a huge amount of data.

So, how to get the first page data efficiently?

The key to efficiency is two points:

1. Reduce the frequency of visits

2. Reduce the amount of data transferred

Content push-pull and incremental updates are key to achieving efficiency.

2. What is content push-pull?

In general app design, if you need to know if there is content update on the homepage, you need to access the API through a polling mechanism to know if there is any content update.

However, the disadvantage of polling is also obvious:

1. Power consumption

2. Consumption flow

Polling is a typical pull pattern that pulls data from the server to the app every few short periods of time, regardless of whether there is actually data updates. This consumes a lot of network traffic and also increases the pressure on the server.

How to reduce the number of polling times.

The answer is through push mode. On the server, whenever you know there is data update, through the push system, tell the relevant app, you have data update, hurry to fetch. When the app receives the notification of this data update, it then calls the API to get the appropriate data.

Of course, can not only use push mode, because of the complexity of the network environment of the mobile phone, can not guarantee that the data update notification will reach the app, so also to use the polling method to periodically pull the data. Of course, the use of push-pull combined with polling can be set a long time, because mainly in case of use.

This combination of push and pull mode can greatly reduce the frequency of access and the amount of updated data.

3. What is an incremental update?

In Sina Weibo app, from other pages into the homepage, in the absence of the network, the home page has received the micro-blog is still able to display, which is obviously the relevant data stored in the app Local.

App local storage that uses data can reduce network traffic while greatly improving the user experience (think about how much data is available locally in the app, and the speed of course is fast). After you have used local storage, you need to consider the incremental update scenario for the data.

What is an incremental update of data? Suppose that the first page of user A has 40 data in the data table, and Id1-40,app gets 10 data each time. First run, the app gets the ID1-10 data from the data table and stores it locally. Assuming the user leaves this page and then goes back to the homepage, the app needs to retrieve the data from the database again, since 10 data (ID1-10) have been stored locally in the app. Then the 10 data that needs to be fetched from the database now is obtained from the remaining 30 data (id11-40) and stored locally in the app. This is a typical example of an incremental update.

The principle of incremental update is that in the database, each data must have update_time this value, record the time of the last update of the data, when the app obtains data from the server once (the returned data must be sorted by time, update_time the most recent in the first), Record the update_time of the first data, and when you get the data again, you only need to get the data that was updated at the last point in time to access the server.

Because of the existence of paging mechanism, this algorithm is quite a lot of places to pay attention to, below I give a simplified example detailed description:

Some assumptions:

1. App with 4 parameters per request

http://test/api/timeline?count=3&page=1&since=1100&max=1200

Count: Number of bars per page (default = 3)

Page: Current page number (default = 1)

Since: timestamp, if this parameter is specified, returns the result of a timestamp greater than or equal to since (should be the update_time of the last fetched data)

Max: timestamp, if specified, returns a time stamp that is less than or equal to Max (should be sent at time)

When querying on SQL, use condition since<=update_time<= max

2. The data returned by the API contains

{

"Size": 10,//The amount of data actually returned (because of paging gets, so often less than total value)

"Total": 284,//amount of data that should be returned

"Page": 1,

"Count": 3,

"Max": 0,//max for Update_time of the last data obtained

"Since": 0

},

{//Returned data entity

Data: .....

}

3. The update_time in the local data stored in the app refers to the update time of this data in the server, not the update time of this data in the app.

Start the discussion now:

(1) When the app is installed, the data in the server data sheet is 3, and the local data stored by the app is empty.

Data for the server's data table

Id	Update_time
1	1100
2	1101
3	1101

Local data stored by the app

Id	Update_time

(2) When the app first runs (time is 11:05), because it is the first run, since is 0,max for the current point in time 1105, in the server's data table to get all the data.

The request sent is: http://test/api/timeline?count=3&page=1&since=0&max=1105

(3) After sending a request from (2), the API returns data, data from the server's data table, and the app stores the local data as follows:

The data returned by the API

{

"Size": 3,//amount of data actually returned

"Total": 3,//amount of data that should be returned

"Page": 1,

"Count": 3,

"Max": 1101,

"Since": 0

},

{//Returned data entity

Data: .....

}

Data for the server's data table

Id	Update_time
1	1100
2	1101
3	1101

Local data stored by the app

Id	Update_time
1	1100
2	1101
3	1101

here is the focus of the Strategy (1): Max in the API return data must be the update_time of the last piece of data

(4) The current time is 11:20, the user clicked the "Get More" button on the page, the app should pull the data from the server's data table, before sending the request, the data table of the server is as follows:

Data for the server's data table

Id	Update_time
1	1100
2	1101
3	1101
4	1118
5	1118
6	1119
7	1119

As you can see, the server's data table has more data with ID 4,5,6,7 than the last time the data was pulled.

At this point the API request is sent, and the policy is focused (2): When the API returns data Size=total, the since value is larger than the last one, because the data is now complete and there is no need to repeat the data obtained last time (remember the condition since<= update_time<= Max? So the since value is set to 1101+1=1102,max for the current point in time: 1120, the requested URL is as follows:

http://test/api/timeline?count=3&page=1&since=1102&max=1120

The return data of the API after the request is sent and the local data stored by the app are as follows:

The data returned by the API

{

"Size": 3,//The amount of data actually returned (because of paging gets, so often less than total value)

"Total": 4,//amount of data that should be returned

"Page": 1,

"Count": 3,

"Max": 1119,

"Since": 1102

},

{//Returned data entity

Data: .....

}

App's data:

Id	Update_time
1	1100
2	1101
3	1101
4	1118
5	1118
6	1119

Here is the focus of the Strategy (3): in the database, there are 4 data update_time for 1101~1120, but because of paging, only 3 (from the size and total parameters can be determined), which means that 1101~ 1120 This period of time the data is not complete, the app gets the last piece of data update_time is 1119, the server's data table is not available to the app data in two cases:

A.update_time is exactly 1119.

B.update_time Greater than 1119

Because we can not judge what kind of situation, if we pull the data next time since greater than 1119, the Server data table ID 7 data will not be retrieved, then the app will be lost in the ID 7 data, so for the last data acquisition is incomplete, The next time the data is fetched, the since must be equal to 1119, although it is possible to obtain duplicate data.

(5) The current time is 11:30, the user clicked the "Get More" button on the page, the app should pull the data from the server's data table, before sending the request, the data table of the server is as follows:

Data for the server's data table

Id	Update_time
1	1100
2	1101
3	1101
4	1118
5	1118
6	1119
7	1119
8	1120

At this point the API request is sent, and here is the focus of the Strategy (4): When the API's return data size is less than total, in order to avoid data loss, since is the max value for the return data of the last received API: 1119,max is the current point in time: 1130. For strategic focus (4), please combine the focus of the Strategy (3) to understand it together.

The requested URL is as follows:

http://test/api/timeline?count=3&page=1&since=1119&max=1130

The return data of the API after the request is sent and the local data stored by the app are as follows:

The data returned by the API

{

"Size": 3,//The amount of data actually returned (because of paging gets, so often less than total value)

"Total": 3,//amount of data that should be returned

"Page": 1,

"Count": 3,

"Max": 1120,

"Since": 1119

},

{//Returned data entity

Data: .....

}

This is the focus of the Strategy (5): The API returns data in the ID 6 of the data, in the app's local data already exist, for this data, the app should discard the duplicate insert.

The last app stores local data as follows:

App's data:

Id	Update_time
1	1100
2	1101
3	1101
4	1118
5	1118
6	1119
7	1119
8	1120

OK, the policy for the entire incremental update has been analyzed. In this strategy, the page parameter is almost useless and is reserved for compatibility with paging without since,max. For the policy of this incremental update, carefully understand the analysis of the Policy focus (1) (2) (3) (4) (5).

Incremental update policy, and also to handle a synchronization issue that deletes data. Suppose that in the server's data table to delete a piece of data, how to notify the app local also delete this data. Our solution is to add an identity is_delete to the server's data table, set the is_delete of this data to 1 and update update_time when it is necessary to delete the business logic. When the app incremental update detects this Is_delete 1 data, it deletes the data in the app's local data. To avoid storing too much data on the server, set a crontab on the server and periodically delete the data that has been identified as Is_delete 1.

--------------------------------------------------------------------------------------------------------------- ------------

Open the total list of articles in the Link app backend series to see all the original "app backend" articles I've published.

"Author" Zeng Jiansen
"QQ" 190678908
"app back-end QQ Group" 254659220
"public number" Appbackend
"Sina Weibo" @newjueqi
"blog" Http://blog.csdn.net/newjueqi

21.app backend How to efficiently update content