Data intermediate platform (PHP, MySQL, redis)

I. Requirement Description

Obtain XML format data from a (external interface, HTTP or socket interface, etc.), store the data to platform B (intermediate platform), provide external interfaces, and return JSON format data, platforms c, d, e ..... the requirement for retrieving data from platform B is actually very simple. This is just an analysis of possible problems and how to solve them. I hope you will give more valuable comments.

It takes about 10 minutes to retrieve data from a for a long time. In addition, external platforms such as C, D, and E frequently connect to B. However, it is difficult to estimate the specific peak value.

 

2. Development Environment

Linux + PHP + Apache + MySQL + redis

 

Iii. Design Ideas

1. The data retrieval module retrieves data from a through the HTTP interface and enters MySQL for persistent storage. At the same time, the data enters redis for caching. The redis structure corresponding to the MySQL table needs to be designed.

Data Retrieval takes a long time. You can use PHP persistent connection or PHP batch processing to avoid service interruption.

For specific ways to see PHP persistent connection processing finishing: http://blog.csdn.net/lxzo123/archive/2011/04/28/6370715.aspx

2. the HTTP interface is released externally, and the switch controls whether to retrieve data from redis or MySQL. It declares that the data is highly time-sensitive and the latest data is returned for each external access.

You need to provide a paging interface. By default, the redis cache is connected externally. You can query the MySQL database only when the corresponding data cannot be obtained in redis.

3. redis data deletion. Because there are a lot of data every day, you need to regularly delete some old redis data, which can be performed according to certain rules, such as deleting the data a few days ago, or if there is more than a certain amount of data in redis, the earliest data will be deleted.

4. Statistical analysis functions, such as the number of user connections.

 

Iv. Specific implementation description

1. MySQL table structure design, which is replaced by simple design because it is a company project.

Create Table news_category (

Id integer, news topic ID

CATEGORY varchar2 news topic name

);

 

Create Table news_info (

Id integer, news ID

Title varchar2 news title

Create date news occurrence time

Topic ID of categoryid

);

 

News content table create table news_content (

Id integer news ID

Content varchar news content

);

Other omitted.

2. redis Structure Design

News topic uses hash to store key: News: Category field: topic id value: topic name

The news ID is stored in zset. Because the page turning process is required, the list does not provide the return index interface, and the page turning process is not perfect (I will explain it later). The zset Structure

Key: News: ID: Index: [Topic id] Score: (next: ID: Index auto-increment into a string) Member: News ID

The news struct stores key: News: info: [Topic id]: [News id] field: field name in the Table value: Field Value

 

Come here first, continue to supplement in the next day