Design and implementation of a lightweight interface access frequency limited service model "go"

Original address: http://www.iam3y.com/html/878.html

Recently, we need to design the interface frequency control related implementation of the Open API and consult the relevant documentation.

Interface frequency control mainly includes two aspects:

(1) The number of times a business ID is accessed within an interface at some time interval (for example, one minute)

(2) Limit of the number of times a business ID is accessed within a period of time (such as one day)

For services that store and perform frequency counting, the following features are required:

(1) Self-updating ability, at a certain point in time to all nodes (node) Self-updating operations, which is often said factory settings

(2) The Protocol light capacity, the agreement must be as simple as possible to achieve efficient purposes

(3) Adding and removing the ability to search and change, allowing business callers to change the data of a node

The nodes mentioned above, such as node, generally contain the following branches

Node.bizid, Business ID, the business ID assigned to a customer, is the customer's unique identifier in the frequency counting system

Node.m_count, number of visits in a minute period

Node.count, a count of the number of times a visit, such as a day.

Node.starttime, counting start cycle

Node.endtime, counting end cycle, recording the last time the customer interface access, also used for frequency counting system self-updating

Node.m_starttime, minutes count start time

Node.m_endtime, minutes count end time, when (node.m_endtime-node.m_starttime) <60 && Node.m_count>m_max_count, interface access is denied, When (Node.m_endtime-node.m_starttime) > 60, while updating Node.m_endtime,node.m_starttime for the current time, reset Node.m_count

Similarly, the principle of the day counting cycle is also true.

Based on the above objectives, we see how the system to achieve, the following content is reproduced content.

the design and implementation of a lightweight access frequency limited service model with original link

In order to prevent the public interface of open system from being excessively called, this paper presents a light-duty model of access frequency limit service for the system, and gives the key technologies such as generic key-value index and frequency constraint algorithm. The Service model abstracts the access frequency restriction logic of various business, records the access interval and time of each key value to a particular business, and provides the access frequency constraint basis for the common interface of the system through the frequency constraint algorithm, which makes the public interface can deny the external request, and eliminates the system security hidden trouble caused by the interface over call. Improve the service quality of the system and the validity of the interface call.

Keywords: access frequency limit; mmap cluster; generic key-Value Index

1. Introduction
For large open platforms, the exposed service interfaces are frequently called by the outside world, and these interfaces may even be excessively invoked without any access frequency control policies. Bring additional burdens to the system.
Excessive invocation, will lead to two aspects of the consequences: the interface callers through the service interface to obtain system security information indirectly, the system is a hidden danger, the common is: a short time to login service interface, through enumeration combination, to crack the user system password. On the other hand, because of the large number of users, a large number of user calls to the interface will affect the quality of service of other users and reduce the effective load of the system.

User access to a certain level of the system platform will encounter the above problems, and each has a corresponding solution. However, a ready-made approach is to include the access restriction function as a standalone module, and the system needs to use this functionality within the business to incorporate the copy of the module into the sub-module of the business. This kind of system organization way is not conducive to maintenance, also is not conducive to management. and the types of access constraints that apply are often fixed and not easily extensible. User access to the storage aspects of data, there are two main ways: client record, or on the server with the database storage. The former's access data may be tampered with, there is a security risk, as a basic function module, the latter is more expensive, it may consume a large number of database connections, reduce business processing power.

To solve the above problems, this paper presents a lightweight access frequency limited service model (Access Frequency Restrict service models, AFRSM). The model itself is a private service interface that is oriented to the inside of the platform. All modules and external service interfaces within the boundary of the system that require access to the frequency limit can be used together by a simple protocol (Frequency Restrict Service, FRS). Ultimately, the purpose of unified management and use of access frequency limit services is achieved.

2.FRSM Overview
2.1 Architecture
Frequency-limited Services (Frequency Restrict Service, FRS), as a dependent service for CGI, Web-service and other public service interfaces, have higher performance requirements. We focus on concurrent processing and lightweight data storage when designing models.
FRS consists of three chunks: the micro-server framework, the Mmap file group, and a set of configuration files. The server we use a high degree of concurrency Epoll model, in addition to the daemon can also configure N sub-process to further increase its concurrency performance. The service handles only two types of requests: Query and update, respectively, corresponding to the FRS's querying interface and updating interface. Public service (recorded as PS) through the FRS query interface can be learned: whether the current user/current IP still has the right to use the service (PS), if the user has access to the service (PS) is not limited, and vice versa, it means that the user in the service (PS) has been constrained.

In the core processing module of FRS, a number of core algorithms are included to access and update the Mmap file base.
Config-cache is one of the important components of FRS, when the daemon starts, the specified profile content opportunity is cached in, and each access to the configuration file will actually hit the desired configuration item in memory. There are two types of configuration files: (1) server-related configuration (srv.xml), which specifies the number of child processes on the server, the host ports that are bound, and (2) the Business Configuration (Map.xml), which is used to register the frequency limit rules for each business: business identity, interval, maximum number of visits within the interval.

2.2 Protocol
The FRS employs the TCP protocol and builds on it on top of the application layer protocol. Since FRS is an internal service and is not publicly disclosed, there is no confidentiality of packets, and FRS is a lightweight service that does not transmit too much data during a service request, so there is no need to consider packet compression. The application layer protocol is built to be simple and easy to scale.
The Http-xml protocol is the application layer protocol we use. That is, the use of HTTP headers to describe the length of the packet body, request command type information. The specific request/response data is described by the XML protocol for extended modification on this basis.

2.3 Deployment and use of
FRS is an internal service whose consumer is the public service interface at the outermost part of the system, CGI, Web-service, and other servers. The common Http-xml protocol makes it easy to implement FRS calls in the logic of other services.
The client usage mode for FRS is relatively fixed, with major points = two steps: A. Query, B. Update.
When and only if key is not constrained in the service (BIZID), the caller key can continue to use the business and, after use, must update the record key's usage of the business
4 FRS usage Patterns
Since the result from step (02) requires a dependent step (01), Step (01) must request FRS using synchronous mode. If the business uses the asynchronous I/O model [3], the possibility of special processing here, to take a partial synchronization request.

3. Analysis of key issues
3.1 Key-value index for generics
Access records in FRS we use lightweight mmap storage, which simply maps the file data into memory. However, traversing a file to find the data you really need is a time-consuming operation, especially if the file is large. To this end, this paper uses the generic key-value index, divides the mmap file into M-Block storage area (called the key node, Key-node), and estimates a key node beforehand before the target data is stored or retrieved, so that mmap can quickly locate the region. Data with different key values may be arranged in the same key node, called estimating conflicts. The conflicting data is arranged on a conflict chain of the key node, and a small area traversal is retrieved.

The upper side describes a single mmap file store. The vertical arrangement is the key node, which is connected to the conflict chain of the node. The conflict chain is limited in length because, in extreme cases of m=1, the conflict chain will fill the entire file, and the retrieval algorithm will degenerate into a file traversal search. The ideal situation is to minimize the length of the conflict chain and hit it as soon as possible. When a conflict chain is exhausted, we need to consider the elimination algorithm to erase/reuse the expired nodes. The size of the mmap file is calculated by assuming that the allowable conflict lengths are length conflict, which contains the key nodes themselves, and each node stores a size node.
Size MMAP = Size Node * Length conflict * m

The generics described above refer to the extensibility of FRS. Programming extensions in FRS can support: use different data types as index primary keys, as long as you implement an index estimation interface for that data type, and compare interfaces. Among them, the index estimation interface is used to compute key nodes under the condition of knowing the total m of the critical nodes. For example, the key value of integer type can get the key node index by the modulo operation, while the string type can use the ready-made Bob hash algorithm [5], while the comparison interface is used to compare the key values in the conflict chain, and finally locate the target data.

The mmap cluster is composed of multiple isomorphic mmap files. It is suitable for larger data volume storage. Assume that there are N storage files in a mmap group. That is equivalent to expanding the data storage capacity by N times. According to the above instructions, the total data storage size can be calculated as
Size MMAP = Size Node * Length conflict * m* N.

In general, using the Mmap group pattern to store and retrieve data, a level 2 search is actually used: first level, index to file, second level, index to key node.
For ease of use, we encapsulate the above access steps for MMAP into a set of operations: Init-mmap, Find-node, Erase-node, Update-node. Because Mmap is a multi-process shared access, these operations, especially those that modify file data, require the use of semaphore mutex processing.
Init-mmap: Initializes the size of the Mmap/mmap group based on the configuration file when the daemon starts.
Create-node: Creates a new node.
Find-node: Locate the target data node.
Erase-node: Deletes the target node. Usually called when the process is retired.
Update-node: Updates the target data node.

3.2 Frequency Limit
The frequency limit is one of the core modules of FRS. To ensure its effectiveness, we try to simplify its implementation logic. It is based on the following simple data structures and algorithms (Figure 6):
where interval and Max are the time interval and number of restrictions (set in the configuration file) that correspond to a business ID, respectively. These two parameters control the number of times an individual accesses a business within a time period.
The algorithm shows that within the interval time period:
The 1th time a business is accessed, a new record node (recorded as node) is created through FRS and node is put. Start and node.end update to current timestamp, node.count as 1 Figure 6 frequency limit algorithm schematic
The nth time (N<=max) accesses the business, then it remains node. Start does not change, node. End is updated to the current timestamp, node. Count is N;
Nth time (n> Max) accesses the business and continues to maintain node. The value of start does not change, node. End is updated to the current timestamp, but because of node. Count has reached the limit value, access is denied.
The Node.start is reset to the current timestamp only if and only if it is satisfied on the next visit: The time interval constraint (node.end-node.start>=interval) condition is lifted. Back to the state of handling the 1th access logic. There are two situations that cause FRS to invoke Create-node to create a new node: (1) first access, (2) because the node is obsolete and needs to be created.

4. Concluding remarks
With a simple protocol and a highly concurrent server model, we have abstracted the access frequency limiting function into an internal service interface in this article and used the MMAP cluster in the implementation to ensure the lightweight of its data access storage. In the actual application operation, the service model can be used as an open platform for 10,000,000-level daily traffic, ensuring effective service quality and system security. The practice shows that the internal function of the platform is abstracted as service, which can improve its application degree, obviously reduce the maintenance and expansion cost of the function, and has certain popularization significance.

Design and implementation of a lightweight interface access frequency limited service model "go"