The company wants to record audit logs for some systems. These logs are not commonly used by our developers.ProgramFor example, logs recorded using log4j), but for the purpose of future use by the audit department, the log feature has strong business requirements. The architecture has been designed by other colleagues in the company. Although I am only doing some supplementary work, I am very interested in this proposition. Today I carefully considered such a business requirement, I think it is very interesting. There will be some technical content to do this. The following are my ideas. These ideas are not all independently developed by myself, and I have also borrowed some colleagues' designs.
First, I want to set up a business scenario: we create an audit system for an important commercial payment website. The most important technical requirement is that the audit system cannot affect the performance of the business system and the stability of the business system.
The following is the design I think:
-
- If you want the audit system to not affect the performance of the business system, I personally think that the audit system is separated from the business system, if the resources are sufficient, it is best to have a dedicated server to independently deploy the audit system. If there are no new servers as resources, we can open up limited resources on the servers with business systems to provide the audit system, however, the ratio of the audit system to the computer resources occupied by the server must be well controlled.
- Audit logs often contain a lot of business requirements, so users record the AuditCodeIt will be coupled with the code for business operations, especially if the audit needs to be abnormal, every function point in our business system may record the code of audit logs (resulting in changes to the Code of each business method, which is painful ), this coupling will inevitably bring risks to the system. How can we design the best interaction between the audit system and the Business System? Because both systems are independent in my design, it indicates that the two systems are a set of heterogeneous systems, and the information interaction between heterogeneous systems must be through communication, there are many communication methods: WebService, HTTP, message queue, or enterprise internal message system. (I am not familiar with message queues. I am familiar with the first two types of message queues. The latter does not mean that message queues have the same defects) in my understanding, no matter which communication method there will be a request and response process, and this request and response process is not always secure, if the communication is blocked for some reason, and the request and corresponding process cannot be completed smoothly, the system will produce errors, posing a risk for system stability, therefore, it is recommended that the code embedded in the Business System Code record audit logs not contain requests and corresponding functions, so that the corresponding functions of the requests are coupled to the business functions. I personally think it is best to separate the log recording function into one thread. This thread cannot interfere with the user request thread, but only needs to pass the value to the interface of the business method code, if the value is successfully transferred, you do not need to worry about the Business Code. Because I have to embed a thread pool in the business system. The threads in this thread pool are used to record the log function. This thread pool will also run along with the Web Service Startup. The business system records the log information and adds it to the thread as a thread record, as long as the business system records the log, the log and business operations in the business system are separated, so that everyone does not interfere with each other, in this way, the log recording function reduces the interference to the business system, thus reducing the impact of the log recording system on the security of the business system. The thread in this thread pool is responsible for transmitting data to the audit system. As long as the data is transmitted to the audit system, the audit system returns a success ID. The thread does not care whether the operations in the audit system are successful.
-
- The design in the audit system is also worth consideration. I personally think so. We put the data received from the business system into a queue ). To improve the efficiency of log requests, I still audit a thread pool for the audit system. The data used by this thread pool is the queue we have built. What are the advantages of this solution:
- Concurrent operations will definitely improve the system running efficiency. I don't think anyone has any objection;
- If we deploy the audit system and business system on a server, we can limit the system resources consumed by the Audit Log System by controlling the queue size and thread pool size.
- Some people may wonder why I use queue storage instead of directly uploading data to the thread. I have two purposes: first, advanced data queue. This operation is very simple and not prone to errors, after entering the queue, immediately send a response to the business system, which will reduce the system's unstable factors. First, we first let the data into the queue rather than the thread pool, therefore, we can be prepared to reduce the thread pool size, especially when the audit system is deployed on the server of the business system.
The above is my preliminary idea. The difficult technologies involved in my design include: the communication mode of heterogeneous systems, the thread pool technology, and an efficient and secure queue design.
I personally feel a difficulty: I want to independently design a thread pool in the business system to record log information. The threads in this thread pool must be independent of the user request thread, what should we do in the web system developed using Java? I don't know which kids shoes can give me some advice.
Some children's shoes may have the following question: I opened a new thread in the business system to record logs, which does not affect the performance of the business system. This problem is actually better solved, because the opening of new threads to record logs only consumes server resources rather than the system resources that each user request in the business system loads, the addition of the audit log function should affect the number of concurrent users that can be borne by the servers deployed in the business system, rather than the burden on each user system. As for how to improve the concurrency, the solution is relatively simple, improve the performance of the Business Server (such as adding CPU and memory) or add several servers.
I am most interested in the thread pool technology involved here. I recently reviewed some thread pool technologies and found that the thread pool is not as simple as I thought, the threadpool technology that comes with JDK is worth studying.
Finally, I hope that the Daniel in the blog will see if this article can give me some suggestions to see if my design is good or not.
