Practices for ensuring user access speed in Web data mining (SQLite + quartz)

I have been entangled in this problem for a long time. I have thought a lot of ideas before:

1) record in webdb;

2) Adopt asynchronous thread record;

3) Use JS like cnzz or Google;

4) That's all you remember.

However, all of them were killed in my mind. The first method was that webdb was already overloaded, and every request was recorded too impractical, and the collected data structure would change from time to time, it is usually determined based on the current research direction. The second method is better, but how to keep the thread alive in the current environment is also a problem. The third method is that many JS Ajax files cannot be recorded.

At this time, SQLite is a good choice. Running on the web side will not put pressure on the database server, and in the Server Load balancer, I only need one to enable collection.

It is also convenient to change the data structure. After the change, you can take the original data and back up the data. The benefits of file-based processing are obvious.

In terms of efficiency, writing with high loads is still very difficult, but quartz solves this problem. With the help of quartz, I don't need to worry about when it will be executed, just like asynchronous, this does not affect the current request. You can also manage write tasks and add listeners.

At present, it is still very good, and the effect remains to be observed.