Projects can also learn a lot of things

Share today some insights on a little project you have done recently.

1. Small projects can also learn a lot of knowledge

Recently done is a data collation of a project, back-end system needs some statistical data, if the query from the original system, will greatly affect the performance of the other side, and relative to the mobile phone to direct users, the back-end system is facing the company personnel and operators, the number of users, but there are many data query, A lot of computation is required, and this amount of computation consumes the performance of the server. In order to ensure the performance of the online phone and front end, the database has been optimized to build the index is based on the user to establish. Background system search conditions are particularly large, without the establishment of the index of the case, in a large number of data query data, also includes a large number of calculations, which results in the impact of performance, then the need for space to change time, and the background system requirements of real-time is not high, then you can take care of the data out, put into the new database Then in the new database to aggregate, to get back-end system needs data, here you can see the concept of Data warehouse. So I did this project is to extract data from the original database to the new database, the task is simple.

However, this small project let me learn a lot of knowledge:

1. Database format problems, the developer created a database of the fields are not unified, because the company used two databases, MySQL and MongoDB, from the two databases to extract data, because I use the MySQL when the data storage, in the creation of table fields, there are particularly much trouble, Field length is not the same, many data are not mandatory specification, MySQL and MongoDB data is not the same, but this is not uniform and normal, but the program has a specification, if done in accordance with the program, but the database has a large number of non-standard data fields.

2. Do a good job of data-tolerant processing, a lot of data, incremental extraction, that is, extracting data from the previous day, but suddenly found that the first two days of data a little problem, then how to do, you need to start another listener thread in the program, the listener thread is every 10 minutes read a field in the database, the field is true, True to remove these days of data and re-extract them.

3. Because of the order of running data, such as the need to first extract the user's information, and then to the user information statistics, if the user's information error, then the statistics will be a problem. Here I have designed a small framework for task handling. In the order in which the tasks need to be configured, the previous task is not completed, the successor task cannot be executed, the task in one queue can be executed in parallel, and the tasks of different priorities are executed serially. In the implementation of this logic, learned the Java queue, tree structure, learning multi-threaded knowledge, in order to design beautiful, learning a singleton mode, observer mode (Eventbus), in order to be able to greatly simplify the code, using annotations, Learning Guava package, Common-lang package class, And after you add an exception to the project, send the mail feature.

This is a very small project, in which I have learned a lot of knowledge. A lot of times, in complaining every day in the work of adding or removing changes, feel every day do is physical activity, completely do not mind the work, actually, want to work well, need to spend a lot of experience. Write interface to do is add and delete work, but the ability to operate the space is great, use springboot do project, whether in the project added interception work, on the request to do some basic work such as log printing, to the exception whether there is a unified processing work, follow-up do unit testing, For the interface to do the pressure test, to a function or a project to do a good job, it takes a lot of effort, so small projects, if we are willing to do, find there will be a lot of things to do, and in this small project can learn a lot of things.

2. Dare to practice and turn ideas into code

It is very difficult to turn your thoughts into code, but to be able to completely turn ideas into code.

Many times, we see others to share, we also remember how it operates, such as Redis as a cache, encounter cache avalanche, cache penetration and other problems can say a Sanlai, but seldom go to the hands of practice. It is really rare to encounter such scenes, such as me, developed for two years, the company's data are very few, concurrency is not high, a small redis cluster, fully support the online access, and will not encounter the cache penetration caused by slow query speed situation. If there is no real, hands-on solution to the problem, and can not be very good to understand what the problem is a situation, even if others share the very thorough, that is he really to solve, formed their own experience. And we looked, also live only has such a problem. To see the problem, you need to reproduce the problem, and in accordance with the way others share the solution, so that they can really learn to accept.

Business does not encounter such a problem, then what to do, then to simulate, their own things to write programs, and then use the tool to simulate a large number of requests, to achieve high concurrency, to observe high concurrency in the case of Redis and the database, and then in accordance with the situation to solve the problem, and finally to verify the problem. When you have done these things, you can really say that they have done in the case of high concurrency, to solve the problem of Reidis cache.

Again, such as sub-database sub-table, there are a lot of articles on the Internet to introduce its ideas. But the online data in the case does not need to sub-database sub-table, then just read the article, and did not go deep to understand, a surface image, if it is to the next company, suddenly asked the sub-database of the problem, although it can be simple to say the idea, but really talk about the operation, it is likely to be wrong. And when it is already the online program, need to do the sub-database table work, need to do, how to deal with, the online program stopped to maintain, the data moved to the new database? Whether there is a better way, whether you can do double-write operation, that is, the insertion of data is not only written to the old database, but also to the Library sub-table, and then the data Update delete operations sent to the message middleware, when the data in the old database are moved to the database, and then the program's database switch component database, Consumer message middleware, and finally write a program to compare the data. This series of operating procedures, only in the actual operation can be tested, but only in practice in order to the knowledge of others into their own.

Summary: Programming is a need to spend effort to do things, do not complain about the daily work is only to delete and change the job, small projects can become a very rich project, in small projects can also learn a lot of knowledge.

Projects can also learn a lot of things