The "parenting" experience of the dairy dad programmer

The "parenting" experience of the dairy dad programmer

I used to be a. net programmer. I was responsible for transferring the project from. net to java in the second half of last year, and there were cross-data center migration, hundreds of millions of visits, and app server projects.

I have been working for eight years without becoming an architect, management, technical master, or sharing master. I have been working on the business, and I am going further and further on this road. Sometimes I feel embarrassed, but sometimes I feel quite suitable for myself.
Before the Chinese New Year, my wife gave birth to a small public exercise. The baby is hungry. "My wife is going to feed the baby !", The baby is hungry again. "My wife is going to feed the baby !", The baby woke up and went hungry. "My wife is going to feed me !"...... My wife said, "I feel like I am a cow "! As a "Daddy", I feel deeply!
The project you are responsible for is like a child. The first thing you think of is this dad. Dad gets involved (often gets up in the middle of the night), the diaper changes (the server is faulty), the feeding will be fed (bug ). If you are ill, feed the medicine. If you don't take the medicine, ask the doctor outside ). If a problem occurs with the baby's milk powder, you can't wait to take the knife and kill the profiteer. (when someone else's service is called, the service goes down and affects yourself ). The baby is full of food, sleeping quietly, Dad can also rest assured!

-------------------------------------------------------------------------------
Next, let's take a look at our "parenting" experience:
I. Technology Selection
Development language:Java, go, php, nodejs
As I said at the beginning, I used to be a C # programmer. C # has a subtle syntax. The development and. net versions of the last day are updated quickly, so I don't even remember the latest version. So many new features, if the server is always installed with. net3.5, what is the use for us.
In the beginning, it was highly resistant to java. I learned it many times and didn't put it into use. This time, I have to go on. I have to say that over the years, java has been growing, stable, mature, and can solve almost all problems, and its performance is not bad. Over the past six months, java's water has been really deep, asynchronous, and parallel, and many others have not been touched yet.
The benefits of go will not be mentioned. If you are studying, you can stick to it if you like it and feel that you can hold it. (Ha, it will certainly not be as easy as you said ).
Nodejs is too convenient for front-end, and some people use it for the server interface layer.
Php is used as a front-end page, just like using java on the server side. It is mature, stable, uses a lot of people, and has a lot of information.
In a word, there is no best, only the most suitable!
Java is selected because many of our backend microservices are developed in java for convenient calling. There is also something that can be used without java.
Storage:Mysql, mongodb, and redis
When database sharding fails to solve the problem, table sharding is especially important and has a feeling of unlimited scalability. Oracle used previously, only database sharding, no table sharding, can't hold it anymore.
The storage type is as few as possible, the better. redis cache is generally not used.
Many people in the team reject mongodb. Let alone the specific reason. If redis can solve the problem, it should not be used.
There is no better sentence, but it is only suitable. Put the corresponding data in the most suitable storage.
Mq:Rabbitmq, activemq
You will certainly use mq, even if not, it will certainly be in the future.
Java framework:Spring mvc
Many users are used, mature, and pitfall have been trampled on by everyone. If you encounter any problems, please check the information and solve them.
Ibatis and struts are not used in our project.

Ii. source code management tools
The language is selected, the framework is selected, and the code is to be written. The question is, what is the management of the written code?
Git!SVN!
Git branches can solve too many problems and are very powerful!
SVN tag is also good, used for good Publishing.

No matter what tools are used, you must make proper specifications, such as git branch naming. As the number of branches grows over a long period of time, it will be messy if it is not standardized.
For example, how to merge the branches, and how not to merge them.
The process of the branch, where to submit the test, where to publish, and so on.
To resolve conflicts between branch merge, you must communicate with each other in advance. Otherwise, the code is lost and cannot be found.

Iii. Commencement
1. Set up a shelf
It is better to monitor the system here. Monitoring is too important and should be considered clearly at the beginning. Otherwise, it will be painful to add more information later.
If you want to use a framework such as hystrix, you must first understand the implementation method, which is too painful to implement later.
2. Code specifications
At the beginning, the architect thought perfectly. If there were no rules, there would be too many people going back and forth, and there would be a mess in it.
Of course, even if you have made a standard, it will be messy, but it will be a little messy.
3. Some basic code
For example, logging, http requests, how to add monitoring, how to call rpc, and online interface query documentation.
4. debugging
After the code is written, how to run it, how to debug it, whether to use jetty or tomcat locally, and how to release the test server should be adjusted according to the actual situation.
5. Development Tools
It seems a bit late now. For java, some people like eclipse and some like idea. Another example is the git management tool. Some people like sourcetree and others like to execute commands, which makes it more difficult to unify.
6. Code Review
It's okay to check the code written by your colleagues. Isn't the Code placed in the correct place? Is there any duplicate code? Don't mess up your shelf.
Findbugs, this plug-in is very useful. You can find many low-level and unexpected problems, such as possible null references and String. format formatting problems.
7. Team communication
There will be many things, and there will certainly be some strange problems. At this time, we need to communicate and solve them together.
To solve the underlying framework problems, you must fix and optimize them quickly.
Process Problems, optimize the process specifications, and notify the team.
It would be better to have a wiki, a forum, and so on to remember the pitfalls that everyone has stepped on.
Regularly hold regular meetings to synchronize progress and synchronization problems.

Iv. Test
After the code is written, a powerful test team is required to test the function.
Our project is an interface that outputs json data externally. If there is no automated comparison tool, it will go crazy with the naked eye.
Missing fields, Case sensitivity issues, format issues, and various issues will be exposed one by one. It is time to optimize the specifications.
1. Automated Testing
I am not quite familiar with the test strength. how important it is to have a test bull.
2. Interface documentation
When developing interfaces, you must update the document and modify the code to update the document in a timely manner.
Of course it is too annoying to write documents. java annotations can be used for this and then automatically generate online documents.
3. Stress Testing and concurrency Testing
Whether the interface can support the business or how to evaluate it depends on the pressure test results. Not up to standard? Optimized!

5. Launch
At this time, you have to figure out how to standardize the release process. If the company has a team to help you automate the integration and release, it would be awesome. If not, you have to renew it on your own.
For jenkins, package the script and start it on your own.
What, the Code server and the online server are not in the same IDC? It doesn't matter. svn, git helps you solve it. Tomcat also supports uploading war packages.
1. Apply for server
Evaluate the number of servers based on traffic.
Do you want to split interfaces? For example, the abc interface only accesses server A, the def interface only accesses server B, and the nginx access layer is used up.
2. Release
Release process and system.
Do you need phased release?
Can I roll back quickly.
Configure the management system.
3. Monitoring
One of the most important tasks for a programmer is monitoring.
What call volume, peak call volume, success rate, failure rate, timeout rate, average time consumption, etc.
It is best to view the code returned by the interface call online to quickly locate the problem.
In addition to the monitoring of app calls to your service, you can also call the monitoring of others' services. One is to discover your own problems, and the other is to discover other people's problems. evidence must be provided to both force and force.
Failure Rate, timeout rate, and average time consumption are necessary and are also the KPIs of dairy programmers.
It is better to monitor cpu, memory, and nic usage.
Further, it would be even better if jvm monitoring is available!
Monitoring has been done, alarm must be available, text messages, emails, you know!
4. Logs
The second most important task for a programmer is even more important than monitoring.
Monitoring is delayed, and sometimes problems cannot be detected in time. It is important to view the error log at this time.
Whether the system is running normally is determined by logs.
The error log will help you discover the problem, or even solve the problem before you discover it.
The programmer needs to check whether the logs are too many. If the number of logs is too low, the number of logs is incorrect. Are there any important business logs. This is a long-term process and will be adjusted gradually based on the situation.
Observing the logs of online servers is a top priority for milk Dad. It is not easy for her to discover problems before the user, solve problems before the customer service event, and escort!
Therefore, you 'd better have a log storage system to view the filtering system online.
5. Stability assurance
In the nginx access layer, nginx is used as the access layer before tomcat, and nginx is used for distribution and server grouping. The surrounding services are suspended and core services cannot be affected.
Another benefit of nginx is forwarding. For example, if you have an a.site.com site to be revised, the domain name of the old version a.site.com cannot be changed. The new version is on another server, and the domain name of a.site.com must also be used, then the functions of the nginx access layer will be apparent. All the new pages will be directed to the new site, and other pages will be directed back to the old site.

There are also a large number of microservices behind you, and each microservice may be dependent on each other. If one of them fails, it may be a disaster for you. How can we avoid this situation?
This is what we are doing recently. After a round of searching, we are all talking about Netflix's Hystrix. This should be a mature solution.
The overall deployment lockdown and downgrade can only ensure that microservices are not exposed to the original market, but cannot ensure that errors occur at the front end. Therefore, the combination of the front end provides better prompts of soft and good experience errors.

Last
My dad's work is more than that. It's not that easy to be a parent.
Discuss requirements with products, troubleshoot customer service incidents, discuss technical solutions with colleagues, test bugs, and report to leaders. What? Why can't the app be opened again? What? Is the data center faulty? What? Someone is refreshing the interface? Never Ending...

------------------------------------------------------

Say a few traps you have stepped on:
1. network card full
When the network card is full for the first time, I think it is amazing that there is A feeling between Niu A and Niu C. So far, it is nothing new that the network card is full.
During stress testing, qps was always unavailable. Later I found that the NIC of the stress testing machine was full.
The service crashes, the Cache Server Nic is full, and the storage of a single key is too large.
Summary: The network card is full of things that have to be considered, especially when you transfer back and forth. The solution is that the value of a single key should not be too large. No matter what the cache is, as the value size increases, the performance decreases sharply. Therefore, it can be split and distributed cache is used now, the more keys, the more average each server will have, and the increase in keys will have little impact on performance.

2. tomcat is always restarted.
As soon as the peak period reached, tomcat was restarted automatically. The reason was not found, and my father was challenged.
As I said above, if your child is ill and you cannot take any medicine, ask the doctor to prescribe it and ask others to cure it.
Later, dump and tcp monitoring were performed during tomcat restart. Later, it was found that there were too many wait requests at the time of tomcat restart. It was inferred that a service call encountered a performance problem and there was too much backlog, if it is accumulated to a certain extent, it will explode.
Later, it was done by increasing the number of rpc calls of a service.
Another point is that you should be careful when using the synchronized keyword. Be careful when performance problems occur, which may cause congestion.

3. The monitoring curve suddenly disappeared.
The curve suddenly disappeared and fell to 0. Then, the system immediately gave up the alarm and checked the log. It found that an error was reported during monitoring reporting and the reported thread crashed.
Change it. If an error is reported, it will be wrong!

-----------------------------------------------------------------

Summary:
The above are some of my experiences. It may be nothing special for you. You can only say that you are not easy to learn and cannot keep up with the pace of technological development. If you have a better experience, please don't hesitate to give us some advice and let everyone grow together!
All of the above is huge for small teams in small companies or projects from scratch. Monitoring systems, log systems, O & M, and publishing are everywhere for you to step on. The rpc call scheme, cache, and mq we currently use are all third-party. Although we have technical support, we still have a lot of pitfalls.
There are traps everywhere. What we need to do is to step on the trap and not fall down!

Happy New Year!