Tomcat starts very slowly and is stuck in root webapplicationcontext:initialization completed at boot time

Every time you restart your service tomcat needs to be stuck for a long time, each time the log stops at
Root Webapplicationcontext:initialization completed in 744 ms this place and then don't know what's going on waiting
What, I saw a blog post on the internet, mark

Problem phenomenon

Beauty colleagues to find me to solve a problem, said Tomcat started very slowly. At first I thought it was a problem with the program, so I removed all the programs under WebApps. (keep Tomcat only) The supernatural thing happened, and Tomcat stopped in--

I looked at the process, and the JVM process that Tomcat was in was started, so it was possible to exclude the problem caused by the JVM's exit. So the problem is that the JVM is, for some reason, BlockingThe
Analysis

The problem is tricky, I ruled out the CPU, insufficient memory problems caused by the lack of hard disk space caused by problems, I even observed the network I/O, hard disk I/O situation, are very normal. Program is blocked in general it must be waiting for a resource, and now the situation is that all resources are plentiful, so I can hardly imagine what the problem is. I started to suspect that it was the KVM hypervisor virtualization problem (with a virtual machine) I changed my strategy. Download the tomcat boot directly on VMware's two virtual machines. One of them soon started and the other one was blocked , and the problem was reproduced. See To face in front of the beautiful woman, my glorious and magnificent image to vanish. In this case I can't go to the "masturbate" code it? Moreover tomcat so many people use, really has such obvious bug already fried boiling. (Tomcat is still very reliable, not like xxxxstack.) Think about what I need to find out where Tomcat stopped. There's something going on in the code, but I can't go to the code. I decided to try the Strace, which is a tool for tracking system calls , either Java or Pyhton many resource requests become system call. (such as opening a file, creating a new thread, reading and writing data, waiting for I/O) through this tool I can at least know which system call the Tomcat is stopping on , so that I can infer the cause of the problem.
Strace-f-o strace.out./catalina.sh Run

Strace has a lot of parameters, I used two parameters
-F tracks the sub-processes of the fork, which, in layman's words, keeps track of all thread system calls

(
-F--follow forks,-FF--with output into separate files

)

-O to output the content to a file

Other parameters please search by yourself below to analyze the Strace.out file, the method of analysis is from bottom to top (the blocked place must be at the end). First we need to get rid of the system call caused by Tomcat stop, they are not what we need. Search from forward to find SIGINT

The red section above is the system call that caused the blockage, with a whole bunch of futex
Call, which is a lightweight synchronization method in Linux, so we can tell that the top of the list is definitely the real culprit for blocking. Skip all the Futexthis read
is to cause a string of Futex to follow .
The real reason that strace very clever it not only gives the system call back the parameters and return values passed, read reads the 51st file handle, and does not return success (unfinished). Along the way, let's see what the 51st file handle is./dev/random is a random function generator under Linux, and reading it is equivalent to generating random numbers. Search for it, the first one is the wiki so it seems that all the truth,/dev/random will be based on NoiseGenerates a random number if NoiseIt will clog up if it is not enough. Linux is collected by means of I/O, keyboard terminals, memory usage, CPU utilization, etc. NoiseOF, if NoiseWhen a random number is not generated, it will be Blocking。
In-depth analysis

If we use tomcat/dev/random as a keyword, we can basically answer our doubts. Thesession ID of the TOC mat is computed by the SHA1 algorithm, which must have a key when calculating the session ID . To improve security, Tomcat generates a key at random when it is started. There is an explanation in Http://wiki.apache.org/tomcat/HowTo/FasterStartUp (Entropy source section). StackOverflow also has a large number of these instructions, so there is no more introduction. Understand the cause of the problem is very simple to solve--replace/dev/random to/dev/unrandom, with pseudo-random function generator (/dev/urandom) to replace the random function generator (/dev/random).
By modifying the Tomcat boot file-djava.security.egd=file:/dev/urandom

By modifying the java.security file in the JRE securerandom.source=file:/dev/urandom

Of course, the JVM developer is not a fool, and the Tomcat developer is not 250. The reason why they did not choose/dev/urandom is to improve the security of the system,/dev/urandom is not really random behavior. (In fact,/dev/urandom is also safe enough to be "repetitive")
Solve the problem thoroughly

The two methods described above are replaced/dev/random with/dev/urandom, in fact there is a third way-increasing the entropy pool of/dev/random. The problem is that because the entropy pool is not large enough, it is the most thorough way to increase it. by Cat/proc/sys/kernel/random/entropy_avail
We can look at the current entropy pool size; we need to find a way to raise that value. If your CPU has DRNG features, you can take advantage of the hardware to increase the speed of the entropy pool. by Cat/proc/cpuinfo | grep Rdrand
You can see if your CPU is supported, and generally Intel's CPU for the Ivy_bridge architecture is supported (i3, i5 need to be aware of that architecture, i7 and Xeon are basically supported), and AMD's CPUs are built after 2015. (If you are a virtual machine you need to turn on additional parameters). If your hardware does not support it, it does not matter, we can let/dev/unrandom to do "entropy source". Taking Centos7 as an example,
Yum Install Rngd-tools
or yum install Rng-tools
Installing the RNGD Service (entropy service)

Systemctl Start Rngd
Start the service
If your CPU doesn't support the DRNG feature or uses a virtual machine like I do, you can use/dev/unrandom to simulate.

Cp/usr/lib/systemd/system/rngd.service/etc/systemd/system

Edit/etc/systemd/system/rngd.service
Service Summary, Execstart=/sbin/rngd-f-r/dev/urandom

Systemctl Daemon-reload
Re-loading Service

Systemctl Restart Rngd
Restart Service

After the above modification, we observe/proc/sys/kernel/random/entropy_avail
Basically at about 3000. We can test the generation speed of random numbers.
Watch-n 1 Cat/proc/sys/kernel/random/entropy_avail
Observe this value

Open a new shell and test the random number with the DD command. DD If=/dev/random Of=random.dat count=40960

[[email protected] bin]# dd if=/dev/random of=random.dat count=40960 recorded 0+40960 's read-write 3074362 bytes (6004+1 MB) copied, 5.01017 sec, 614 kb/sec

40,960 random numbers are generated in 5 seconds, and the/proc/sys/kernel/random/entropy_avail will change drastically, and all random numbers will remain at around 3000 after they are generated.
which solution to choose

Personal advice to choose a third way, the entropy pool is not only Tomcat, all applications under Linux generate random numbers will use this, so not only Tomcat may be blocked . If you search you will find Apache, Nginx, OpenSSL have been the problem of the pit . If we modify the Java configuration to solve this problem is only to solve the problem of Java applications, can only be a palliative to the root. The method of radical cure should be through RNGD
Increase the speed of random number generation.
Summarize

Experience is not an experience. Using other people's experience to solve a problem is not difficult, it is difficult to start from the beginning of the road, more difficult is to overthrow the experience of predecessors to a problem can have their own views and understanding. This case deepened my understanding of strace.
Understanding, for the air refueling
This type of system debugging has its own experience, and I've found a better way to do it through deep analysis of the causes. This is Kant's spirit --thinking, criticizing, rationality.
How to Reproduce the failure

Can easily reproduce the failure described in the article
Systemctl Stop Rngd
Stop RNGD Service (if you have boot Rngd)

View the current entropy pool size Cat/proc/sys/kernel/random/entropy_avail

Head-c1024/dev/random
, forcing the consumption of 1024 random numbers, the system does not respond for a long time. Direct CTRL + C

Check the size of the entropy pool again Cat/proc/sys/kernel/random/entropy_avail
, make sure it's as small as possible.

Start Tomcat, and you'll find long waiting times

There is also a solution

In the Sun Bug database has been given, that is, in the Java program startup parameters added:-djava.security.egd=file:/dev/urandom, using/dev/urandom to generate random numbers.

Linux Modify catalina.sh File

java_opts= "-server-dfile.encoding=utf-8-XMS=512M-XMX1024M-XX:PERMSIZE=128M-XX:MAXPERMSIZE=256M-VERBOSE:GC- Xloggc:${catalina_home}/logs/gc.log ' Date +%y-%m-%d-%h-%m '-xx:+useconcmarksweepgc-xx:+cmsincrementalmode-xx:+ PRINTGCDETAILS-XX:+PRINTGCTIMESTAMPS-NOCLASSGC "

Random number generators in Linux

In the Linux operating system, there is a special device file that can be used as a random number generator or pseudo-random number generator.

/dev/random

On reading, the/dev/random device returns random bytes that are less than the total entropy pool noise. /dev/random can generate a high-randomness public key or one-time cipher book . If the entropy pool is empty, the read operation on the/dev/random will be blocked until sufficient ambient noise is collected from other devices.

Of course you can also set to not plug, when you set the parameter O_nonblock when open, but when you read, if the entropy pool empty, will return 1

/dev/urandom

A copy of the/dev/random is/dev/urandom ("unlocked", a non-blocking random number generator [4]) that reuses data in the entropy pool to produce pseudo-random data. This means that the read operation on the/dev/urandom is not blocked, but its output entropy may be less than/dev/random. It can be used as a pseudo-random number generator for generating lower strength passwords and is not recommended for generating strong long-term passwords.

Reference:

https://www.jianshu.com/p/576d356dc163

Tomcat starts very slowly and is stuck in root webapplicationcontext:initialization completed at boot time