After Tomcat has run normally for a while, Tomcat stops abnormally and the process is killed

Tomcat abnormal stop, process is killed

The corresponding Tomcat logs are as follows:

/application/tomcat-service-8080/bin/catalina.sh:line 386:4370 killed "/usr/local/jdk/bin/java" "- Djava.util.logging.config.file=/application/tomcat-service-8080/conf/logg
Ing.properties "-DJAVA.UTIL.LOGGING.MANAGER=ORG.APACHE.JULI.CLASSLOADERLOGMANAGER-SERVER-XMS4096M-XMX4096M-XX: Permsize=128m-xx:maxnewsize=512m-xx:maxpermsize=256m-djava.awt.headless=tr
ue-djava.endorsed.dirs= "/application/tomcat-service-8080/endorsed"-classpath

No error in log, no memory overflow outofmemoryerror

After Google learned about a Linux OOM Killer

The Linux kernel allocates memory according to the requirements of the application, usually the application allocates memory but is not actually all used, in order to improve performance, this part of the useless memory can be reserved for its use, this part of the memory belongs to each process, the kernel is directly recycled, it is more troublesome, So the kernel uses an over-allocating memory (Over-commit memories) method to indirectly utilize this part of "idle" memory to improve the efficiency of overall memory usage. This is generally not a problem, but when most applications consume their own memory, the trouble comes when the memory requirements of these applications add up beyond the capacity of physical memory (including swap), and the Kernel (OOM killer) must kill some processes to make room for the system to function properly. In the case of banks, it may be easier to understand, some people take money when the bank is not afraid, the bank has enough to deal with, when the National people (or the vast majority of) to withdraw money and everyone wants to take their own money out of the bank when the trouble comes, the bank actually does not have so much money for everyone to take.

The kernel detects that the system is out of memory, picks up and kills a process, can refer to the kernel source code LINUX/MM/OOM_KILL.C, Out_of_memory () is triggered when the system is low on memory, and then calls Select_bad_process () Choose a "bad" process to kill, how to judge and choose a "bad" process, can not be randomly selected it? The process of selection is determined by oom_badness (), and the algorithm and ideas chosen are simple and straightforward: the worst process is the most memory-intensive process.

View/var/log/messages

2 20:01:03 iz25crr8mcwz systemd:started Session 65527 of user root.
2 20:10:14 iz25crr8mcwz systemd:created slice user-0.slice.
2 20:10:25 iz25crr8mcwz systemd:starting Session 65528 of user root.
2 20:10:53 iz25crr8mcwz systemd:started Session 65528 of user root.
2 20:12:24 iz25crr8mcwz Kernel:java invoked oom-killer:gfp_mask=0x201da, order=0, oom_score_adj=0
2 20:12:24 iz25crr8mcwz Kernel:java cpuset=/mems_allowed=0
2 20:12:24 iz25crr8mcwz kernel:cpu:0 pid:4385 comm:java tainted:gf O--------------3.10.0-123.9.3.el7.x86_64 #1
2 20:12:24 iz25crr8mcwz kernel:hardware name:qemu Standard PC (I440FX + Piix, 1996), BIOS rel-1.7.5-0-ge51488c-20140 602_164612-nilsson.home.kraxel.org 04/01/2014
2 20:12:24 iz25crr8mcwz kernel:ffff880008bf0b60 00000000f4c1c64c ffff880230453938 ffffffff815e239b
2 20:12:24 iz25crr8mcwz kernel:ffff8802304539c8 ffffffff815dd372 ffffffff810b69b8 ffff880036a9a050
2 20:12:24 iz25crr8mcwz kernel:0000000000000202 ffff880008bf0b60 ffff8802304539b0 FFFFFFFF811030AF
2 20:12:24 iz25crr8mcwz Kernel:call Trace:
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff815e239b>] dump_stack+0x19/0x1b
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff815dd372>] dump_header+0x8e/0x214
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff810b69b8>]? Ktime_get_ts+0x48/0xe0
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff811030af>]? Delayacct_end+0x8f/0xb0
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff8114526e>] Oom_kill_process+0x24e/0x3b0
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff81144dd6>]? Find_lock_task_mm+0x56/0xc0
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff81145a96>] Out_of_memory+0x4b6/0x4f0
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff8114b5d9>] __alloc_pages_nodemask+0xa09/0xb10
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff81188349>] alloc_pages_current+0xa9/0x170
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff81141a67>] __page_cache_alloc+0x87/0xb0
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff81143da8>] filemap_fault+0x188/0x430
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff81167e9e>] __do_fault+0x7e/0x520
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff8116c1e5>] Handle_mm_fault+0x3e5/0xd90
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff815edb06>] __do_page_fault+0x156/0x540
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff810c2be2>]? Do_futex+0x172/0x5b0
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff8109b836>]? __dequeue_entity+0x26/0x40
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff81011619>]? __switch_to+0x179/0x490
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff815edf0a>] do_page_fault+0x1a/0x70
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff810b6769>]? Do_gettimeofday+0x29/0x70
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff815ed599>] Do_async_page_fault+0x29/0xe0
2 20:12:24 iz25crr8mcwz kernel: [<ffffffff815ea178>] async_page_fault+0x28/0x30
2 20:12:24 iz25crr8mcwz Kernel:mem-info:
2 20:12:24 iz25crr8mcwz kernel:node 0 DMA per-cpu:
2 20:12:24 iz25crr8mcwz kernel:cpu 0:hi:0, Btch:1 usd:0
2 20:12:24 iz25crr8mcwz kernel:cpu 1:hi:0, Btch:1 usd:0
2 20:12:24 iz25crr8mcwz kernel:cpu 2:hi:0, Btch:1 usd:0
2 20:12:24 iz25crr8mcwz kernel:cpu 3:hi:0, Btch:1 usd:0
2 20:12:24 iz25crr8mcwz kernel:node 0 DMA32 per-cpu:
2 20:12:24 iz25crr8mcwz kernel:cpu 0:hi:186, btch:31 usd:84
2 20:12:24 iz25crr8mcwz kernel:cpu 1:hi:186, btch:31 usd:113
2 20:12:24 iz25crr8mcwz kernel:cpu 2:hi:186, btch:31 usd:109
2 20:12:24 iz25crr8mcwz kernel:cpu 3:hi:186, btch:31 usd:45
2 20:12:24 iz25crr8mcwz kernel:node 0 Normal per-cpu:
2 20:12:24 iz25crr8mcwz kernel:cpu 0:hi:186, btch:31 usd:0
2 20:12:24 iz25crr8mcwz kernel:cpu 1:hi:186, btch:31 usd:0
2 20:12:24 iz25crr8mcwz kernel:cpu 2:hi:186, btch:31 usd:0
2 20:12:24 iz25crr8mcwz kernel:cpu 3:hi:186, btch:31 usd:0
2 20:12:24 iz25crr8mcwz kernel:active_anon:1714079 inactive_anon:38316 isolated_anon:0
active_file:106 inactive_file:169 isolated_file:0
unevictable:0 dirty:0 writeback:0 unstable:0
free:25831 slab_reclaimable:17057 slab_unreclaimable:6114
mapped:3441 shmem:96445 pagetables:4873 bounce:0
free_cma:0
2 20:12:24 iz25crr8mcwz kernel:node 0 DMA free:15900kb min:132kb low:164kb high:196kb active_anon:0kb inactive_anon:0 KB active_file:0kb inactive_file:0kb unevictable:0kb Isolated (anon):
0kB Isolated (file): 0kB present:15992kb managed:15908kb mlocked:0kb dirty:0kb writeback:0kb mapped:0kb shmem:0kb slab_ RECLAIMABLE:0KB slab_unreclaimable:8kb kernel_stack:0kb pagetables:0kb uns
TABLE:0KB bounce:0kb free_cma:0kb writeback_tmp:0kb pages_scanned:0 all_unreclaimable? Yes
2 20:12:24 iz25crr8mcwz kernel:lowmem_reserve[]: 0 2801 7792 7792
2 20:12:24 iz25crr8mcwz kernel:node 0 DMA32 free:44996kb min:24252kb low:30312kb high:36376kb Active_anon:2645684kB I NACTIVE_ANON:86640KB active_file:352kb inactive_file:608kb Unevictab
LE:0KB Isolated (anon): 0kB isolated (file): 0kB present:3129216kb managed:2869240kb mlocked:0kb dirty:0kb writeback:0kb mapped:10716kb shmem:230460kb slab_reclaimable:35324kb slab_unreclaimable:
8996kB kernel_stack:2048kb pagetables:9864kb unstable:0kb bounce:0kb free_cma:0kb writeback_tmp:0kb pages_scanned : 1747 all_unreclaimable? Yes
2 20:12:24 iz25crr8mcwz kernel:lowmem_reserve[]: 0 0 4990 4990
2 20:12:24 iz25crr8mcwz kernel:node 0 Normal free:42428kb min:43192kb low:53988kb high:64788kb Active_anon:4210632kB INACTIVE_ANON:66624KB active_file:72kb inactive_file:68kb Unevictabl
E:0KB Isolated (anon): 0kB isolated (file): 0kB present:5242880kb managed:5110124kb mlocked:0kb dirty:0kb writeback:0kb mapped:3048kb shmem:155320kb slab_reclaimable:32904kb slab_unreclaimable:15
452kB kernel_stack:3408kb pagetables:9628kb unstable:0kb bounce:0kb free_cma:0kb writeback_tmp:0kb pages_scanned:347 All_unreclaimable? Yes
2 20:12:24 iz25crr8mcwz kernel:lowmem_reserve[]: 0 0 0 0
2 20:12:24 iz25crr8mcwz kernel:node 0 dma:1*4kb (u) 1*8kb (u) 1*16kb (u) 0*32kb 2*64kb (U) 1*128kb (u) 1*256kb (u) 0 *512KB 1*1024kb (U) 1*2048kb (R) 3*4096kb (M) = 15900kB
2 20:12:24 iz25crr8mcwz kernel:node 0 dma32:1204*4kb (UEMR) 950*8kb (UEM) 1208*16kb (UEM) 252*32kb (UEM) 72*64kb (UE M) 6*128kb (UE) 0*256kb 0*512kb 0*1024kb 0*2048kb 0*4096kb = 45184kB
2 20:12:24 iz25crr8mcwz kernel:node 0 normal:400*4kb (UEM) 326*8kb (UEM) 543*16kb (UEM) 237*32kb (EM) 145*64kb (UEM) 61*128KB (UEM) 15*256kb (UEM) 0*512kb 1*1024kb (M) 0*2048kb 0*4096k
B = 42432kB
2 20:12:24 iz25crr8mcwz kernel:node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 Hugepages_size=2048kB
2 20:12:24 iz25crr8mcwz kernel:96713 Total pagecache pages
2 20:12:24 iz25crr8mcwz kernel:0 pages in swap cache
2 20:12:24 iz25crr8mcwz Kernel:swap cache stats:add 0, delete 0, find 0/0
2 20:12:24 iz25crr8mcwz Kernel:free swap = 0kB
2 20:12:24 iz25crr8mcwz kernel:total swap = 0kB
2 20:12:24 iz25crr8mcwz kernel:2097151 pages RAM
2 20:12:24 iz25crr8mcwz kernel:94167 pages reserved
2 20:12:24 iz25crr8mcwz kernel:274302 pages shared
2 20:12:24 iz25crr8mcwz kernel:1961600 pages non-shared
2 20:12:24 iz25crr8mcwz kernel: [PID] uid tgid total_vm RSS nr_ptes swapents Oom_score_adj Name
2 20:12:24 iz25crr8mcwz kernel: [338] 0 338 14844 3705 0 0 systemd-journal
2 20:12:24 iz25crr8mcwz kernel: [354] 0 354 26113 0 0 Lvmetad
2 20:12:24 iz25crr8mcwz kernel: [357] 0 357 10509 149 from 0-1000 SYSTEMD-UDEVD
2 20:12:24 iz25crr8mcwz kernel: [510] 0 510 164576 2774 0 0 rsyslogd
2 20:12:24 iz25crr8mcwz kernel: [513] Bayi 513 7158 101-0-900 Dbus-daemon
2 20:12:24 iz25crr8mcwz kernel: [515] 0 515 31583 163 0 0 Crond
2 20:12:24 iz25crr8mcwz kernel: [519] 519 363742 743 0 0 NSCD
2 20:12:24 iz25crr8mcwz kernel: [609] 0 609 1621 8 0 0 Iprupdate
2 20:12:24 iz25crr8mcwz kernel: [612] 0 612 1621 9 0 0 Iprinit
2 20:12:24 iz25crr8mcwz kernel: [649] 0 649 9781 9 0 0 Iprdump
2 20:12:24 iz25crr8mcwz kernel: [850] 850 7399 155 0 0 ntpd
2 20:12:24 iz25crr8mcwz kernel: [895] 0 895 386 4 0 0 Aliyun-service
2 20:12:24 iz25crr8mcwz kernel: [19586] 0 19586 20697, 0-1000 sshd
2 20:12:24 iz25crr8mcwz kernel: [22773] 996 22773 21196 206 0 0 Zabbix_agentd
2 20:12:24 iz25crr8mcwz kernel: [22774] 996 22774 21196 285 0 0 Zabbix_agentd
2 20:12:24 iz25crr8mcwz kernel: [22775] 996 22775 21226 238 0 0 Zabbix_agentd
2 20:12:24 iz25crr8mcwz kernel: [22776] 996 22776 21226 238 0 0 Zabbix_agentd
2 20:12:24 iz25crr8mcwz kernel: [22777] 996 22777 21226 254 0 0 Zabbix_agentd
2 20:12:24 iz25crr8mcwz kernel: [22778] 996 22778 21196 237 0 0 Zabbix_agentd
2 20:12:24 iz25crr8mcwz kernel: [13392] 0 13392 59021 2804 0 0 salt-minion
2 20:12:24 iz25crr8mcwz kernel: [13395] 0 13395 170977 5407 0 0 salt-minion
2 20:12:24 iz25crr8mcwz kernel: [1901] 0 1901 30507 151 0 0 Wrapper
2 20:12:24 iz25crr8mcwz kernel: [1911] 0 1911 627491 17315 0 0 Java
2 20:12:24 iz25crr8mcwz kernel: [5580] 0 5580 6338 162 0 0 Aliyundunupdate
2 20:12:24 iz25crr8mcwz kernel: [7267] 0 7267 48303 1580 0 0 Aliyundun
2 20:12:24 iz25crr8mcwz kernel: [4955] 0 4955 27501 0 0 Agetty
2 20:12:24 iz25crr8mcwz kernel: [18909] + 18909 28313 0 0 catalina.sh
2 20:12:24 iz25crr8mcwz kernel: [18910] + 18910 1077 8 0 0 Cronolog
2 20:12:24 iz25crr8mcwz kernel: [18911] + 18911 1245006 596619 1349 0 0 Java
2 20:12:24 iz25crr8mcwz kernel: [19657] 0 19657 8671 0 0 Systemd-logind
2 20:12:24 iz25crr8mcwz kernel: [4368] + 4368 28313 0 0 catalina.sh
2 20:12:24 iz25crr8mcwz kernel: [4369] + 4369 1077 7 0 0 Cronolog
2 20:12:24 iz25crr8mcwz kernel: [4370] + 4370 1796041 1028084 2224 0 0 Java
2 20:12:24 iz25crr8mcwz kernel: [30951] 0 30951 33414 294 0 0 sshd
2 20:12:24 iz25crr8mcwz kernel: [31046] + 31046 33447 311 0 0 sshd
2 20:12:24 iz25crr8mcwz kernel: [31047] + 31047 29136 402 0 0 Bash
2 20:12:24 iz25crr8mcwz kernel:out of Memory:kill process 4370 (Java) score 514 or sacrifice child
2 20:12:24 iz25crr8mcwz kernel:killed process 4370 (Java) total-vm:7184164kb, anon-rss:4112336kb, file-rss:0kb
2 20:12:25 iz25crr8mcwz systemd-logind:removed session 65322.
2 20:20:01 iz25crr8mcwz systemd:created slice user-0.slice.
2 20:20:01 iz25crr8mcwz systemd:starting Session 65529 of user root.

The service killed the large memory process 4370, so the Tomcat process unexpectedly disappeared

After Tomcat has run normally for a while, Tomcat stops abnormally and the process is killed