Troubleshooting of too temporary open files

 

Reprinted with the source. Thank you ~

Http://blog.csdn.net/shootyou

 

 

Some time ago, I helped the company solve a tomcat environment server exception. I have time to summarize it.

 

Problem:

After the Tomcat service is restarted, it will bear a large amount of access in a short period of time. Because the cache has not yet been established, each access consumes a certain amount of resources (database connection or file IO ), when the concurrency is around 2000, the Tomcat service starts to throw a large number of too program open files exceptions, mainly the exception of file I/O. If the database connection pool is configured properly, no exception will be thrown.

 

Problem Analysis:

This is a typical exception in the exhaustion of file handles. in Linux, "Everything is a file", so although the prompt "file" is opened too much, in fact, there may also be too many sockets or too many devices.

 

What is the file handle?

 

The handle is used as a person's name, and "you can use the name (handle) to reference and access sentences", it seems like this metaphor is close, but the problem is that many people may call the same name, but the handle makes sense because it is unique, so this metaphor also has a problem, the handle indicates that your process is connected to the external I/O. Is it more appropriate to regard it as a connection name? A file handle is a 16-bit binary code (32-bit unsigned integer after Windows95) that represents the channel number of the opened file. With this handle, your application can use the corresponding handle to perform random access to the file. To put it bluntly, the file handle is actually a string of numbers with special meanings; of course, the system uses handles to associate with some resources. When managed by the system, the code of some resources dynamically allocated to your application is, you can use the handle to access the corresponding resources, especially in Windows systems, there are many things that use the handle, such as window and socket.

In short, the program obtains the resource reference through the handle to open and close the resource.

 

Why is the file handle exhausted?

 

Linux has two levels of restrictions on the number of file handles. One is the system-level total number limit, and the other is the user limit. By default, each user can use 1024 handles. In general, 1024 is enough, but in large-capacity systems, especially those that frequently use network communication and file IO, 1024 will soon be used up. So first, we need to adjust this value. The modification method is as follows:

1. ulimit-A: view the current user's file handle limit <br/> 2. Modify the user-level handle limit. <Br/> Modify/etc/security/limits. add the following code in conf: <br/> the user name (or use * to represent all users) soft nofile 65535 <br/> the user name Hard nofile 65535 <br/> has two restrictions, one is soft restrictions. When the number of Soft limits is exceeded, the system will give a warning of warning, but when the hard limit is reached, the system will reject or exception. <Br/> after modification, you may need to restart the shell to take effect. <Br/> 3. modify the number of system-level handles. <Br/> sysctl-w fs. file-Max 65536 <br/> or <br/> echo "65536">/proc/sys/fs/file-max <br/>, the former changes the kernel parameters, and the latter directly acts on the files corresponding to the kernel parameters in the Virtual File System (procfs, psuedo File System. <Br/> run the following command to view the new limits. <br/> sysctl-A | grep FS. file-max <br/> or <br/> CAT/proc/sys/fs/file-max <br/> modify Kernel Parameters <br/>/etc/sysctl. conf <br/> echo "FS. file-max = 65536 ">/etc/sysctl. conf <br/> sysctl-P <br/> View the System Restrictions command: CAT/proc/sys/fs/file-max <br/> run the following command to view the number of file handles currently used by the entire system: CAT/proc/sys/fs/file-Nr <br/> check the handle opened by a process: lsof-p pid <br/> A process has several handles: lsof-p pid | WC-L <br/> you can also see which process occupies a directory/file, and display information about all processes that have opened the directory or file: lsof path/filename <br/>

 

How much should this value be set?

Priority (open file descriptors ):
Soft limit

 

In fact, there is no specific limit on this value, but if the allocation value is too large, it will affect the system performance. Therefore, you need to balance the allocation according to the specific application.

 

Solution:

First of all, you must modify the number of Linux handles to a proper value.

Then there is an adjustment to the application itself. There are several situations:

1. Optimize the database connection pool. You must use the connection pool. Otherwise, the database crashes when the handle is not used up...

2. httpclient may be used when capturing resources. Use the connection pool as much as possible to control the number of connections.

For more information about httpclient connection pool configuration, see my other article: http://blog.csdn.net/shootyou/archive/2011/05/12/6415248.aspx

3. Grasp the settings of the Connection Pool, set the connection timeout time, read timeout time, number of connections, wait time, and so on, all need to be configured with a suitable value, otherwise the performance of the connection pool will not be available.

 

 

 

Reference: http://terrywang.net/archives/549