Experience sharing-a workaround for disk space not being released after deleting large files under Linux

This blog is very early registration, but has not been updated, recently resigned retreat to repair, strive to write a technical blog every day ~ ~

Not much gossip, just start to cut to the point ~

It happened in 2014 April, life's first job-hopping, the successful entry of a newly started CDN company, did an incompetent research and development operations, was just in time to catch up with the former OPS, the first company to use more than squid, As the company is now the team of squid more understanding of the people have gone, in addition to more squid has a serious disadvantage, is that the contents of the cache between many squid processes can not be shared, sometimes the same file will be cached many points, very waste of disk space, Because many squid process scheduling algorithm at that time did not perfect, resulting in cache hit rate is not high ...

(here to explain to the friends who do not know squid, why to open more squid process on the same server, squid is a relatively "old" reverse proxy cache software, it uses a single process architecture, low efficiency, in order to improve efficiency, In this machine to open a number of squid process to provide services, which is the above said more squid)

According to the above problems encountered, in addition to the company has not been recruited to squid special knowledge of the research and development staff (two or three-line city ... You know... ）

At that time decided to the operation of the multi-squid replaced with nginx, the individual think with Nginx to do reverse proxy and cache, the efficiency will be much higher than squid, the main reasons are the following two points.

1. Nginx natively supports multiple processes, which are shared across multiple processes, and the same file is not cached for multiple points.

2. Configuration is more convenient and flexible, can be different customer site configuration files into multiple sub-configuration files
   

At that time the company's business volume is not particularly large cases, obtained the approval of the eldest, ready to slowly move the company's cache business from squid to nginx, the entire structure of a transformation.

Nginx cache trial for half a month, compared with the previous squid, disk utilization, and hit rate has been significantly improved ~

(Remember that it was almost a year since the operation, see the company's CDN architecture optimization has made little progress, then felt the feeling of accomplishment)

But as a commercial CDN, it's so easy to replace the cache software to make changes to the current architecture ... Because the squid replaced with Nginx, the log record format is completely different from the previous, resulting in the previous log collection system, billing system, cache refresh push system, all have to re-get.

Because it involves the customer's billing, refresh and other issues, at that time more anxious, their own lack of work experience, wrote a shell script + planning task to do a log cutting, log cutting processing rough, in addition to the cache at the time of the Max_ Size does not give a good fit (when the cache and log are placed under the same partition ...) Due to the lack of work experience and not taking into account how much disk space the log occupies, the result is to bury themselves a relatively large "pit".

The new architecture formally launched two weeks or so, before the time bomb finally broke out, there are several areas of the upper intermediate node can not provide services, after troubleshooting the error log, the final confirmation is left to the Nginx cache partition is almost full.

At that time, it is relatively simple, full of the deletion of the log, when trying to avoid deleting the cache, because it will reduce the hit rate, and log files occupy a larger space, then the day's log files were deleted, but the disk space is still not released.

After that, we looked at the sum of all the files in the next/data directory, and found that the sum of all the files in this directory was much lower than the size of the partition, but I couldn't find out where the file that occupies the most disk ...

At that time, the business through the LVS to cut the business to other transit nodes, the mistake of restarting the server, the result, disk space was released, back to normal.

Later on the internet search those great God wrote documents, just know, this and the Linux file deletion principle has a direct relationship!!

I will describe this knowledge point in the document of the Great God in a concise and succinct description.

In the Linux system, want to completely delete a file, depending on two "counter", the two counters is a disk reference "counter" (record the file has several hard links), and the other is memory Reference "counter" (record this file is being called by several processes), when the two " The counter "All is 0, that is, this file does not have a hard link, no process in the call, this file will be really deleted."

This is the reason why the disk space is still not released because a large log file was deleted before.

Let's analyze it for a moment.

The current cache access log, is being used by Nginx, directly rm deleted the log file, in fact, only the disk reference "counter" set to 0, but the nginx process is still calling this log file, the log file to write something, memory Reference "counter" is not set to 0, So this log file is not really deleted, resulting in space can not be released!!!

At this time due to lack of experience of the mistakes made ... So remind the novice like me that when the program is running, the log file being written must not be deleted directly by RM, even if it is deleted, the space will not be released.

finally do a supplement ~, found on the Internet a Linux command, you can view the deleted, but still be occupied by the program file name-level path.

lsof | grep del

After this situation, you can use this command to see if there is a clear has been deleted, but still be occupied by the program files.

If the log file is being written by the program and you want to empty it

You can use the following methods

Cat/dev/null >/data/access.log

This article is from the "Rebirth" blog, make sure to keep this source http://suhaozhi.blog.51cto.com/7272298/1904976

Experience sharing-a workaround for disk space not being released after deleting large files under Linux