"Summary" Filebeat process writes full disk case processing

Using Filebeat to collect logs, log files frequently rotate, resulting in filebeat occupied files are not released, as long as the filebeat keep the deleted file open state, the operating system will not free disk space, resulting in a gradual decrease in available disk space.

Using the lsof command to view the file resources maintained by Filebeat, you can find many invalid file (deleted) files that are filebeat occupied space.

Deleted state files are not released and always occupy disk space

Workaround:

View filebeat configuration file location:/etc/filebeat/filebeat.yml

Add close_timeout:5m to the configuration file to ensure that file handler is closed every 5 minutes, regardless of whether you encounter the EOF symbol or not.

It is important to note that the Close_timeout parameter causes data loss in cases where filebeat is not processed to the end of the file and the file is deleted.

Filebeat.prospectors:
-Type:log
?? Paths
??? -/opt/apps/ecm/service/storm/1.0.1/package/apache-storm-1.0.1/logs/workers-artifacts/xyz*/*/worker.log
?? Tail_files:false
?? Force_close_files:true
?? Close_timeout:5m
Processors:
-Add_cloud_metadata: ~
Output.logstash:
?? Hosts: ["10.109.3.193:6667"]
?? Loadbalance:true
?? Worker:1

"Summary" Filebeat process writes full disk case processing