filebeat is not closing the file descriptor incase of file is been
removed/rotated with version 5.5 and due to which disk usage is full
14327 0 lrwx------ 1 xxx xxx 64 Oct 13 13:23 /proc/1079/fd/7 -> /tmp/ffi4PXrM7\ (deleted)
15953 0 l-wx------ 1 xxx xxx 64 Oct 13 13:23 /proc/1265/fd/3 -> /var/log/temp.log\ (deleted)
my filebeat configuration looks as below:
filebeat.prospectors:
document_type: log
So if filebeat queues up, it means the receiving end is probably not fast enough. Can you share your filebeat log and general setup? Please also share the full filebeat config and make sure the formatting stays correct (use 3 backticks before and after) or paste it into a gist and link it here.
Could you please respond. we are blocked on this and due to huge number of file descriptor is hanging for deleted files the disk is getting full and internal communication inside node is getting lost.
Currently once we restart the filebeat the disk space recovered but i understand this is not the right approach to be followed.
As mentioned above by Nicolas looks like log file content not transferred completely and there is a connection failure from filebeat to logstash in mean while is the file is rotated and removed.
There could be chances where the file descriptor is left open.
Have the same issue. Old logs are rename each hour, then delete after hour of inactivity by script. But filebeat still keep all or many of them open. Can't reproduce in test lab, only on prod get it. Maybe it's related to intensive of logging.
Config:
There are three almost the same servers with similar services but only one has this issue.
And only service restart or close_timeout option really help.
My logs are recreating (renaming) every hour and not populating after rename. So 2 hours is good for me. But of course I prefer correct work of filebeat instead restarting/timeout using.
It strange. filebeat fails three times on close that files in my configuration. It should close file on rename (didn't), then on 5 minutes of inactivity after rename (didn't) then on file delete in 1 hour after rename (also didn't).
In general FB has an at least once guarantee to send files. So if not all lines of a file are confirmed by LS, the file is kept open. This can be "stopped" by close_timeout in the newer FB versions. In case the file is deleted before FB picks it up again, some of the log lines will be lost.
Having the send error in the logs is a very good indication that the connection to LS should be checked and the LS logs on why this is happening. There are quite a few other threads with the same issue.
In case these errors don't show up we need to dig deeper on the issue. Could you share your LS output configuration parts too for the ones that didn't so far?
@swethamahesh There seem to be some entry in your FB config which are not in the right place or indented correctly. It should not be related to this problem but thought it's worth mentioning.
We have three almost the same systems with the same software and processes. And only one has this issue. The only thing that could be the reason is such filebeat and logstash logs content:
Logstash:
{:timestamp=>"2017-11-06T15:19:49.287000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Beats input", :exception=>LogStash::Inputs::Beats::InsertingTo
QueueTakeTooLong, :level=>:warn}
{:timestamp=>"2017-11-06T15:19:49.307000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the
current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:war
n}
{:timestamp=>"2017-11-06T15:19:50.361000+0000", :message=>"CircuitBreaker::Open", :name=>"Beats input", :level=>:warn}
{:timestamp=>"2017-11-06T15:19:50.362000+0000", :message=>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the
current connection and rejecting new connection until the pipeline recover.", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::OpenBreaker, :level=>:warn}
{:timestamp=>"2017-11-06T15:19:50.362000+0000", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :l
evel=>:warn}
And at the same time in filebeat logs:
2017-11-06T15:19:49Z ERR Failed to publish events caused by: EOF
2017-11-06T15:19:49Z INFO Error publishing events (retrying): EOF
2017-11-06T15:20:20Z ERR Failed to publish events caused by: read tcp 127.0.0.1:33444->127.0.0.1:5044: i/o timeout
2017-11-06T15:20:20Z INFO Error publishing events (retrying): read tcp 127.0.0.1:33444->127.0.0.1:5044: i/o timeout
Looks like LS never resume that files and FB keeps them open forever.
The logstash logs looks like form very old Logstash version. Logstash is complaining about the internal pipeline being stalled -> closes connections. The circuite breaker allows for new connections only after some timeout.
Consider updating your Logstash version.
Figure out why logstash is blocking? Unresponsive output? Some grok pattern going crazy?
You're right. There is version 2. We had issues with version 5 there. But it was before we installed filebeat there. Ok, we'll discuss about LS upgrade with team to check that.
We moved to LS 5.6 (the last version from yum repo). Now we have no any ERR/WARN logs in LS and FB. But the situation is the same. Our log scheme is next:
Java processes write their logs to processname-running.log, every hour that logs are renaming (same inode but no any new output) and process recreate the same processname-running.log (new inode).
Cron script removes any log that older then 61 minutes. In general every our log lives 2 hours (the same inode).
But after removing looks like all that logs (or almost all) keep open by FB (we use lsof to confirm it). So what else can we do?
As I said before - we have three the same servers with the same FB+LS configs and versions but issue exist only on one of them and exist permanently.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
Lets see what we can do here.