# Filebeat never releases unupdated logfiles **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 4, 2017, 9:50am UTC](https://discuss.elastic.co/t/filebeat-never-releases-unupdated-logfiles/102673 "2017-10-04T09:50:14Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![giakalisn](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@giakalisn](https://discuss.elastic.co/u/giakalisn)\ **Post date:** [October 4, 2017, 9:50am UTC](https://discuss.elastic.co/t/filebeat-never-releases-unupdated-logfiles/102673/1 "2017-10-04T09:50:14Z")
Sometimes, filebeat keeps old files opened forever. lsof shows us that filebeat is at EOF, but for some unknown reasons it doesn't close the file. I am using latest version of filebeat Here is our setup : close\_inactive: 1m close\_timeout: 2h close\_renamed: true close\_removed: true force\_close\_files: true ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [October 4, 2017, 10:17am UTC](https://discuss.elastic.co/t/filebeat-never-releases-unupdated-logfiles/102673/2 "2017-10-04T10:17:00Z")
Could you share the following additional details? - Full config file - Your setup (LS, ES, ...) - Exact version (I assume 5.6.2?) Also interesting to hear would be on how you rotate the logs. ---
**Author:** ![giakalisn](https://avatars.discourse-cdn.com/v4/letter/g/7993a0/32.png) [@giakalisn](https://discuss.elastic.co/u/giakalisn)\ **Post date:** [October 4, 2017, 11:58am UTC](https://discuss.elastic.co/t/filebeat-never-releases-unupdated-logfiles/102673/3 "2017-10-04T11:58:56Z")
Hello, Config file : ###################### Filebeat Configuration Example ######################### # This file is an example configuration file highlighting only the most common # options. The filebeat.full.yml file from the same directory contains all the # supported options with more comments. You can use it as a reference. # # You can find the full configuration reference here: # [https://www.elastic.co/guide/en/beats/filebeat/index.html](https://www.elastic.co/guide/en/beats/filebeat/index.html) #=========================== Filebeat prospectors ============================= filebeat.prospectors: # Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations. - input\_type: log paths: Version 5.6.2 We also tried 6 Oracle linux 6.5 ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [October 6, 2017, 12:47pm UTC](https://discuss.elastic.co/t/filebeat-never-releases-unupdated-logfiles/102673/4 "2017-10-06T12:47:02Z")
Could you share the full config file including the output part and also: - Your setup (LS, ES, ...) - Details on how you rotate the logs. Please make sure the formatting of the config files look good in the post. You can use three ticks before and after the config files. ---
**Author:** ![voudas75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voudas75/32/22868_2.png) [@voudas75](https://discuss.elastic.co/u/voudas75)\ **Post date:** [October 8, 2017, 5:03pm UTC](https://discuss.elastic.co/t/filebeat-never-releases-unupdated-logfiles/102673/5 "2017-10-08T17:03:02Z")
1. Concerning setup : > output.logstash: > # The Logstash hosts > hosts: ["xx.yyy:zzz.141:5702","xx.yyy:zzz.142:5702"] > loadbalance: true 1. Concerning rotate procedure : No specific rotation procedure exist . New log file is created under specific directory its time a new "operator" logins the system and traces respective actions ( similar to audit ) . Note ; Respective Log file could be idle for a long time period ( e.g 20 minutes ) before written again . Housekeeping procedure is implemented on OS level using a script which proceeds with gzip when respective file is idle ( based on its file time modification ) for e.g 30 minutes and after a period of time is moving from respective directory . Our issue appears when such files ( not all of them) , due to idle time ( file time modification ) \> 30 minutes are gzipped . Based on close\_inactive conf attr ( 1m or 5m ) , it is expected that FileBeat has already closed the file but no respective message _"Closing because close\_inactive of 1m0s reached"_ exist @ Filebeat logfile . Due to above when file is gzipped , a .deleted file appears on lsof ( or .nfs in case of NFS dir ) which reserves space @ disk . The only way to get rid of specific entries and release space is restart of filebeat process . Some points of interest based on our investigation - **Close\_timeout** parameter - when used for testing - is working as expected . - There are consecutive log entries with _"Harvester started for file: ...."_ for the same filename , without any _"Closing because close\_inactive of 1m0s reached"_ between of them . ( Start time of new harvest matches a new entry at respective log file after a long idle time ) Is it expected ? Have you noticed similar behavior ? - We intend to use **clean\_inactive/ignore\_older** parameters in order to clean registry . Do you agree with that ? ---
**Author:** ![voudas75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voudas75/32/22868_2.png) [@voudas75](https://discuss.elastic.co/u/voudas75)\ **Post date:** [October 8, 2017, 8:32pm UTC](https://discuss.elastic.co/t/filebeat-never-releases-unupdated-logfiles/102673/6 "2017-10-08T20:32:16Z")
1.Concerning not expected behavior of close\_inactive flag , could it be related to LS functionality ? Should we check on LS side ? Is it possible that LS do not send expected ack , and as a result close\_inactive timer is not enabled ? 1. Respective messages appears often : `ERR Failed to publish events: write tcp xxx:36859->yyy:5702: write: connection reset by peer` Could it be related with our issue ? Do you suggest increasing value : **client\_inactivity\_timeout**? 2. When using close\_timeout , we have 2 different behaviors on log file : a. > 2017-10-05T07:35:41+03:00 INFO Closing harvester because close\_timeout was reached. > 2017-10-05T07:35:41+03:00 INFO Reader was closed: /New\_sbllog/logs/xxxx\_0013\_13631523.log. Closing. b. Only the first message appears without Reader message for specific log file . ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [October 9, 2017, 1:01pm UTC](https://discuss.elastic.co/t/filebeat-never-releases-unupdated-logfiles/102673/7 "2017-10-09T13:01:37Z")
The error above was what I was kind of looking for (or one of them) in when I asked for the logs. It means there is something off with the connection to LS. As long as FB did not get the confirmation that all events are sent, it will keep the file open. The only option to prevent this is `close_timeout` which will force close the file. Do you see anything in the LS logs on why this connection is reset? Do you have a Load balancer or something similar in place? As I don't see pipelining in the above LS output config, I assume you don't have it eanbled? ---
**Author:** ![voudas75](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/voudas75/32/22868_2.png) [@voudas75](https://discuss.elastic.co/u/voudas75)\ **Post date:** [October 9, 2017, 2:25pm UTC](https://discuss.elastic.co/t/filebeat-never-releases-unupdated-logfiles/102673/9 "2017-10-09T14:25:43Z")
Thanks for your immediate response . Configuration : ``` output.logstash: # The Logstash hosts hosts: ["xx.yyy:zzz.141:5702","xx.yyy:zzz.142:5702"] loadbalance: true ``` Pipelining is not enabled . No respective errors exist on LogStash Log FIles concerning same timestamp with Filebeat Log Messages. Similar errors exist , but on different time frames. * * * Can you suggest an appropriate configuration for Logstash ? 2 Logstash instances running on 2 servers . Should we increase client\_inactivity\_timeout as mentioned on respective issue ? > [@\[Solved\] Filebeat -\> Logstash : connection reset by peer](https://discuss.elastic.co/t/solved-filebeat-logstash-connection-reset-by-peer/87012): > > Hi there ! I'm testing a pipe like that for dumping my logs : filebeat -\> logstash -\> elasticsearch and I have strange errors from filebeat : 2017-05-24T18:08:51+02:00 DBG Try to publish 2 events to logstash with window size 105 2017-05-24T18:08:51+02:00 DBG handle error: read tcp 172.17.1.5:45543-\>172.17.105.2:5044: read: connection reset by peer 2017-05-24T18:08:51+02:00 DBG 0 events out of 2 events sent to logstash. Continue sending 2017-05-24T18:08:51+02:00 DBG close connection 2017-05… One related question : Is there a safe "dummy" configuration to use in order to confirm that we do not have any issue on close\_\* parameters / OS etc. and issue is focused on LS config/output ? ( e.g using file output ) ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [October 10, 2017, 10:55am UTC](https://discuss.elastic.co/t/filebeat-never-releases-unupdated-logfiles/102673/10 "2017-10-10T10:55:32Z")
Increasing `client_inactivity_timeout` should remove the errors but in general it should not directly impact the problem. As far as I understand you, close\_timeout solves the problem? As long as your LS instances are not overloaded, the other `close_*` options should also release the file handlers. Your best test environment to eliminate network in LS problems is definitively writing to files and see what happens. If the the problem does not occur when you write to file, it's a strong indicator that it's related to LS, network or LS output on the beats side. How many events per second do you have? Are your LS instances under load? Do you have any load balancer in between beats and logstash? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 7, 2017, 10:55am UTC](https://discuss.elastic.co/t/filebeat-never-releases-unupdated-logfiles/102673/11 "2017-11-07T10:55:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.