# Filebeat not reading new logs changes when manually updated one of the logs **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 13, 2015, 1:10pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058 "2015-12-13T13:10:02Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [December 13, 2015, 1:10pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/1 "2015-12-13T13:10:02Z")
Manually update one of the logs in the logs directory but filebeat is not reading the logs unless filebeat service is restarted . I'm sending the logs to logstash once it's read. Using filebeat version filebeat-1.0.0-i686 ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 13, 2015, 4:13pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/2 "2015-12-13T16:13:54Z")
Exactly how did you update the log? Did you append to it with e.g. `echo test message >> logfile.log`? ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [December 14, 2015, 5:19am UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/3 "2015-12-14T05:19:04Z")
I've tried to edit it manually by copying many lines and appending to the end of the log file. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [December 14, 2015, 6:08am UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/4 "2015-12-14T06:08:56Z")
> [@magnusbaeck](#): > > append to it with e.g. echo test message \>\> logfile.log I've tried updating the logs with append to it with e.g. echo test message \>\> logfile.log. Still the newly appended logs are read only after restarting the filebeat service. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 14, 2015, 6:32am UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/5 "2015-12-14T06:32:11Z")
I'd try enabling debug logs with `-d *` and see what comes up. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [December 14, 2015, 7:09am UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/6 "2015-12-14T07:09:50Z")
Dec 14 11:48:59 user-VirtualBox ./filebeat[6459]: prospector.go:341: Update existing file for harvesting: /log Dec 14 11:48:59 user-VirtualBox ./filebeat[6459]: prospector.go:383: Not harvesting, file didn't change: /log Dec 14 11:48:59 user-VirtualBox ./filebeat[6459]: prospector.go:219: Check file for harvesting: /log The updates events are not captured . Please find the logs , removed the path and name of logs as this is customer specific logs. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [December 14, 2015, 7:38am UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/7 "2015-12-14T07:38:09Z")
Adding some more information on the logs : Dec 14 13:09:36 user-VirtualBox ./filebeat[32553]: client.go:244: ES Ping(url=http://localhost:9200, timeout=1m30s) Dec 14 13:09:36 user-VirtualBox ./filebeat[32553]: client.go:249: Ping request failed with: Head [http://localhost:9200](http://localhost:9200): dial tcp 127.0.0.1:9200: getsockopt: connection refused Dec 14 13:09:36 user-VirtualBox ./filebeat[32553]: single.go:121: Connecting error publishing events (retrying): Head [http://localhost:9200](http://localhost:9200): dial tcp 127.0.0.1:9200: getsockopt: connection refused Dec 14 13:09:36 user-VirtualBox ./filebeat[32553]: single.go:143: send fail Dec 14 13:09:36 user-VirtualBox ./filebeat[32553]: single.go:150: backoff retry: 2s ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 14, 2015, 7:46am UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/8 "2015-12-14T07:46:05Z")
It's not clear whether that error is related to your initial problem, but it's obviously something you have to address, e.g. by starting Elasticsearch or reconfiguring Filebeat to send events to the host where you _are_ running ES. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [December 14, 2015, 7:50am UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/9 "2015-12-14T07:50:51Z")
> [@vivc](#): > > ./filebeat[32553]: client.go:249: Ping request failed with: Head Currently I've configured filebeat output to logstash . I've configured logstash out put to Elasticsearch. When I restart the filebeat it's reading the updated logs with out restarting the Elasticsearch. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 14, 2015, 8:29am UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/10 "2015-12-14T08:29:33Z")
From the logs it's evident that you've configured Filebeat to send directly to Elasticsearch. If this isn't what you intended, please reconfigure Filebeat first before attempting to solve any remaining issues. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [December 14, 2015, 9:49am UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/11 "2015-12-14T09:49:04Z")
Please find the Filebeat.yml file configuration below : output configuration : elasticsearch: # Array of hosts to connect to. # Scheme and port can be left out and will be set to the default (http and 9200) # In case you specify and additional path, the scheme is required: [http://localhost:9200/path](http://localhost:9200/path) # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200 hosts: ["localhost:9200"] ### Logstash as output logstash: # The Logstash hosts hosts: ["192.168.56.1:5044"] ``` # Number of workers per Logstash host. worker: 1 ``` My intend was to configure filebeat to output the logs to Logstash and from there to elastic search. I think the line in elastic search output section was already uncommented in the downloaded yml file.please correct me if I'm wrong. hosts: ["localhost:9200"] when I tried to comment the line it is throwing the error Error Initialising publisher: no host configuration found ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 14, 2015, 9:52am UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/12 "2015-12-14T09:52:19Z")
Comment out both the "elasticsearch:" and the "hosts:" line. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [December 14, 2015, 1:10pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/13 "2015-12-14T13:10:19Z")
You have both outputers enabled, logstash and elasticsearch. Filebeat is waiting for ACK from all outputers. Due to no elasticsearch instance being available, internal buffers/queues eventually run full and filebeat starts blocking until it can clear it's buffers by send buffered content to elasticsearch and logstash. please comment out the elasticsearch output and check if issue has been resolved. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [December 14, 2015, 3:09pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/14 "2015-12-14T15:09:31Z")
Steffen/magnusbaeck, Thanks for your suggestions , Now the issue is resolved and it's taking the logs dynamically 🙂 ---
**Author:** ![abinay](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@abinay](https://discuss.elastic.co/u/abinay)\ **Post date:** [January 23, 2016, 5:55pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/15 "2016-01-23T17:55:43Z")
Hii I am going through the same issue my friend . I was first using logstash-forwarder to ship my logs to logstash but unfortunately logstash- forwarder was not tailing the file. It used to start reading the file from the beginning which was not a good architectue especially in the situation when my log file can get too large . Even after starting logstash-forwarder with -tail = true command in the terminal logstash-forwarder used to only harvest the file but was not processing the events . Thats why i switched to filebeat . Now the filebeat is tailing the file when tail\_files = true is done in the configuration file but its not pushing the logs to logstash dynamically - meaning i have to manually restart filebeat each and evertime so as to send the logs from filebeat to logstash . I am really pissed of with such behavior of filebeat . Further the window of logstash-forwarder was very friendly as it used to tell how many events are getting processed and all but when i run file beat using service file beat start it does not let me know whether how many events it sent and it also tries only once to connect with logstash unlike logstash-forwarder which continuously keeps trying to connect to logstash server..............So please let me know about this ASAP . We need to get production ready pretty soon ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 23, 2016, 6:52pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/16 "2016-01-23T18:52:17Z")
@abinay—please start a new thread for your unrelated problem. ---
**Author:** ![abinay](https://avatars.discourse-cdn.com/v4/letter/a/838e76/32.png) [@abinay](https://discuss.elastic.co/u/abinay)\ **Post date:** [January 25, 2016, 6:55am UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/17 "2016-01-25T06:55:51Z")
@magnusbaeck done . Can you please answer it there . ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 9:56pm UTC](https://discuss.elastic.co/t/filebeat-not-reading-new-logs-changes-when-manually-updated-one-of-the-logs/37058/18 "2017-07-05T21:56:25Z")