# Filebeat stops sending logs with no reason **URL:** \ **Category:** Beats\ **Created:** [September 20, 2017, 8:19am UTC](https://discuss.elastic.co/t/filebeat-stops-sending-logs-with-no-reason/101130 "2017-09-20T08:19:01Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![serkin](https://avatars.discourse-cdn.com/v4/letter/s/a698b9/32.png) [@serkin](https://discuss.elastic.co/u/serkin)\ **Post date:** [September 20, 2017, 8:19am UTC](https://discuss.elastic.co/t/filebeat-stops-sending-logs-with-no-reason/101130/1 "2017-09-20T08:19:01Z")
I have an unusual issue I have server sending logs to _elasticsearch_ using _filebeat_ Everything goes ok until suddenly _filebeat_ stops sending logs. But if I curl to _elasticsearch_ it responds. tcpdump shows that _filebeat_ doesn't even try to send logs. If I restart _elasticsearch_ everything works fine for a while. Sometimes in addition I need to restart filebeat to start sending logs again Noting helpful in elastic trace I have output from _filebeat_ like below: ``` 2017-09-19T18:05:16+03:00 DBG Run prospector 2017-09-19T18:05:16+03:00 DBG Start next scan 2017-09-19T18:05:16+03:00 DBG Check file for harvesting: /var/log/nginx/access.log 2017-09-19T18:05:16+03:00 DBG Update existing file for harvesting: /var/log/nginx/access.log, offset: 103391024 2017-09-19T18:05:16+03:00 DBG Check file for harvesting: /srv/current/var/logs/prod.log 2017-09-19T18:05:16+03:00 DBG Harvester for file is still running: /var/log/nginx/access.log 2017-09-19T18:05:16+03:00 DBG Update existing file for harvesting: /srv/current/var/logs/prod.log, offset: 701936 2017-09-19T18:05:16+03:00 DBG Harvester for file is still running: /srv/current/var/logs/prod.log 2017-09-19T18:05:16+03:00 DBG Prospector states cleaned up. Before: 26, After: 26 2017-09-19T18:05:16+03:00 DBG Prospector states cleaned up. Before: 61, After: 61 ``` My config is: ```auto filebeat.prospectors: - input_type: log document_type: nginx_access_log paths: - /var/log/nginx/access.log - input_type: log document_type: symfony_log paths: - /srv/current/var/logs/prod.log output.elasticsearch: hosts: ["elastic:9200"] index: "my_logs" max_retries: 1 ``` Filebeat version: 5.4.1 Elasticsearch version: 5.5.1 I run elasticsearch in docker container [docker.elastic.co/elasticsearch/elasticsearch:5.5.1](http://docker.elastic.co/elasticsearch/elasticsearch:5.5.1) m.max\_map\_count=262144 is set Approximately I send 20Mb/minute logs Playing around with the settings didn't help Any ideas where to dig? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 20, 2017, 1:32pm UTC](https://discuss.elastic.co/t/filebeat-stops-sending-logs-with-no-reason/101130/2 "2017-09-20T13:32:10Z")
Having the debug logs is the right way to go. Any chance you could share more of the debug log? What is the rotation algorithm you are using? I assume the access logs are rotated and you want to finish read them. I would recommend you to specifiy something like `access.log*` to make sure also the rotated logs are read. In the few log lines you provide above it states that 61 states are there but you only have 2 log lines active. So I assume the state is not fully cleaned up. I'm not sure what is causing your issue yet but it's kind of strange that restarting ES sometimes helps and sometimes you need to restart FB. ---
**Author:** ![serkin](https://avatars.discourse-cdn.com/v4/letter/s/a698b9/32.png) [@serkin](https://discuss.elastic.co/u/serkin)\ **Post date:** [September 20, 2017, 2:23pm UTC](https://discuss.elastic.co/t/filebeat-stops-sending-logs-with-no-reason/101130/3 "2017-09-20T14:23:55Z")
Certainly I can produce more logs. From filebeat or elasticsearch? Are you saying that `Prospector states cleaned up. Before: 61, After: 61` means I have 61 file pointers opened but I have only 2 files? How can I completely clean them? for _/var/log/nginx/access.log_ I have rule in logrotate ``` /var/log/nginx/*.log { daily missingok rotate 14 compress delaycompress notifempty create 0640 www-data adm sharedscripts prerotate if [-d /etc/logrotate.d/httpd-prerotate]; then \ run-parts /etc/logrotate.d/httpd-prerotate; \ fi \ endscript postrotate invoke-rc.d nginx rotate >/dev/null 2>&1 endscript } ``` for /srv/current/var/logs/prod.log I don't have rotation set up yet I just do `$ cat /dev/null > /srv/current/var/logs/prod.log` daily. In addition I would like to say that everything worked well until we moved to production and started to send logs massively. My colleague noticed that if we remove pipelines from elasticsearch ingest modes logs keep sending longer. ---
**Author:** ![serkin](https://avatars.discourse-cdn.com/v4/letter/s/a698b9/32.png) [@serkin](https://discuss.elastic.co/u/serkin)\ **Post date:** [September 20, 2017, 3:20pm UTC](https://discuss.elastic.co/t/filebeat-stops-sending-logs-with-no-reason/101130/4 "2017-09-20T15:20:31Z")
more logs from filebeat ``` 2017-09-20T18:15:59+03:00 DBG Run prospector 2017-09-20T18:15:59+03:00 DBG Start next scan 2017-09-20T18:15:59+03:00 DBG Check file for harvesting: /srv/current/var/logs/prod.log 2017-09-20T18:15:59+03:00 DBG Update existing file for harvesting: /srv/current/var/logs/prod.log, offset: 75091 2017-09-20T18:15:59+03:00 DBG Harvester for file is still running: /srv/current/var/logs/prod.log 2017-09-20T18:15:59+03:00 DBG Prospector states cleaned up. Before: 63, After: 63 2017-09-20T18:16:03+03:00 ERR Failed to perform any bulk index operations: Post http://elastic:9200/_bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers) 2017-09-20T18:16:03+03:00 INFO Error publishing events (retrying): Post http://elastic:9200/_bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers) 2017-09-20T18:16:03+03:00 DBG send fail 2017-09-20T18:16:09+03:00 DBG Run prospector 2017-09-20T18:16:09+03:00 DBG Start next scan 2017-09-20T18:16:09+03:00 DBG Check file for harvesting: /var/log/nginx/access.log 2017-09-20T18:16:09+03:00 DBG Update existing file for harvesting: /var/log/nginx/access.log, offset: 11950635 2017-09-20T18:16:09+03:00 DBG Harvester for file is still running: /var/log/nginx/access.log 2017-09-20T18:16:09+03:00 DBG Prospector states cleaned up. Before: 30, After: 30 2017-09-20T18:16:09+03:00 DBG Run prospector 2017-09-20T18:16:09+03:00 DBG Start next scan 2017-09-20T18:16:09+03:00 DBG Check file for harvesting: /srv/current/var/logs/prod.log 2017-09-20T18:16:09+03:00 DBG Update existing file for harvesting: /srv/current/var/logs/prod.log, offset: 75091 2017-09-20T18:16:09+03:00 DBG Harvester for file is still running: /srv/current/var/logs/prod.log 2017-09-20T18:16:09+03:00 DBG Prospector states cleaned up. Before: 63, After: 63 2017-09-20T18:16:11+03:00 DBG ES Ping(url=http://elastic:9200, timeout=1m30s) 2017-09-20T18:16:11+03:00 DBG Ping status code: 200 2017-09-20T18:16:11+03:00 INFO Connected to Elasticsearch version 5.5.1 2017-09-20T18:16:11+03:00 INFO Trying to load template for client: http://elastic:9200 2017-09-20T18:16:11+03:00 DBG HEAD http://elastic:9200/_template/filebeat 2017-09-20T18:16:11+03:00 INFO Template already exists and will not be overwritten. ``` ---
**Author:** ![serkin](https://avatars.discourse-cdn.com/v4/letter/s/a698b9/32.png) [@serkin](https://discuss.elastic.co/u/serkin)\ **Post date:** [September 25, 2017, 1:53pm UTC](https://discuss.elastic.co/t/filebeat-stops-sending-logs-with-no-reason/101130/5 "2017-09-25T13:53:06Z")
It turned out that the problem is in elasticsearch. Will ask elasticsearch support. Thank you anyway) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 11, 2017, 8:19am UTC](https://discuss.elastic.co/t/filebeat-stops-sending-logs-with-no-reason/101130/6 "2017-10-11T08:19:34Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.