# Filebeat is not able to ship data after few scan cycles

**URL:** <https://discuss.elastic.co/t/filebeat-is-not-able-to-ship-data-after-few-scan-cycles/65227>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 7, 2016, 1:27pm UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-ship-data-after-few-scan-cycles/65227 "2016-11-07T13:27:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chaitra\_Kurdekar](https://avatars.discourse-cdn.com/v4/letter/c/e480ec/32.png) [@Chaitra\_Kurdekar](https://discuss.elastic.co/u/Chaitra_Kurdekar)\
**Post date:** [November 7, 2016, 1:27pm UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-ship-data-after-few-scan-cycles/65227/1 "2016-11-07T13:27:10Z")

</div>

Hi,  
Issue 1) i have configured file beat (filebeat-5.0.0-rc1-linux-x86\_64) with elasticsearch (without logstash). Its observed after few scan cycles it is not able to ship the data from log files to elasticsearch engine.

here is the command executed to run filebeat as bacjground process  
./filebeat -c filebeat.full.yml -e"\*"&

Issue 2) Is there any way to safely shutdown filebeat without killing process?  
Issue 3) also, on restart of filebeat, it is reading contents from scrach instaed of reading from last line which was added in log file. timebeing i have set tail\_files: true not sure is this going to solve the problem.

Please help. Note i have raised 3 question, pl respond by each issue wise.

here is the filebeat.full.yml contents:

```auto
filebeat.prospectors:
- input_type: log
  paths:
    - /opt/apache-tomcat-8.0.27/logs/brandster-request-response.log
  document_type: brandster
  json.message_key: timestamp
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  tail_files: true
filebeat.idle_timeout: 60s
output.elasticsearch:
  hosts: ["10.122.65.43:9200","10.122.65.45:9200","10.122.65.46:9200","10.122.65.47:9200","10.122.65.48:9200"]
  index: "brandster-idx"
  template.enabled: true
  template.path: "/opt/filebeat-5.0.0-rc1-linux-x86_64/filebeat.template.json"
  template.overwrite: true
output.file:
  enabled: true
  path: "/opt/filebeat-5.0.0-rc1-linux-x86_64/temp"
output.console:
  enabled: true
  pretty: true
logging.to_files: true
logging.files:
  path: "/opt/filebeat-5.0.0-rc1-linux-x86_64/logs"

```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 7, 2016, 5:43pm UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-ship-data-after-few-scan-cycles/65227/2 "2016-11-07T17:43:52Z")

</div>

1. Can you share some log files here? There should be more data on why no further data is shipped.
2. You could be interested in [https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-global-options.html#shutdown-timeout](https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-global-options.html#shutdown-timeout)
3. I think something goes wrong with persisting the state to disk or sending your data. If data is not sent, state is also not persisted. I assume solving problem 1 will solve 3 automatically. No need for tail\_files

Please use GA release instead of rc1.

---

<div class="post-metadata">

**Author:** ![Chaitra\_Kurdekar](https://avatars.discourse-cdn.com/v4/letter/c/e480ec/32.png) [@Chaitra\_Kurdekar](https://discuss.elastic.co/u/Chaitra_Kurdekar)\
**Post date:** [November 9, 2016, 6:26am UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-ship-data-after-few-scan-cycles/65227/3 "2016-11-09T06:26:34Z")

</div>

Thanks Ruflin.

1. Log file is less than 1 MB  
2-3) Is there any way that filebeat will run forever and will not shutdown?  
even if i specify shutdown\_timeout as 24 hours what will happen? is it recommonded?  
how we can recognise filebeat is running or has stopped? if we we filebeat as background process PID is still showing as active.  
what is standard way to watch and restart filebeat?

where i can find out GA release? filebeat-5.0.0-linux-x86\_64 is it GA release?  
Thanks  
Chaitra

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 10, 2016, 8:59am UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-ship-data-after-few-scan-cycles/65227/4 "2016-11-10T08:59:08Z")

</div>

1. I meant if you could share the content of the log file. Best paste it into a gist and link it here.
2. shutdown\_timeout only has an affect when you stop filebeat. Here are the docs: [https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-global-options.html#shutdown-timeout](https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-global-options.html#shutdown-timeout)

Here you can download the most recent releases: [https://www.elastic.co/downloads/beats](https://www.elastic.co/downloads/beats)

About running filebeat: That depends on your environment and how you installed filebeat.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 8, 2016, 8:59am UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-ship-data-after-few-scan-cycles/65227/5 "2016-12-08T08:59:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.