Filebeat is not able to ship data after few scan cycles

[Chaitra_Kurdekar]

Hi,
Issue 1) i have configured file beat (filebeat-5.0.0-rc1-linux-x86_64) with elasticsearch (without logstash). Its observed after few scan cycles it is not able to ship the data from log files to elasticsearch engine.

here is the command executed to run filebeat as bacjground process
./filebeat -c filebeat.full.yml -e"*"&

Issue 2) Is there any way to safely shutdown filebeat without killing process?
Issue 3) also, on restart of filebeat, it is reading contents from scrach instaed of reading from last line which was added in log file. timebeing i have set tail_files: true not sure is this going to solve the problem.

Please help. Note i have raised 3 question, pl respond by each issue wise.

here is the filebeat.full.yml contents:

filebeat.prospectors:
- input_type: log
  paths:
    - /opt/apache-tomcat-8.0.27/logs/brandster-request-response.log
  document_type: brandster
  json.message_key: timestamp
  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  tail_files: true
filebeat.idle_timeout: 60s
output.elasticsearch:
  hosts: ["10.122.65.43:9200","10.122.65.45:9200","10.122.65.46:9200","10.122.65.47:9200","10.122.65.48:9200"]
  index: "brandster-idx"
  template.enabled: true
  template.path: "/opt/filebeat-5.0.0-rc1-linux-x86_64/filebeat.template.json"
  template.overwrite: true
output.file:
  enabled: true
  path: "/opt/filebeat-5.0.0-rc1-linux-x86_64/temp"
output.console:
  enabled: true
  pretty: true
logging.to_files: true
logging.files:
  path: "/opt/filebeat-5.0.0-rc1-linux-x86_64/logs"

[ruflin]

1. Can you share some log files here? There should be more data on why no further data is shipped.
2. You could be interested in https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-global-options.html#shutdown-timeout
3. I think something goes wrong with persisting the state to disk or sending your data. If data is not sent, state is also not persisted. I assume solving problem 1 will solve 3 automatically. No need for tail_files

Please use GA release instead of rc1.

[Chaitra_Kurdekar]

Thanks Ruflin.

1. Log file is less than 1 MB
   2-3) Is there any way that filebeat will run forever and will not shutdown?
   even if i specify shutdown_timeout as 24 hours what will happen? is it recommonded?
   how we can recognise filebeat is running or has stopped? if we we filebeat as background process PID is still showing as active.
   what is standard way to watch and restart filebeat?

where i can find out GA release? filebeat-5.0.0-linux-x86_64 is it GA release?
Thanks
Chaitra

[ruflin]

1. I meant if you could share the content of the log file. Best paste it into a gist and link it here.
2. shutdown_timeout only has an affect when you stop filebeat. Here are the docs: https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-global-options.html#shutdown-timeout

Here you can download the most recent releases: https://www.elastic.co/downloads/beats

About running filebeat: That depends on your environment and how you installed filebeat.